Fixing ECS Tasks stuck in a pending state

TL;DR:
If Your ECS tasks are stuck in PENDING, SSH into the instance and run:
sudo stop ecs;
sudo start ecs;
If that doesn't work, Google for another blog post.

I had a bunch of ECS Tasks after a deploy that were all stuck in a pending state. Nothing in the Event log besides “service NAME has started 6 tasks: …”

You havent seen the ‘O RLY’ owl in years have you?

Not to be aggro, but no you didn’t start anything, bruh.

I SSH’d into one of the instances with ssh ec2-user@YOUR_EC2_IP and tailed the ECS agent’s log with:

docker ps | grep ecs-agent # Gets the agent's ID
docker logs — tail 20 -f ID_HERE
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: c593e4c1d8b42460c5100f5db31d326fb003556bcaeab10ccda0534132d7f15f
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: 801a8f0c19dc0e30a89dc78ba2f03673a04be28b029e1c7f929ca7fb07886b14
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: 57efc8940556007f3c807afde27cfb11efec37d6a98e0341fe8576ca55400763
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: 4e021c844cfc69274b23041444981146c9a8c06fd067eb7055b074ade9183d05
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: 85cf8659660be22da2535e46e68b267756348911d780ac25b6636024ac50544d
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: dace4a94aca97865ccf8e64686fc576976f795bca51858d997370874e3b7876e
2017-03-09T18:20:33Z [WARN] Error getting instance metrics: No task metrics to report

Which obviously means something, but I have no idea what. I have this little script in my bash profile to help me figure out what ECS’s intensions are with my tasks:

Which will give some output like:

So:

1. ECS wants to start my task
2. ECS will not start my task
3. Yay, good morning.

There is this really handy command I run, its equivalent to re-installing Windows XP when your cable goes down.

sudo stop ecs;
sudo start ecs;

Which restarts the ECS agent. Which worked.