Fixing ECS Tasks stuck in a pending state
TL;DR:
If Your ECS tasks are stuck in PENDING, SSH into the instance and run:sudo stop ecs;
sudo start ecs;If that doesn't work, Google for another blog post.
I had a bunch of ECS Tasks after a deploy that were all stuck in a pending state. Nothing in the Event log besides “service NAME has started 6 tasks: …”
Not to be aggro, but no you didn’t start anything, bruh.
I SSH’d into one of the instances with ssh ec2-user@YOUR_EC2_IP
and tailed the ECS agent’s log with:
docker ps | grep ecs-agent # Gets the agent's ID
docker logs — tail 20 -f ID_HERE2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: c593e4c1d8b42460c5100f5db31d326fb003556bcaeab10ccda0534132d7f15f
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: 801a8f0c19dc0e30a89dc78ba2f03673a04be28b029e1c7f929ca7fb07886b14
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: 57efc8940556007f3c807afde27cfb11efec37d6a98e0341fe8576ca55400763
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: 4e021c844cfc69274b23041444981146c9a8c06fd067eb7055b074ade9183d05
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: 85cf8659660be22da2535e46e68b267756348911d780ac25b6636024ac50544d
2017-03-09T18:20:33Z [WARN] Error getting cpu stats, err: No data in the queue, container: dace4a94aca97865ccf8e64686fc576976f795bca51858d997370874e3b7876e
2017-03-09T18:20:33Z [WARN] Error getting instance metrics: No task metrics to report
Which obviously means something, but I have no idea what. I have this little script in my bash profile to help me figure out what ECS’s intensions are with my tasks:
Which will give some output like:
So:
- ECS wants to start my task
- ECS will not start my task
- Yay, good morning.
There is this really handy command I run, its equivalent to re-installing Windows XP when your cable goes down.
sudo stop ecs;
sudo start ecs;
Which restarts the ECS agent. Which worked.