NOT containers 101: bringing your own AMI (or configuring on the fly)

Or, “ah-mee”, if you’re an Amazon employee that isn’t me 🙅🏻

As with everything, sometimes the defaults aren’t enough. Maybe you have a custom package or libraries that you want to install directly on the host, maybe you need to run 3rd-party daemon processes. Maybe you’re just a rebel.

Two main options for configuring: 1) bring a custom AMI with your requirements already installed, or 2) configure at instance boot with EC2 user data. Or do both! Let’s break all these options down.


Bringing your own AMI

With EKS (Elastic Container Service for Kubernetes), this is easy: just select your own when you create and register your worker nodes.

With ECS (Elastic Container Service), this is a bit more complex 😅

So, you might know that Amazon ships something called the “Amazon ECS-optimized AMI”: a pre-configured and tested image that contains the basic requirements for an ECS instance. According to the documentation, the current version includes:

  • The latest minimal version of the Amazon Linux AMI
  • The latest version of the Amazon ECS container agent (1.20.2)
  • The recommended version of Docker for the latest Amazon ECS container agent (18.06.1-ce)
  • The latest version of the ecs-init package to run and monitor the Amazon ECS agent (1.20.2-1)

If you don’t want to use this AMI- that’s ok! A few steps/requirements:

  1. Make sure your AMI has the right requirements! Mainly, a Linux distribution running version 3.1 or above of the kernel.
  2. Install the ecs-agent
[ec2-user ~]$ sudo yum install -y ecs-init

3. Install Docker daemon (if it’s not already installed, it should be on Amazon Linux)

[ec2-user ~]$ sudo yum install docker -y

4. Register instance with cluster. If you only have one variable to set here, you do that like this (we’ll get to how to set more than one):

#!/bin/bash echo "ECS_CLUSTER=<cluster_name>" >> /etc/ecs/ecs.config

5. Optional (but good): some sort of init process to manage the ecs-agent.

[ec2-user ~]$ service docker start
[ec2-user ~]$ sudo start ecs

You’ll also need to ensure you have the right role when you launch your new instance from the AMI.

For the purposes of this post, I’m assuming you’re building a custom AMI on Amazon Linux. If you’re not, the same process is relevant, but the install instructions for the agent will be different. Start here

All of these steps can be executed in EC2 user-data as part of a launch configuration (more on that later). Alternatively, you can start an EC2 instance, perform whatever customizations you want, stop the instance, and then from the Instance Actions menu, select “Create Image”.

A (semi) brief footnote on setting multiple values in /etc/ecs/ecs.config

The ecs-agent supports a ton of different parameters, controlling everything from logging, to roles, to garbage collection. We already looked at how to set a single value (ECS_CLUSTER), but what about multiple values?

#!/bin/bash 
cat <<'EOF' >> /etc/ecs/ecs.config
ECS_ENABLE_TASK_IAM_ROLE=true
ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true
ECS_LOGFILE=/log/ecs-agent.log
ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs"]
ECS_LOGLEVEL=info
ECS_CLUSTER=default
EOF

Full list of supported parameters is here. Like the steps above, these values can be set in user-data. Speaking of which:

Configuring instances with EC2 user-data (and more) 🐉

EC2 User-data is amazing: it’s just shell scripts that you can run during your instance boot process. You can do pretty much anything here. And anything you can’t do with user-data, you can probably do with cloud-init/cloud-boothook ✨

So what’s the difference between user-data and cloud-init? Cloud-boothook formatted user-data (cloud-init) will run earlier in the configuration process than a shell script. Cloud-init-per is a utility that controls how often these boothooks execute.

cloud-init-per frequency name cmd [ arg1 [ arg2 [ ... ] ]

Let’s look at a container-related example: passing Docker daemon flags. Fun fact: you can enable Docker flags that aren’t directly supported by the ECS Console by passing them in user-data. There’s a catch, though- you need to pass them before the Docker daemon starts up, which means you need to use cloud-init-per, rather than a shell script.

cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=20G"' >> /etc/sysconfig/docker

If you want to pass multiple options:

#cloud-boothook 
cloud-init-per instance docker_options
cat <<'EOF' >> /etc/sysconfig/docker
OPTIONS="${OPTIONS} --storage-opt dm.basesize=20G" HTTP_PROXY=http://proxy.example.com:80/
EOF

For user-data, the sky is pretty much the limit: you can install packages, start and stop services, configure users. If it can be done with Bash, it can be done here. When we looked at the steps for configuring a custom AMI for ECS earlier, we could do all of these steps in user-data:

#!/bin/bash 
sudo yum update -y
sudo yum install -y ecs-init docker

cat <<'EOF' >> /etc/ecs/ecs.config
ECS_ENABLE_TASK_IAM_ROLE=true
ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true
ECS_LOGFILE=/log/ecs-agent.log
ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs"]
ECS_LOGLEVEL=info
ECS_CLUSTER=default
EOF
sudo service docker start
sudo start ecs

OK, but what if I want to pass Docker daemon options, AND configure ECS? You can do this with a MIME multi-part file, which will let you run both cloud-init-per directives, and a shell script:

Content-Type: multipart/mixed; boundary="==BOUNDARY==" 
MIME-Version: 1.0
--==BOUNDARY==
Content-Type: text/cloud-boothook; charset="us-ascii"
# Set Docker daemon options 
cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=20G"' >> /etc/sysconfig/docker
--==BOUNDARY== 
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash # Set any ECS agent configuration options echo "ECS_CLUSTER=my-ecs-cluster" >> /etc/ecs/ecs.config 

--==BOUNDARY==--

Putting the pieces together for ECS

Once you have your pre-baked AMI, or your user-data scripts written, it’s time to actually use them! For ECS, you do this the same way as with EC2: either by a) launching a single EC2 instance, or b) by creating an autoscaling group and launch configuration. For a single EC2 instance, you can follow the documentation here. If you need to create an autoscaling group and launch configuration, you can start here.


Got a cool user-data trick, or a question/comment/something I should cover next time? Let me know here, or on Twitter. I’m @abbyfuller.