Streamlining the management of AWS Marketplace AMIs

The slippy game of AMI management! (credit: Unsplash)

The AWS Marketplace is our de facto store to publish PredicSis.ai. It is a very convenient way for our users to deploy the product within minutes. In fact the simplicity of the AWS Marketplace is a key feature that PredicSis.ai also benefits from. As we believe that our users should get their first predictive insights in minutes rather than months, it is also key they could provision it in no time, without any configuration.

As of today there are two versions of the product based on Amazon Machine Image (AMI). Creating an AMI and submitting it to the AWS marketplace to launch a new product is a fairly straightforward process. The difficult part is to automate production of these AMIs. And once you achieve this automation you then face a second issue: how do I keep track of these AMIs?

AMI versioning is key here. Before looking in-depth at version management I would like to briefly explain how PredicSis.ai’s AMIs are built. Obviously we do not create AMIs manually through the AWS web console. PredicSis.ai is based on a microservice architecture [1] where each microservice has its own Git repository. Every new commit pushed to one of the repositories will trigger a new build handled by AWS CodePipeline . The last step of the Pipeline is to build the AMI with Packer . As you can imagine, we end up with a pretty long list of AMIs 😏

The AWS Management Console is a great tool but it is not designed to ship production-ready AMIs to the AWS marketplace. Grégoire, our amazing DevOps, has started developing a tool called ShipAMI which initially only listed AMIs. A few iterations later, it now enables us to share the right AMI with the AWS Marketplace account safely and rapidly.

ShipAMI is a command-line tool similar to AWS CLI .

Here’s a quick workflow which demonstrates how we use ShipAMI.

Pick an AMI and “release” it

Every major version of PredicSis.ai is built automatically by CodePipeline. Let’s take version 3.6 as an example.

[~]$ shipami list

ami_id ami-b5eae5d3 is the latest build available for this version and after some serious testing we decide to release it under version 3.6.1.

[~]$ shipami release ami-b5eae5d3 3.6.1 --name prds-ai
ami-16ebe570

This newly created AMI, ami-16ebe570, will be the one sent to the AWS Marketplace.

“copy” the release AMI to the us-east-1 region

[~]$ shipami --region us-east-1 copy ami-16ebe570 --source-region eu-west-1
ami-93116085

“share” the AMI on the AWS Marketplace account

Once the AMI has been successfully copied you can share it the AWS Marketplace account. Again, ShipAMI provides an easy way to do that:

[~]$ shipami — region us-east-1 share ami-93116085

By default ShipAMI shares the AMI with the AWS Marketplace account. The shared AMI now appears on the management portal:

Job done! 👏 Within a few days your AMI will be on the AWS Marketplace.

Synthesis

Thanks to our fully automated build chain, less than 20 minutes is required from the time a developer pushes a new commit and the time the new AMI is ready. This is a huge increase in efficiency for our development team. Even better is that every new AMI is automatically deployed to an EC2 instance thanks to AWS CloudFormation . Our QA team can then validate new features or bug fixes continuously (and of course they get a notification in Slack 😏)

I would greatly appreciate reading your thoughts and questions in the comments. And don’t hesitate to open issues on ShipAMI’s GitHub repository!

Bastien

[1] If you’re interested in building an app based on a microservice architecture I highly recommend the book “Building Microservices, by Sam Newman”.

Article initially published on our blog