The road to automating our Alexa Skills deployments

By Eli Ezeugoh & Tim Swann

Join us on this whistle-stop tour of how we built our Alexa skills for Wireless’ radio content using Amazon’s new Radio Skills Kit. We cruise through how we made the process repeatable, developer-friendly and high quality all while keeping an eye on costs. Finally, we highlight some of the pitfalls we found along the way in the hope that you can learn from our mistakes. We also want to encourage discourse around the development of this now ubiquitous media consumption method, so it enriches the body of work in this space.

As part of the launch of Times Radio, we began consolidating our Alexa skills to help our listeners consume our content but also as a means to streamline the process of provisioning new streams for our smart speaker user base. We built them using Amazon’s Radio Skills Kit (RSK), which is a Amazon’s latest set of interfaces that enable selection and control of digital radio content streamed through an Alexa-enabled device.

Figure 1: Alexa Skill Development Stages

Before we began the build, we set out a few imperative requirements:

• The build must be repeatable with all skill components configuration managed in code.
• The developer experience must be first class.
• We would need a reliable way to keep an eye on the running costs in production.

We recently implemented an internal tagging system for our organisation-wide Amazon hosted resources so we could reliably see the total cost of ownership of our services (we may talk about this in a future post). These new Alexa skills needed to align with these processes.

As we didn’t have much experience developing Alexa skills, the repeatable builds and developer experience aspects were uncharted waters for us, but we thought we’d try all the same.

This post captures how we went about it and aims to help you if you embark on a similar journey. We are, of course, always open to feedback so we improve in the future.

Site clearance

There were a few bits of pre-work we needed to undertake as part of getting set up for skill development:

• At the heart of skill development in the Amazon ecosystem is ask-cli v2, which we installed via npm so that we can enjoy continuing feature development.
• We created a Service Account in Alexa Developer Console with ask configure. We used this to create a profile that links our AWS account to our Alexa Developer Vendor account. The profile is ina ~/.ask/cli_config file and needs to be present for all invocations of ask-cli.
• We set up an empty Lambda function — you need its Amazon Resource Name resource identifier (ARN) in your skill manifest JSON file to be able to create the skill. Note: be sure to set up an Alexa Skills Kit trigger for the Lambda with skill verification disabled; it will save you some heartache. Also, be sure not to reuse the name of the Lambda in your skills code Cloud Formation YAML as on this road lies pain.
• We created both skills via ask-cli with JSON manifest files (you’ll find example manifest.json content in Amazon’s online docs). Make a note of the skillId after you create each skill.

Listing 1: Creating a Skill

To verify that you have created your skill correctly:

Listing 2: Verifying skill status

• For each skill, we created and associated a Radio Catalogue. You’ll need to inform the Alexa folk of your Radio Catalogues so they can start ingesting them.
• Finally, we set up infrastructural elements to support the development and deployment of the skill. The configuration includes:
• IAM roles (used in our deployment pipeline)
• S3 buckets (to house our multi-az lambda functions’ code and skill deployment state)
• Global DynamoDB tables (to store lookup information needed by the lambdas).
• The deployment state storage is crucial since ask-cli will look at this every time you trigger a deployment to determine what bits to redeploy. The buckets are required and used within the deployment configuration for our skills otherwise ask-cli and its Cloud Formation deployer kept creating arbitrary buckets in our AWS accounts. The Global tables ensured that our multi availability zone Lambda functions were capable of quickly looking up data in their regions.

Paving the road to developer happiness

Our approach to repeatability and improved developer experience was to have a pre-production and production implementation of our skills exist side-by-side. Our pre-production skills gave us a way to test both infrastructural deployment and skill implementation in a safe way before triggering a production deployment.

We setup two pipelines:

• Infrastructure pipeline with all the Terraform scripts that created the basis for a deployment to pre-production and production.
• Skill code pipeline that tested, packaged and deployed the Lambda code and radio stations catalogue.

Structurally, our pipeline has 5 stages that look as follows:

Figure 2: Our CircleCI Pipeline

build-and-test-skill, deploy-skill and integration-test-skill are triggered for every feature branch check-in while the deploy-skill-prod step would only be triggered for production builds from check-ins to our master branch on approval.

We use CircleCI for our pipeline steps, but any continuous integration tool will do.

At the heart of it, the deployment script looked as follows (redacted for brevity):

Listing 3: Deploying RSK skill and its radio station catalogue

Skill deployment is a single line of code.

The ask deploy step takes care of three parts of the deployment process:

• Checking the provisioning of the lambda function(s)
• Packaging, uploading and updating the lambda functions, and locking to our skill ID
• Updating the skill manifest, and lambda endpoints within the Alexa Developer Console

The new default mechanism with ask-cli v2 provisioning of AWS resources is Amazon CloudFormation. We tell the ask deploy command how to deploy our resources via the ask-resources.json file:

Listing 4: sample ask-resources.json

So there are two areas where we define our multi-region; the code block and the regionalOverrides block. This same ask-resources.json file is crucial to the separation of our pre-prod and production pipelines too. More on that later.

The skill-stack.yaml referenced above in the skillInfrastructure block defines our AWS resources to be provisioned by the ask deploy command. This stack is a template that is executed once per region.

The skill-stack.yaml as can be seen in Listing 5 file allows us to provision resources, but to set permissions and, as we mentioned earlier, to apply tags to resources allowing us to monitor TCO. On configuring your skill with ask configure, a basic template skill-stack.yaml will be generated for you on which you can build out your customisations. For us that was focused on resource tagging and setting up relevant Environment variables.

Listing 5: Example skill-stack.yaml.

Separation of Production and Pre-Production

The separation for prod and pre-prod deployments, from an ask-cli perspective is relatively straightforward. We make use of an additional .ask/cli_config profile.

We used a differently named profile, one that was connected to the same Amazon Alexa vendor account, such that our dev or pre-prod skill sits alongside the production skill in the Alexa Developer Console but connected to different AWS credentials such that the lambda, and DynamoDB tables were on separate AWS infrastructure.

When you trigger a deployment with ask, you pass the — profile option eg:

ask deploy — profile my-prodution-profile
ask deploy — profile my-pre-prod-profile

The profile is used by ask deploy to check which codebase is being deployed, which cloud formation stack and which assets to deploy. See Listing 4 for the necessary customisations as highlighted.

As you can see from the changes in the ask-resources.json file, we can separate out our different production and pre-production deployments via

• Different profile — crucial
• Different AWS credentials/account (not essential)
• A new skill-stack.yaml (different tags, different function name)
• Skill metadata in skill-package/skill.json and skill-package/pre-prod/skill.json
  This is basically the skill manifest, you’ll need to set different Lambda ARNs in here.

Building the confidence to deploy automatically

Having worked on our infrastructure and gotten a working ask deploy command, we needed to ensure that the command is only triggered if we think it safe to do so.

For our production deployments we have those only triggered via the master branch — committing to master requires merge approvals from several developers. Still, we wanted a suite of tests that can be run to prevent feature regression at the very least. For these we used a combination of mocha, chai, chai-as-promised and nyc to give us unit testing with coverage reports.

Amazon’s skill documentation is very good at providing both example requests that would be fed into a skill, and expected responses. This enabled us to write tests and assertions for each of the Music/Radio API methods that we intended to handle. These we trigger as part of the build-and-test-skill step within our CircleCI pipeline. The CircleCI platform saves our test and coverage reports and allows us to review these at any time.

We also gave ourselves an extra level of confidence by invoking a number of these test requests against our deployed Lambdas as part of our build-and-test-skill step after skill and infrastructure deployment to confirm that they are both deployed correctly and work end-to-end. See Figure 1 for all steps in our pipeline.

Counting the pennies

As mentioned before we wanted to keep an eye on everything in terms of TCO. To achieve this we tagged all AWS resources including Lambdas, Dynamo Tables, CloudWatch logs etc. Now we are using CloudHealth to ingest the costs allotted to the relevant tags as a means to get a sense of what the skill costs us in production.

Looking back: 7 Pain Points and Lessons Learned

There were a few areas where we encountered problems, these could be down to the steps or order of steps we were taking, or to the ask-cli itself, which at the time of writing has been getting updated at break-neck speed, so it’s possible that we caught a bug on one day that was gone the next.

1. Having to create an empty lambda before we could create our skill ID. For many this might not be too big of an issue, but for us, where our production environment is 100% hands-off that was a problem. Given that the deploy step sorts that out, we didn’t feel like this should be a requirement for creating a blank skill/skillID.
2. Dual region deployment. The documentation surrounding this was pretty sparse. There was maybe 1 comment in the dev forums about the regionalOverrides block but not in relation to dual lambda deployments, and the CloudFormation docs don’t really cover it either. There was a fair bit of trial and error to get this part right, we knew we were on the right track and really close but maybe not always defining the right field in the right section of the JSON files. Hopefully this post saves someone the time we spent trying to figure that part out.
3. For the initial deployments we would encounter variations on the same theme of error, namely invalid lambda triggers, or confusing lambda already exists. This drove us mad and we can’t honestly say that we’ve figured it out entirely.
4. By having to manually create the lambda at the start to get the ARN for the skill creation, this caused an error on first deployment because the functions already existed… so naturally we deleted them, they were empty after all. On the next deployment, it complained that it didn’t exist, so we created one and added the skill verification to the trigger (mistake). On the next deployment it complained about the lambda triggers, so we deleted the triggers, then after several trials and errors, it eventually worked without complaint, and continues to work without complaint.
5. Getting that initial deploy was cumbersome and we couldn’t see a consistent behaviour that caused an error. Obviously there must be a correct default initial setting but we struggled with this. Sometimes it would complain only about one region but not the other, sometimes it complained about functions already existing, sometimes because they didn’t exist — it never seemed to make sense. We worked out that creating the Lambda with Skills Kit trigger verification disabled solves this issue but we’re open to suggestions and ideas as to how to avoid this for future projects.
6. Catalogue ingestion is one we also feel they’re working on as we would upload the radio station catalogues and it took a while for them to get ingested. In the end it turns out that Amazon has to be explicitly informed to allow ingestion otherwise it could sit there for days.
7. There were times when we felt like skills manifest parsing errors were not reported properly. We would get nondescript messages like Manifest failed when a little more detail like what attributes were invalid or required might have made bugs easier to track down and resolve. Our hope is that in the future these issues will be addressed as the Alexa Skills Kit developer tools mature.