Lambda, a level up (part 2): Automation
In this occasion, I’d like to share a couple of ways to enhance the deployment of your Lambda functions, aiming for its complete automation:
- Handling tasks not achievable with CloudFormation.
- Use of CF resources in your code via environment variables.
Is Cloud Formation enough?
I guess this is the first question to ask when deploying Lambda functions via Serverless.
Through the Serverless configuration system, which in its most basic form consists of a single serverless.yml file, we are able to deploy our whole stack. Essentially, the configuration of our serverless project will work as a wrap of a CloudFormation template that will be used to deploy our functions and all the resources we need:
The Serverless Framework translates all syntax in serverless.yml to a single AWS CloudFormation template. By depending on CloudFormation for deployments, users of the Serverless Framework get the safety and reliability of CloudFormation.
Reference: https://serverless.com/framework/docs/providers/aws/guide/deploying/
That’s great but we still have to face a couple of issues when trying to deploy our projects:
- Sometimes, as part of our deployment process, we need to carry out some tasks that are independent of AWS e.g: we need to create a webhook in a third party service that will request our function through an API Gateway.
- Or, at times, there are AWS related tasks that can’t be executed via CloudFormation templates (or at least entirely). We’ll procede with an example.
Serverless plugin scripts
To solve the problem of having to deal with these tedious tasks, we’ll use the plugin Serverless plugin scripts.
This plugin will allow you to hook one of the lifecycle events belonging to the different framework commands to run custom scripts. This tool will exponentially elevate the capabilities of your automated actions.
In our case, we’ll locally invoke a Lambda function to create a Cognito user with a permanent password (the user that would request the AppSync API of our first article) when deploying our project via sls deploy.
We just need to include what event we want to hook and the script to run in our serverless.yml
plugins:
- serverless-plugin-scripts
...custom:
scripts:
hooks:
'deploy:finalize': sls invoke -f create-our-user
This approach will have an additional advantage, we can make use of the same environment variables defined in our project configuration to create the user and subsequently use it in other functions:
import {
APIGatewayProxyEvent,
APIGatewayProxyResult,
Context,
Handler
} from "aws-lambda";import OurUserFactory from "./OurUserFactory.example";export const entrypoint: Handler = async (
event: APIGatewayProxyEvent,
context: Context
): Promise => {
const {
cognitoUserName,
cognitoEmail,
cognitoPassword
} = process.env; console.log("Ensuring Cognito Admin user...");
// FIRST, we should figure out if the user already exists
// and in that case, skipping a redundant step.
const userExists = await OurUserFactory.checkUserExists(cognitoEmail); // If not, let's create the user...
if (!userExists) {
try {
console.log("User not found, creating...");
const userDetails = await await OurUserFactory.createUser(cognitoUserName, cognitoEmail, cognitoPassword); console.log(`User created successfully with email: ${cognitoEmail}`); return {
statusCode: 200,
body: JSON.stringify(userDetails)
};
...
Handler create-our-user.ts
import * as AWS from "aws-sdk";import { credentials } from "./aws-exports";/**
* OurUserFactory class.
*/
export default class OurUserFactory { /**
* Checks if a user already exists.
*
* @param email - They user email.
*
* @returns - TRUE if the user already exists in the Pool.
*/
public static checkUserExists(email: string): Promise {
...
} /**
* Create a Cognito user with permanent password.
*
* @param username - The username.
* @param email - The user email.
* @param password - The user password.
*
* @returns - The user details.
*/
public static createUser(username: string, email: string, password: string):
Promise {
return new Promise((resolve, reject) => {
const {
cognitoClientId,
cognitoPoolId,
} = process.env; // Create the user.
let params: AWS.CognitoIdentityServiceProvider.AdminCreateUserRequest = {
UserPoolId: cognitoPoolId,
Username: username,
// Do not send welcome email. MessageAction: "SUPPRESS",
TemporaryPassword: password,
UserAttributes: [
{ Name: "email", Value: email },
// Mark email as already verified.
{ Name: "email_verified", Value: "true" },
{ Name: "custom:role", Value: "admin" }
]
}; const awsCognito = new AWS.CognitoIdentityServiceProvider(credentials);
awsCognito.adminCreateUser(params, (error, userData) => {
if (error) {
reject(error);
return;
} // For security reasons, the first password is always considered
// as temporary and can't be used for auth.
// Let's set the permanent password...
let params: AWS.CognitoIdentityServiceProvider.AdminInitiateAuthRequest = {
AuthFlow: "ADMIN_NO_SRP_AUTH",
ClientId: cognitoClientId,
UserPoolId: cognitoPoolId,
AuthParameters: {
USERNAME: username,
PASSWORD: password
}
};
awsCognito.adminInitiateAuth(params, (error, data) => {
if (error) {
reject(error);
return;
} let params = {
ChallengeName: "NEW_PASSWORD_REQUIRED",
ClientId: cognitoClientId,
UserPoolId: cognitoPoolId,
ChallengeResponses: {
USERNAME: username,
NEW_PASSWORD: password,
},
Session: data.Session
};
awsCognito.adminRespondToAuthChallenge(params, (error, data) => {
if (error) {
reject(error);
return;
} resolve(userData.User);
});
});
})
})
}
}
Class OurUserFactory.example.ts
Next time we deploy we’ll be able to see our user is created, saving us any manual step!
functions:
create-our-user: OurStack-ourstage-create-our-userServerless: Removing old service artifacts from S3...
{
"data": {
"Username": "xxxx-xxx-xxxx-xxxx",
"Attributes": [
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:role",
"Value": "our-role"
},
{
"Name": "email",
"Value": "our-email@gravitywell.co.uk"
}
],
"UserCreateDate": "2019-04-05T09:23:42.944Z",
"UserLastModifiedDate": "2019-04-05T09:23:42.944Z",
"Enabled": true,
"UserStatus": ...
}
}
Serverless resources in your Lambda functions
Now, that we can control complex steps in our deployments, let’s get back to CloudFormation and the relationship between our created resources and its utilisation in Lambda functions.
A typical scenario when making use of the Serverless framework could be presented as:
- Create a resource with CloudFormation via serverless.yml
- Use the resource, generally through the AWS SDK, in your functions code. A standard approach would be accessing the desired resource properties, like the ARN, through environment variables.
Problems
- Serverless doesn’t have (yet) a system to reference our created resources and be assigned to environment variables.
- Though in some places is advised you could use Ref, it looks like there is currently a bug when referencing CF resources and functions in a development environment in Serverless. Besides, sometimes, we’d need to access to properties other than the resource ARN, eg. the URL of the Graphql endpoint.
- We’d like to save any manual step involving the deploy of the stack to manually set our enviroment variables with the resource properties through some copy/pasting.
CloudFormation Outputs
Serverless provides an alternative solution for these problems allowing to reference CloudFormation Outputs in your configuration files and, this is the most interesting part, assigning them to environment variables that can be used in your code.
CloudFormation Outputs is a way to store data derived from the resources created by your template in the stack. This data will be available as a variable to be used from external sources or be referenced from a different CF stack, which is called cross stack references.
So, in our project configuration file, after the Resources section, we’ll add the outputs we need to use in our code. Following our first example, where we needed to work with a Cognito Client, we’ll create a variable containing the id of the App Client:
Outputs:
OurCognitoClientId:
Value:
Ref: OurBackendPoolClient
Export:
Name: OurCognitoClientId-${self:provider.stage}Ok, now we can set one of the environment variables to retrieve the value in one of our functions:
provider:
name: aws
runtime: nodejs8.10
region: eu-west-1
...
environment:
cognitoClientId:
${cf:${self:service}-${self:provider.stage}.OurCognitoClientId}serverless.yml
import * as AWS from "aws-sdk";
import { credentials } from "./aws-exports";/**
* Gets a Cognito user access token using ADMIN_NO_SRP_AUTH auth.
*
* @param awsConfig - Aws stack info.
*
* @returns - The access token.
*/
export const getCognitoAdminToken = (awsConfig: {
cognitoClientId: string,
cognitoPoolId: string,
cognitoUserName: string,
cognitoPassword: string
}): Promise => {
return new Promise((resolve, reject) => {
const {
cognitoClientId,
cognitoPoolId,
cognitoUserName,
cognitoPassword
} = awsConfig; const authRequestParams: AWS.CognitoIdentityServiceProvider.AdminInitiateAuthRequest = {
AuthFlow: "ADMIN_NO_SRP_AUTH",
ClientId: cognitoClientId,
UserPoolId: cognitoPoolId,
AuthParameters: {
USERNAME: cognitoUserName,
PASSWORD: cognitoPassword,
}
};
Extracted code from our first article
But… oh wait, something wrong is going on:
Serverless Error --------------------------------------- Trying to request a non exported variable from CloudFormation. Stack name: "MyStackName-stage" Requested variable: "OurCognitoClientId".
This is the kind of message returned by Serverless when you haven’t deployed your resources yet or add a new Output. And it makes sense… Serverless will try to resolve CF variables but will find that the value doesn’t exist in the stack yet.
Multiple environment configuration files
A normal practice when working on projects that rely on external services for development (as it is normally the case with AWS) is splitting up your environment variables and other settings into 4 different environments: dev, staging and production which would reference the different stages of your external services and local. The local environment can be used as a way to mock those external services or, for example, work offline.
In our case, we’ll create of a separate cli option — our-build local when wanting to make use of our local variables instead of including it as another stage environment. This will allow us to use the stage option (needed for some actions like deployment) and set the environment variables with our local configuration when dealing with Serverless:
Extracting environment variables into separate files will stop Serverless from trying to resolve CF Outputs not created yet. We’ll use the syntax ${opt:our-build, self:provider.stage} to load the local configuration file, if the option — our-build local is detected, or the corresponding to the stage otherwise.
provider:
name: aws
runtime: nodejs8.10
...
environment:
...
cognitoClientId:
${self:custom.env.variables.cognitoClientId}
...
custom:
# Load env variables.
env: ${file(./env/${opt:our-build, self:provider.stage}.yml)}aserverless.yml
Using our local settings:
sls offline start --our-build localvariables:
cognitoClientId: XyyXXXXXyyyXXXXyyyyXXXy
local.yml
With this approach, we’ll be able to carry out the first deployment of our stack without Serverless complaining about referencing non existing resources with:
sls deploy --stage development --our-build localOnce the stack is created (and our Outputs) we can make use of its resources, if we need it, in our functions starting sls as normal:
sls start --stage developmentvariables:
cognitoClientId: ${cf:${self:service}-${self:provider.stage}.OurCognitoClientId}
development.yml
To ensure your stage environment use the created Output values, you just need to deploy getting rid of the local flag:
sls deploy --stage developmentConclusion
Hopefully, these solutions can improve the automation of some of your tasks and be integrated as part of your CI/CD processes.
This is the last of a series of articles that started with Operating with AppSync from AWS Lambda, which raised some questions about possible improvements and resulted, firstly, in Lambda, a level up (1/2): Performance.
Originally published at: https://www.gravitywell.co.uk/latest/aws/posts/lambda-a-level-up-part-2-automation/
