Flyway in Serverless Architectures
The Challenge
Flyway is a great tool that provides version control for your database schema. One problem we’ve experienced recently is that, with the increasing popularity of Serverless environments, there isn’t always an obvious way to run Flyway as part of your Continuous Integration / Continuous Delivery (CI/CD) pipeline.
Flyway needs access to your database, which should not be publicly accessible. But Flyway needs to run from a location with Database access. Traditionally this could be achieved by having a permanent bastion instance as part of your infrastructure. In a Serverless environment, it is likely that your stack has no fixed instances with DB access. Furthermore, bastion instances introduce an unnecessary security risk and financial cost.
The Solution
We’ve developed a solution for this in the form of an AWS Lambda function that can be invoked to perform the Flyway migration against your Database. The project can be found on Lydtech’s Github. The diagram below outlines how the solution works.
Figure 1
Your CD pipeline will control Flyway execution via a Lambda. It will use an S3 bucket to upload the Flyway scripts. The general flow as depicted in figure 1 is:
- CI / CD pipeline uploads Flyway artifacts (SQL / Java files) to an S3 bucket
- CI / CD pipeline invokes Lambda
- Lambda retrieves files from S3 bucket
- DB user password is retrieved from Secrets Manager
- Lambda executes S3 bucket against RDS instance
The result is that once the required components have been provisioned (details below). Your CI / CD pipeline can just do the following to achieve points 1 and 2 above and run a flyway migrate. Consider that your flyway scripts are in ./flyway/sql
, your S3 bucket is myapp-flyway-migrations
and your lambda is named myapp-flyway-migration
:
# Synchronise s3 bucket with sql directory (--delete deletes any files on the target which no longer exist)
aws s3 sync ./flyway/sql s3://myapp-flyway-migrations --delete# Invoke flyway lambda, passing 2 parameters
aws lambda invoke \
--function-name myapp-flyway-migration out \
--log-type Tail \
--payload '{ "bucket_name": "myapp-flyway-migrations", "secret_name": "myapp_db_creds" }' \
--cli-binary-format raw-in-base64-out
The Details
In order to use this approach, the following setup steps need to be performed before being able to invoke the Lambda:
- S3 bucket setup
- Create secret for DB password
- Create IAM Lambda execution role
- Deploy the Lambda
- Create IAM user to invoke Lambda
The remainder of this article details each of these steps. Note: it assumes familiarity with AWS.
1. S3 bucket setup
With your AWS credentials set as environment variables, run the following:
#create bucket
aws s3api create-bucket \
--bucket myapp-flyway-migrations \
--region eu-west-2 \
--create-bucket-configuration LocationConstraint=eu-west-2#block all public access
aws s3api put-public-access-block \
--bucket myapp-flyway-migrations \
--public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
2. Create secret for DB password
aws secretsmanager create-secret \
--name myapp_db_creds \
--secret-string '{"db_password": "myDbPassword"}'
3. Create IAM Lambda execution role
Create an execution role for the Lambda, following the principle of least privilege this role will only have access to the S3 bucket and the secret manager entry:
#Create role
aws iam create-role --role-name myapp-flyway-lambda \
--assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'# Attach required managed policies
aws iam attach-role-policy \
--role-name myapp-flyway-lambda \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRoleaws iam attach-role-policy \
--role-name myapp-flyway-lambda \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole# Allow lambda to access s3 bucket
aws iam put-role-policy --role-name myapp-flyway-lambda \
--policy-name allow-s3-access --policy-document \
'{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::myapp-flyway-migrations",
"arn:aws:s3:::myapp-flyway-migrations/*"
]
}
]
}
'# Allow lambda to access db password secret
aws iam put-role-policy --role-name myapp-flyway-lambda \
--policy-name allow-secret-access --policy-document \
'{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue"
],
"Resource": [
"{db_password_secret_arn}"
]
}
]
}
'
Note: The db_password_secret_arn will be provided by step 2.
4. Deploy the Lambda
Download the code artifact from Github:
curl -L -o flyway-lambda.jar \
https://github.com/lydtechconsulting/flyway-lambda/releases/latest/download/flyway-lambda-jar-with-dependencies.jar
Deploy the lambda to your AWS account
Set the following variables as per the table below
PlaceholderDescriptionExamplesubnet-id
Subnet to deploy lambda to. Must be able to access the DB.subnet-42f69738security-group-id
Security group to place the Lambda in. Must be able to access the DB.sg-4994912flambda-execution-role-arn
obtained from step 3arn:aws:iam::123456789:role/myapp-flyway-lambdadb-url
JDBC URL to the databasejdbc:postgresql://myapp-prod-db.c7c5nbuwkxpy.eu-west-2.rds.amazonaws.com/myappflyway-user
Username to connect to DBpostgresschemas
Name of schema that Flyway will managemyapp
# Set variables
SUBNET_ID={subnet-id}
SEC_GROUP={security-group-id}
LAMBDA_EXECUTION_ROLE_ARN={lambda-execution-role-arn}
DB_URL={db-url}
FLYWAY_USER={flyway-user}
FLYWAY_SCHEMAS={schemas}# Create Lambda function
aws lambda create-function \
--function-name myapp-flyway-migration \
--runtime java11 \
--zip-file fileb://flyway-lambda.jar \
--handler "com.lydtechconsulting.flywaylambda.FlywayHandler" \
--timeout 120 \
--memory-size 512 \
--vpc-config SubnetIds=${SUBNET_ID},SecurityGroupIds=${SEC_GROUP} \
--role ${LAMBDA_EXECUTION_ROLE_ARN} \
--environment '{ "Variables": {"FLYWAY_URL":"'"${DB_URL}"'","FLYWAY_USER": "'"${FLYWAY_USER}"'","FLYWAY_SCHEMAS": "'"${FLYWAY_SCHEMAS}"'"}}'
5. Create IAM user to invoke Lambda
Create an IAM user to use from your CI server to execute the Lambda. The user will only be able to do 2 things; sync files to the S3 bucket and execute the lambda:
# Create user
aws iam create-user --user-name myapp-flyway-lambda-executor# allow user to access s3 bucket
aws iam put-user-policy \
--user-name myapp-flyway-lambda-executor \
--policy-name flyway-s3-access --policy-document \
'{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::myapp-flyway-migrations",
"arn:aws:s3:::myapp-flyway-migrations/*"
]
}
]
}
'# allow user to invoke lambda
aws iam put-user-policy \
--user-name myapp-flyway-lambda-executor \
--policy-name flyway-lambda-access --policy-document \
'{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction",
"lambda:InvokeAsync"
],
"Resource": [
"arn:aws:lambda:eu-west-2:544193841373:function:myapp-flyway-migration"
]
}
]
}'
# Create access key and secret for user
aws iam create-access-key --user-name myapp-flyway-lambda-executor
Executing the lambda
Now you can configure your chosen CI server to execute the lambda as per the commands shown above. You will need to use your chosen CI provider’s recommended secret management mechanism to store your AWS key and secret and access it in your pipeline.
Conclusion
This article has demonstrated a simple, secure and cost-effective method for running your Flyway migrations in a serverless environment.
Want to read more?
Head over to Lydtech Consulting to read this article and many others on Kafka and other interesting areas of software development.