Flyway in Serverless Architectures

The Challenge

Flyway is a great tool that provides version control for your database schema. One problem we’ve experienced recently is that, with the increasing popularity of Serverless environments, there isn’t always an obvious way to run Flyway as part of your Continuous Integration / Continuous Delivery (CI/CD) pipeline.

Flyway needs access to your database, which should not be publicly accessible. But Flyway needs to run from a location with Database access. Traditionally this could be achieved by having a permanent bastion instance as part of your infrastructure. In a Serverless environment, it is likely that your stack has no fixed instances with DB access. Furthermore, bastion instances introduce an unnecessary security risk and financial cost.

The Solution

We’ve developed a solution for this in the form of an AWS Lambda function that can be invoked to perform the Flyway migration against your Database. The project can be found on Lydtech’s Github. The diagram below outlines how the solution works.

Figure 1

Your CD pipeline will control Flyway execution via a Lambda. It will use an S3 bucket to upload the Flyway scripts. The general flow as depicted in figure 1 is:

1. CI / CD pipeline uploads Flyway artifacts (SQL / Java files) to an S3 bucket
2. CI / CD pipeline invokes Lambda
3. Lambda retrieves files from S3 bucket
4. DB user password is retrieved from Secrets Manager
5. Lambda executes S3 bucket against RDS instance

The result is that once the required components have been provisioned (details below). Your CI / CD pipeline can just do the following to achieve points 1 and 2 above and run a flyway migrate. Consider that your flyway scripts are in ./flyway/sql, your S3 bucket is myapp-flyway-migrations and your lambda is named myapp-flyway-migration:

# Synchronise s3 bucket with sql directory (--delete deletes any files on the target which no longer exist)
aws s3 sync ./flyway/sql s3://myapp-flyway-migrations --delete
# Invoke flyway lambda, passing 2 parameters
aws lambda invoke \
    --function-name myapp-flyway-migration out \
    --log-type Tail \
    --payload '{ "bucket_name": "myapp-flyway-migrations", "secret_name": "myapp_db_creds" }' \
    --cli-binary-format raw-in-base64-out

The Details

In order to use this approach, the following setup steps need to be performed before being able to invoke the Lambda:

1. S3 bucket setup
2. Create secret for DB password
3. Create IAM Lambda execution role
4. Deploy the Lambda
5. Create IAM user to invoke Lambda

The remainder of this article details each of these steps. Note: it assumes familiarity with AWS.

1. S3 bucket setup

With your AWS credentials set as environment variables, run the following:

#create bucket
aws s3api create-bucket \
    --bucket myapp-flyway-migrations \
    --region eu-west-2 \
    --create-bucket-configuration LocationConstraint=eu-west-2
#block all public access
aws s3api put-public-access-block \
    --bucket myapp-flyway-migrations \
    --public-access-block-configuration  "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

2. Create secret for DB password

aws secretsmanager create-secret \
    --name myapp_db_creds \
    --secret-string '{"db_password": "myDbPassword"}'

3. Create IAM Lambda execution role

Create an execution role for the Lambda, following the principle of least privilege this role will only have access to the S3 bucket and the secret manager entry:

#Create role
aws iam create-role --role-name myapp-flyway-lambda \
    --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
# Attach required managed policies
aws iam attach-role-policy \
    --role-name myapp-flyway-lambda \
    --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam attach-role-policy \
    --role-name myapp-flyway-lambda \
    --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
# Allow lambda to access s3 bucket
aws iam put-role-policy --role-name myapp-flyway-lambda \
    --policy-name allow-s3-access --policy-document \
'{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::myapp-flyway-migrations",
                "arn:aws:s3:::myapp-flyway-migrations/*"
            ]
        }
    ]
}
'
# Allow lambda to access db password secret
aws iam put-role-policy --role-name myapp-flyway-lambda \
    --policy-name allow-secret-access --policy-document \
'{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Resource": [
               "{db_password_secret_arn}"
            ]
        }
    ]
}
'

Note: The db_password_secret_arn will be provided by step 2.

4. Deploy the Lambda

Download the code artifact from Github:

curl -L -o flyway-lambda.jar \
https://github.com/lydtechconsulting/flyway-lambda/releases/latest/download/flyway-lambda-jar-with-dependencies.jar

Deploy the lambda to your AWS account
Set the following variables as per the table below

PlaceholderDescriptionExamplesubnet-idSubnet to deploy lambda to. Must be able to access the DB.subnet-42f69738security-group-idSecurity group to place the Lambda in. Must be able to access the DB.sg-4994912flambda-execution-role-arnobtained from step 3arn:aws:iam::123456789:role/myapp-flyway-lambdadb-urlJDBC URL to the databasejdbc:postgresql://myapp-prod-db.c7c5nbuwkxpy.eu-west-2.rds.amazonaws.com/myappflyway-userUsername to connect to DBpostgresschemasName of schema that Flyway will managemyapp

# Set variables
SUBNET_ID={subnet-id}
SEC_GROUP={security-group-id}
LAMBDA_EXECUTION_ROLE_ARN={lambda-execution-role-arn}
DB_URL={db-url}
FLYWAY_USER={flyway-user}
FLYWAY_SCHEMAS={schemas}
# Create Lambda function
aws lambda create-function \
    --function-name myapp-flyway-migration \
    --runtime java11 \
    --zip-file fileb://flyway-lambda.jar \
    --handler "com.lydtechconsulting.flywaylambda.FlywayHandler" \
    --timeout 120 \
    --memory-size 512 \
    --vpc-config SubnetIds=${SUBNET_ID},SecurityGroupIds=${SEC_GROUP} \
    --role ${LAMBDA_EXECUTION_ROLE_ARN} \
    --environment '{ "Variables": {"FLYWAY_URL":"'"${DB_URL}"'","FLYWAY_USER": "'"${FLYWAY_USER}"'","FLYWAY_SCHEMAS": "'"${FLYWAY_SCHEMAS}"'"}}'

5. Create IAM user to invoke Lambda

Create an IAM user to use from your CI server to execute the Lambda. The user will only be able to do 2 things; sync files to the S3 bucket and execute the lambda:

# Create user
aws iam create-user --user-name myapp-flyway-lambda-executor
# allow user to access s3 bucket
aws iam put-user-policy \
    --user-name myapp-flyway-lambda-executor \
    --policy-name flyway-s3-access --policy-document \
'{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::myapp-flyway-migrations",
                "arn:aws:s3:::myapp-flyway-migrations/*"
            ]
        }
    ]
}
'
# allow user to invoke lambda
aws iam put-user-policy \
    --user-name myapp-flyway-lambda-executor \
    --policy-name flyway-lambda-access --policy-document \
'{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:InvokeAsync"
            ],
            "Resource": [
                "arn:aws:lambda:eu-west-2:544193841373:function:myapp-flyway-migration"
            ]
        }
    ]
}
'
                    
# Create access key and secret for user
aws iam create-access-key --user-name myapp-flyway-lambda-executor

Executing the lambda

Now you can configure your chosen CI server to execute the lambda as per the commands shown above. You will need to use your chosen CI provider’s recommended secret management mechanism to store your AWS key and secret and access it in your pipeline.

Conclusion

This article has demonstrated a simple, secure and cost-effective method for running your Flyway migrations in a serverless environment.

Want to read more?

Head over to Lydtech Consulting to read this article and many others on Kafka and other interesting areas of software development.