EXPEDIA GROUP TECHNOLOGY — SOFTWARE
Development and Testing of AWS without AWS: Localstack
Localstack provides an AWS-like environment that can be used for development and testing without needing AWS available
AWS provides a lot of useful tools and services for developers: Serverless, Lambda functions, SNS queue, DynamoDB, S3 storage. When we start developing something connected to such services, obviously we can use one of the test environments provided by Amazon (LAB). Nonetheless, there is also a “local way” to go: Localstack.
Reduce costs and save time!
Localstack provides an AWS-like environment that can be used for development and/or testing purposes.
Will your service use an SNS queue? Then, you can start the development with LocalStack.
Will your app need information by a Lambda? Then, you can develop it using LocalStack.
Will your frontend need S3 resources? Then, you can use LocalStack S3.
How doest it work?
LocalStack gets started inside a Docker container and it contains a lot of the Cloud APIs of AWS.
Installing
The easiest way to install LocalStack is via pip (I know…that’s for Mac, but there must also be easy ways for other platforms):
pip install localstack
Running in Docker
You can run LocalStack with a command
localstack start
or with a docker compose yml file
docker-compose up
An example of compose yml file is
version: '2.1'
services:
...
localstack:
image: localstack/localstack
ports:
- "4567-4599:4567-4599"
- "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
environment:
- SERVICES=${SERVICES- }
- DEBUG=${DEBUG- }
- DATA_DIR=${DATA_DIR- }
- PORT_WEB_UI=${PORT_WEB_UI- }
- LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
- KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "${TMPDIR:-/tmp/localstack}:/tmp/localstack"How did we use it?
We needed to integrate a new service to an already existing system that automatically pushes messages to a Slack channel using:
- SNS queue
- Lambda function
- Dynamo DB
- Nodejs App
SNS queue: a message queue that receives all the triggers
Lambda: serverless function provided by AWS
DynamoDb: a database that stores needed configurations
NodeJs App: an app with all the required logic
To test and verify the behaviour of the whole system, we set up Localstack to act as the needed AWS services, so we added a YML file that spun up what we wanted:
version: '2.1'
services:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
image: localstack/localstack
ports:
- "4567-4597:4567-4597"
- "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
environment:
- SERVICES=lambda,dynamodb
- DATA_DIR=${DATA_DIR- }
- DEBUG=1
- DEFAULT_REGION=us-west-2
- PORT_WEB_UI=${PORT_WEB_UI- }
- LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
- KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"As you can see, we set up the services Lambda and Dynamodb (we didn’t add the SNS topic because it was of less interest)
- SERVICES=lambda,dynamodb
After the setup we needed to import and configure data and Lambda for our needs. To do this, we inserted a bash script in the repo that:
1. Creates the lambda package
gulp zip
2. Creates a dynamodb table and inserts data in it
awslocal dynamodb create-table --table-name MyTableName --attribute-definitions
AttributeName=slackChannel,AttributeType=S --key-schema AttributeName=slackChannel,KeyType=HASH --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5awslocal dynamodb put-item --table-name MyTableName --item '{"somekey":{"somesubkey":{"key":{"S":"value"}}},"somethingelse":{"S":"anothervalue"}}'
3. Deploys the lambda function
awslocal lambda create-function --function-name myLambda--handler index.handler --environment '{"Variables":{"var1":"val1","var2":"val2"}}' --runtime nodejs10.x --role whatever --zip-file fileb://build/myLambda.zipN.B.
awslocal is a command but basically it’s an alias foraws --endpoint-url=http://localhost:4568it acts as the aws command, but locally. As you can see, you can also specify the runtime for the Lambda, in this case nodejs10.x.
All this setup was in a separate bash script file that, after localstack is up and running, populates it with the required function and data.
This project took me less than I expected — and I admit to be very cautious when it comes to complexity estimation.
Localstack is a very powerful tool, and when you need to test and develop something that requires AWS resources, it can definitely save your time (and some money as well, to be honest!)
During this journey we have learnt about some of the AWS capabilities, localstack, lambda, dynamodb and testing.
Ref.
Localstack repo: https://github.com/localstack/localstack
Thanks for reading
Miro Barsocchi: Software tester and also Electronic engineer, radio speaker, actor, surfer, barman, but only two of these are seriously. You can find me on Twitter or Github or elsewhere.
