How to add k6 load test to Gitlab CI pipeline with AWS Lambda Containers
A great way to ensure both consistency and accuracy is to automate everything the team does
Pipelines are so important to Modanisa team. We are automating the things that can be automated. Imagine a pipeline including memory leak tests, CSS-regression tests and load tests. This tutorial makes you implement a load test job in your pipeline.
What we are going to do? Here is the simple schema of the tutorial. As you guys can see there are two-part. One of them is load testing to the target and the other part is visualize of outputs but influxdb and Grafana setups are not included because easily can find tutorials on google.
¿? Why is Lambda?
It’s Serverless. Decide what to do with your code and let AWS make it server-side with minimized cost.
¿? Why influxdb? Actually, what is influxdb?
InfluxDB is an open-source time series database (TSDB) developed by the company InfluxData. It is written in the Go programming language for storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics.
¿? Why kaniko?
Kaniko is a tool to build container images from a Dockerfile inside a container or Kubernetes cluster. Kaniko doesn’t depend on a Docker daemon and executes each command within a Dockerfile completely in user space. This enables building container images in environments that can’t easily or securely run a Docker daemon such as a standard Kubernetes cluster.
⚡️ Let’s start ️⚡️
1) Lambda container Dockerfile 🐳
It’s a simple Dockerfile including k6 binaries but last line can be interesting. Here is the explanation.
You can run that Dockerfile on your local. But if you run it as a docker container, it doesn’t run that exported function. There is a way to trigger exported function below with payload.
curl -XPOST "http://localhost:8080/2015-03-31/functions/function/invocations" -d '{"payload":"hello world!"}'
2 ) The function inside Lambda container
If you want to start testing with a test configuration, should write k6 run spec.file.js to console and all is done. k6 will initialize itself and will run the test spec file and then shows the result of the test.
In our case, we have k6 binaries on docker container. We can send commands like k6 run with spawn or exec. These are built-in node js packages.
The function has to wait until the completion of the execution k6 process. It is important because the lambda container lives until the function is complete. If the container dies the load test cannot be executed successfully.
Lambda functions take event parameters so the function of inside lambda container takes event parameters too. In code snippet, we have an event parameter and it has targetServiceName field to specify the test spec file to execute. Also, we can pass influxdb endpoint differently for every environment.
k6 shows the result when the test is executed successfully. These lines also include if your thresholds fail or not, kind of assertion like is response time under 100ms or not? You can find more information on k6 Thresholds docs page.
Also, I want to show folder structure to understand last line of Dockerfile and clarify picking test spec file with the event.
3) AWS ECR(Elastic Container Registry) Repo Create
Navigate to ECR Dashboard > Private Tab > Create Repository > Pick a name > Create Repository
Now we have a Container Registry URI. That will be used for hosting our containers and accessing from Lamba. We will paste it to Gitlab CI file which is mentioned below.
We just created a repo without any image. We need a published image because we will pick in lambda create wizard. Don’t worry it’s pretty easy.
Click newly created ECR repo > View Push Commands
This helps us to publish an image to our ECR repo easily. We need to push the previously created Dockerfile to ECR.
4) AWS Lambda Create
Create Function > Select Container Image > Pick a name > Pick your image > Create Function
These steps make you create a Lambda with a container image. The last step is attaching vpc for accessing the internet.
Select the Lambda Function newly created > Configuration Tab > VPC > Edit
Lambda needs to access internet access in some scenarios but some scenarios don’t need to access. In that case, obviously, it needs.
This wizard helps us to pick vpc, subnet and security group. If you don’t know these keywords, feel to free just default vpc, two subnet and default security group.
Note that lambda function name for Gitlab CI file.
5) Gitlab-CI file
There are 3 stages with an if statement. If someone commits to the load test repo package and lambdaDeploy will be executed by Gitlab CI. If another repo trigger that pipeline(as known Downstream keyword) with variable TARGET_SERVICE_NAME then only the load test job will be executed.
By the way, this pipeline can work without if statements for a single repository. But imagine you have two services and one frontend repository. Easily trigger every commit the job without build and package job wait time and less code.
Then add the load test snippet to the target repository for example it can be your service project’s pipeline.
There it is. We added a load test job to our Gitlab CI pipeline.
💈 Visualize 💈
Let’s add our influxdb to grafana as data source and then import a ready-to-use k6 dashboard to our grafana.
Grafana > Configuration > Data Sources > Add data source > Pick InfluxDB
This page needs your influxdb’s URL, user, password and db name. The database name at image “k6db” was used at k6 run parameters above before.
The last step is importing the dashboard.
Grafana > Create > Import
💰 Pricing 💰
Heavily under development scenario with non-included free tier on Ireland region;
300 pipeline in a month > 10 min > 2 GB Ram = 6.14 USD
150 pipeline in a month > 10 min > 4 GB Ram = 6.00 USD
300 pipeline in a month > 5 min > 1 GB Ram = 1.50 USD
150 pipeline in a month > 5 min > 1 GB Ram = 0.75 USD
⚔️ Tests and Constraints ⚔️
There are two important constraints. Durations and Virtual User (VU’s) count.
Durations
AWS CLI lambda invoke command waits until lambda executions end so in these cases, can be 10 min. You will see a HTTP connection when you run with debugging --debug parameter. If this connection is long enough so it dies. What Amazon docs say;
For functions with a long timeout, your client might be disconnected during synchronous invocation while it waits for a response. Configure your HTTP client, SDK, firewall, proxy, or operating system to allow for long connections with timeout or keep-alive settings. RefTests run on our Kubernetes executor for GitLab Runner with default configuration about network connections. So lambda invoke command dies after 5 minutes but it works for 5 minutes(300 seconds). If we consider using it on the pipeline that durations are pretty cool.
Solution
Async invocation can be helpful. CLI doesn’t wait for a response from executions. In that case, we can’t break the pipeline with threshold values because we don't have the response of k6 output. But we know the function name of lambda, we can try to dig the logs as like mentioned command then it returns JSON documents so we can use jq tool to parse.
aws logs filter-log-events --log-group-name /aws/lambda/load-test
--start-time 1641500010000Another constraint is Virtual User count. Lambda supports max up to 1000 VU’s. It has some trouble handling even 1100 VU’s.
⚖️ Results ⚖️
+------+--------------+----------+--------------+-----------+
| RAM | Virtual User | Duration | Project | Req count |
+------+--------------+----------+--------------+-----------+
| 1 GB | 1000 VU | 5 min | Go backend | 150k-180k |
+------+--------------+----------+--------------+-----------+
| 1 GB | 1000 VU | 5 min | Vue frontend | 30k |
+------+--------------+----------+--------------+-----------+
| 1 GB | 4000 VU | 5 min | Go backend | 400k |
+------+--------------+----------+--------------+-----------+
| 6 GB | 1000 VU | 5 min | Go backend | 150k-180k |
+------+--------------+----------+--------------+-----------+
| 1 GB | 1000 VU | 10 min | Go backend | 270k |
+------+--------------+----------+--------------+-----------+Simple tests will use ~1–5MB per VU. (1000VUs = 1–5GB). Tests that are using file uploads can consume tens of megabytes per VU. ref
Also, I tried to understand how much GB is enough for a load test. 1 GB and 6 GB have some request count in our case.
Configuration with 6 GB:
REPORT RequestId: 50b366c9–xxxx–xxxx-xxxx-a332d24b19a5 Duration: 303338.75 ms Billed Duration: 304099 ms Memory Size: 6000 MB Max Memory Used: 732 MB Init Duration: 760.19 msConfiguration with 1 GB;
REPORT RequestId: 3b4ad8f4–xxxx–xxxx-xxxx-e6422e2d4e94 Duration: 309184.01 ms Billed Duration: 309185 ms Memory Size: 1024 MB Max Memory Used: 705 MB
