“java.lang.OutOfMemoryError: Java heap space” in Karate Framework
In today’s technology, the vast majority of applications are served by backend services, so testing the backend is one of the important parts of your product quality lifecycle. You need to develop test cases for each endpoint, especially if your company is using a micro-service architecture.
As you might agree, backend tests should be easy to implement. Considering the popularity of the Karate automation testing framework, ease of developing cases and low coding requirements, we have chosen it as a platform for automation of our backend tests.
If you need to develop a significant amount of test cases and run them in a single run, you may have discovered that this framework has some problems and consumes excessive memory. You get a “java.lang.OutOfMemoryError: Java heap space” error.
This error occurs when the tests are over a certain limit (I can’t specify the exact number). For example, we have a test package with 203 scenarios that run fine, but the other package has 376 scenarios and it’s getting this error.
When we dive deeper into the investigation of an error, we see that the framework has a problem with cleaning the heap space. Even if we increase the heap size to the maximum, we’re still getting the error. We have run the test package which returns this error on an m5.2xlarge EC2 instance with the 8 CPUs and 32GiB memory. The results, unfortunately, were the same.
We have disabled all the logs, minimized the dependencies in pom.xml and monitored the memory usage of the tests again. We found that the CPU usage was around 350% while running only one scenario.
Imagine that you have already developed more than 3,000 test cases, but they aren’t working. What should you do? To solve the issue in a timely manner and support aggressive delivery timelines, we focused on finding a way to run existing tests rather than investing in the reimplementation of the same tests with another framework or even in another language.
How did we run those Karate tests?
We realized that the problem is running many suites together. So if we create an architecture that runs tests case by case, we might find our way around the issue.
Here is a running karate test command with maven.
mvn clean test -Dtest=TestRunner -DfailIfNoTests=false -DstoppingStep=null '-Dkarate.options=--tags @happyPath classpath:services/getirFood/AbortBasket' -f pom.xmlSo, we need a suite name and test case name to run a single test.
1. Services
- Saving the prepared scenario pairs for maven run.
@GetMapping("/karate/{process}/{runId}")
public KarateScenarioResponse setKarateScenario(@PathVariable String process, @PathVariable String runId) {
Karate_scenarios scenario = new Karate_scenarios();
Date now = DateUtils.addHours(new Date(), 3);
scenario.setScenario(process);
scenario.setStatus("ready");
scenario.setRunId(runId);
scenario.setCreatedDate(now);
if (karateScenariosRepository.findOneByScenario(process) == null) {
karateScenariosRepository.save(scenario);
return new KarateScenarioResponse(scenario);
} else {
return new KarateScenarioResponse(karateScenariosRepository.findOneByScenario(process).get_id(), runId, process, "Test already defined", now);
}
}- Getting a single available scenario
@GetMapping("/karate/available-process")
public KarateScenarioResponse getAvailableKarateScenario() {
lock.lock();
try {
Karate_scenarios karate_scenario = karateScenariosRepository.findOneByStatus("ready").get(0);
if (!karate_scenario.equals(null)) {
Date now = DateUtils.addHours(new Date(), 3);
karate_scenario.setStatus("in progress");
karate_scenario.setLastTestedDate(now);
karateScenariosRepository.save(karate_scenario);
lock.unlock();
return new KarateScenarioResponse(karate_scenario);
} else {
lock.unlock();
return new KarateScenarioResponse();
}
} catch (Exception e) {
lock.unlock();
return new KarateScenarioResponse();
}
}- Updating the scenario status
@GetMapping("/karate/{id}/status/update")
public KarateScenarioResponse updateKarateScenarioStatus(@PathVariable String id) {
Karate_scenarios karate_scenario = karateScenariosRepository.findOneBy_id(id);
if (!karate_scenario.equals(null)) {
karate_scenario.setStatus("done");
karateScenariosRepository.save(karate_scenario);
return new KarateScenarioResponse(karate_scenario);
} else {
return new KarateScenarioResponse();
}
}- The database should look like the below. As you can see, each task has a status. Tasks are in “ready” status when the scenario is prepared by the Jenkins job and ready to run. While they are running in containers, tasks switch to “in progress” status and after the test is finished, they go to “done”. The “runId” field is a task run identifier from Jenkins and it’s used in the folder the test reports in the end.
2. A cron job that prepares the scenarios via Jenkins
- PrepareKarateScenarios.groovy
def features = []
def findTags(feature) {
def folder = "${env.WORKSPACE}/src/test/java/services/getirFood"
def file = new File("${folder}/${feature}")
def lines = file as String[]
def tags = lines.findAll { it.trim().startsWith('@') }.collect {
it.replaceAll("\\s", "")
}
for (int z = 0; z < tags.size(); z++) {
String url = "https://bft.getir.test.com/prepare-scenario/karate/${feature},${tags[z]}/${env.BUILD_NUMBER}"
def sendScenarioToBft = new URL(url).getText()
println(sendScenarioToBft);
}
return tags
}
pipeline {
agent { label 'java-karate' }
stages {
stage('Prepare Scenarios') {
steps {
script {
final foundFiles = findFiles(glob: "src/test/java/services/getirFood/*.feature")
for (int i = 0; i < foundFiles.length; i++) {
def filename = foundFiles[i].name
features << filename
}
for (int y = 0; y < features.size(); y++) {
def process = findTags(features[y])
}
}
}
}
}
}This cron pulls the api-automation-test repo from git and sends the ${feature},${tags} info to the database as a single scenario that can run after analyzing the .feature karate files.
3. Runner script
- runner.py
import requests
import json
import subprocess
import os
def get_scenario():
url = "https://bft.getir.test.com/prepare-scenario/karate/available-process"
return json.loads(requests.request("GET", url).text.encode('utf8'))
def prepare_command(tag, feature):
command = "mvn clean test -Dtest=TestRunner -DfailIfNoTests=false -DstoppingStep=null '-Dkarate.options=--tags " + tag + " classpath:services/getirFood/clientApiGateway/" + feature + "' -f pom.xml"
return command
def run_cmd(command):
return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read()
def sync_project():
run_cmd("rm -rf api-automation-test")
clone = "git clone --shared --branch feature/QA-739 https://xxx:xxx@bitbucket.org/xxx/api-automation-test.git"
run_cmd(clone)
os.chdir("api-automation-test")
sync_project()
def sendReport(runId):
fileName = run_cmd("ls -R target/reports | grep '.*\.json'")
pwd = run_cmd("pwd")
files = {
('file',
(fileName.rstrip(),
open(pwd.rstrip() + '/target/reports/' + fileName.rstrip(), mode='r'),
'application/json'))}
url = "https://bft.getir.test.com/report/upload"
headers = {
'directory': runId
}
requests.request("POST", url, headers=headers, data={}, files=files)
while True:
response = get_scenario()
if response['status'] == "in progress":
tag = response['scenario'].split(",")[1]
feature = response['scenario'].split(",")[0]
command = prepare_command(tag, feature)
run_cmd(command)
sendReport(response['runId'])This python script pulls the repo and checks if there is an available single Karate scenario waiting to run and runs it with maven. Thus, if we put this script into a container, it will search for a single test to run until the container is stopped. We need a container that has JDK and executes python. Let’s have a look.
4. Dockerize
- Dockerfile
FROM maven:3-jdk-8-alpine
ENV PYTHONUNBUFFERED=1
RUN echo "**** install Python ****" && \
apk add --no-cache python && \
if [ ! -e /usr/bin/python ]; then ln -sf python /usr/bin/python ; fi && \
\
echo "**** install pip ****" && \
python -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip install --no-cache --upgrade pip setuptools wheel && \
if [ ! -e /usr/bin/pip ]; then ln -s pip /usr/bin/pip ; fi
RUN pip install requests
RUN apk update && \
apk upgrade && \
apk add git
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
RUN echo "PWD is: $PWD"
COPY Runner.py Runner.py
ENTRYPOINT ["python", "Runner.py"]We containerized our script which is okay, but where should the container be running? If we run this container, it will take an available test and run it and repeat the process for each test. It’s working but this is inefficient as you have to wait a long time if you have more than 3,000 test cases. We needed multiple containers so we could run test cases in parallel. We found the Portainer to be the best solution for us in means of scaling and orchestrating containers and it has an easy-to-use GUI.
5. Docker Scaling
- Deploying our stack to the Portainer
- We are now able to scale our container with the docker swarm. 500 containers ask for a single test and after running the test and in the end, they will send the report to S3.
6. Reporting script
- Reporter.py
import logging
import subprocess
import os
from boto.s3.connection import S3Connection
conn = S3Connection('xxx', 'xxx')
bucket = conn.get_bucket('xxx')
directory = '103/'
def prepare_report():
os.chdir("..")
command = "mvn verify -DskipTests"
return run_cmd(command)
def run_cmd(command):
return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read()
def download_jsons():
os.chdir("results")
for key in bucket.list(prefix=directory):
try:
res = key.get_contents_to_filename(key.name.replace(directory, ''))
except:
logging.info(key.name + ":" + "FAILED")
prepare_report()
download_jsons()This script downloads the JSON from the S3 bucket which we have pushed from testing in containers and triggers the maven report plugin. You are good to go with your HTML report!
It‘s clear that the team has put in extra effort and set up this flow to execute existing test packages for only a short time and as a temporary solution. Focus on how we can fix the problem rapidly so we still meet the schedule. This is how we were able to deliver our tests in time. In the meantime, we have started a new project with Rest Assured. Here is an article about how we started a project with Rest Assured written by my dear teammate Sebile. More is on the way. Stay tuned…
Piece of advice and lesson learned: don’t choose the framework that hasn’t been released even as v1.0 :)
Thanks for reading and feel free to reach out :)
Cheers🍻
