OpenTestingApi — Part 2 — pipeline integration and test automation for various environments

Robert Diers
NEW IT Engineering
Published in
3 min readJun 12, 2023

API 1.2 arrived and we have a lot of new useful features, but within this article I will focus on the automation possibilities for your testing teams.

To enhance our basic pipeline integration we investigated into address the “I need to run the tests on various environments automatically” topic.

The idea:

OTA as a end-2-end testing solution can test a dedicated app and also any kind of environment running multiple applications, including tests with realistic work load creation.

So let’s try to find a solution for these 4 scenarios:

  1. System test (test a specific application only)
  2. Integration test @ existing environment
  3. Integration test @ virtual environment
  4. Realistic work load test @ existing scaled environment

There might be more ones like chaos, acceptance, downtime, whatever tests — but the approach will be the same for all of them.

We can use a standalone deployment of OTA to permanently test an environment, but it will not solve the ‘one-time-shot’ requirement of the pipeline — for example when a new application feature is going to be released.

Putting the feature of different environments into OTA implementation itself will make it much more complex and error prone, so we tried to solve it with an enhanced pipeline integration using standard Java tools / code / project. The good thing about it is the adaptability to your own requirements and circumstances — simply change it.

Features:

  • load test cases from resource folder (no Java code required to add new test cases)
  • use placeholders for environment-specific values like connect URLs, maven profiles are used to define an specific environment
  • one-time-creation of execution project, pipeline can use simple ‘maven test’ command with maven profiles to start the env-specific tests
  • integrate test cases from other repositories by using git sub-modules
  • parallel or sequential execution is possible
  • define and use custom repositories
  • preparation test case before real test case execution (create database users and so on, typical ‘service’ tasks to fulfill your external application requirements)
  • testcontainers.org allows us to create virtual environments
  • define the OTA instance to use (code change required)

Example using Maven profiles:

  1. System test → mvn test -Psystemtest
  2. Integration test @ existing environment → mvn test -Penv1
  3. Integration test @ virtual environment → mvn test -Penv2
  4. Realistic work load test @ existing scaled environment → mvn test -Pgivemeacoolnameenv

You are free to use any kind of profile names like commonly-used ones (‘integration’ and so on), I just used a few examples here.

How does it work?

Basic pipeline integrations allows us to execute a OTA test case using testcontainers.org:

The enhanced versions add a few more steps:

  1. add a Maven profile and properties to define your env-specific values (in this case an external Cassandra database): https://github.com/opentestingapi/impl_java_testcontainers/blob/main/stagesupport/execution1/src/test/resources/application-env1.yml
  2. use testcontainer.properties to define your repositories and images: https://github.com/opentestingapi/impl_java_testcontainers/blob/main/stagesupport/execution1/src/test/resources/testcontainers.properties
  3. use placeholders in your test cases, like it is done in this example: https://github.com/opentestingapi/impl_java_testcontainers/blob/main/stagesupport/execution1/src/test/resources/opentesting/templates/testcase001/test.yml

That’s it for using existing environments (required code can be copied from Github). If you like to use virtual environment, here’s an example to startup custom containers (in this case Cassandra, but can be every app): https://github.com/opentestingapi/impl_java_testcontainers/blob/main/stagesupport/execution1/src/test/java/com/example/demo/devenv/Env2.java

Any ideas or suggestions for improvement?

We are happy to hear from you :-) https://github.com/opentestingapi/backlog/issues

Gitlab CI example to get Docker-in-Docker:

.whtever-tests-using-testcontainers:
image: ${YOUR_MAVEN_IMAGE_FOR_TESTPROJECT}
stage: your-systemtest
services:
- name: ${YOUR_DIND_IMAGE}
alias: docker
command: ["--tls=false"]
variables:
# Instruct Testcontainers to use the daemon of DinD, use port 2735 for non-tls connections.
DOCKER_HOST: "tcp://docker:2375"
# Instruct Docker not to start over TLS.
DOCKER_TLS_CERTDIR: ""
# Improve performance with overlayfs.
DOCKER_DRIVER: overlay2
script:
# Execute Tests
- echo ${PROFILE}
- mvn test -P${PROFILE}
- STATUS_RUN=$?
- echo ${STATUS_RUN}
- >
if ([ "${STATUS_RUN}" = "0" ]);then
exit 0
else
exit 1
fi
artifacts:
when: always
paths:
- "**/surefire-reports"
- "**/target/*.log"
- "**/target/test-classes/opentesting"
expire_in: 1 week

and usage for virtual environment: https://github.com/opentestingapi/impl_java_testcontainers/blob/main/stagesupport/execution1/src/test/resources/application-env2.yml

--

--