How We Do It: An Exercise with Robot Framework

At Hunter we focus enabling our customers to not just build software, but to create automated software factories where sustainable capability engineering can be built and deployed all the way to production iteratively and collaboratively. A large part of this is using tools which automate functional user testing so that new capabilities are ensured to not break any old ones. While Hunter consultants use various automation tools depending on the investments already made across our customer base, for Hunter corporate projects we tend to choose Robot Framework. In his post we will use Robot Framework as a generic test automation tool, and use Selenium Webdriver to perform very basic browser automation.

After you complete this exercise, you will have successfully created your first (empty) test case, and executed it using a test runner docker image.

The Challenges of Test Automation

Throughout our customer community we have seen multiple entities struggle with test automation solutions on across the board, specifically with regard to User Experience testing automation. This is because automation for user driven behavior frequently falls short of the available time and resources available to build automation scripts. Accordingly Hunter has developer a set of questions to help provide high level guidance when looking to create UX automation testing.

• How do I ensure that test automation can reflect UI changes?
• How do I ensure that tests are meaningful to business use cases?
• What is the long term cost of testing?
• How to do enable just in time testing? In other words, how do i make testing so fast developers can actually use it?

The Case for Robot Framework

As we mentioned, Robot Framework is our corporate preference when it comes to test automation for several reasons including:

• It is free and open source.
• It provides very pretty reports.
• It has excellent documentation.
• It has a concept of prioritization.
• Test specifications are in a natural language.
• It is a generic test automation framework that can be used in a wide variety of test paradigms including websites, web applications, mobile apps, APIs, or even desktop applications.
• It’s extensible such that it can be used as middleware between multiple testing tools (selenium and cucumber) and leverage existing investments in existing frameworks.

Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and it utilizes the keyword-driven testing approach.

(Source: Robot Framework Site)

Selenium: Hello Darkness, My Old Friend

Every week it seems like there is a new test framework to talk about. Look at Puppeteer, Cypress or TestCafe. All coming to fruiting within the last few years. That said, Selenium remains one of the highly prevalent test automation tools available on the market today. Why use selenium ?

• It has the most robust IDE in open source test tools
• Cross Language Support
• Cross Browser Support
• It has excellent documentation.

Robot Framework comes equipped with SeleniumLibrary, that can be used to interact with a remote SeleniumHub. For this example we will be bringing up the Hub with docker but you can connect to existing hubs thus taking advantage of existing selenium investments.

Getting Setup

In order to complete this exercise you will need to:

• Already have an IDE capable of editing Robot files and script files (Notepad++ will do fine).
• Install Docker. For dev, I always follow worst practice and enable super user for docker so i don’t need to escalate privileges each time. This is really a lazy approach and should never be done in a public facing environment, test or not.
• bash installed and running

For the purposes of this exercise it is assumed that you are somewhat familiar with code editing and working on the command line.

Parameterization

Two methods of parametrization of test runs will be introduced:

• parametrization via env variables for parameters that are to be changed at runtime (for example: browser to use, or target system base URL)
• parametrization via YAML configuration files for parameters that are to be changed occasionally, but due to their nature, require aggregation in one place (for example: web elements selectors, or test data)

Starting with a simple test script

First step: Write the first test script! Robot Framework preferred syntax is using natural language or gherkin style syntax (which is the language of Cucumber). From the perspective of DevSecOps test automation specialist this helps focus on mission oriented aspects of the tests, instead of the underlying implementation. A working example would be to say that it’s more important to say that a financial transaction must complete successfully, than to describe, step by step, how to make a transaction or what aspects of the HTML on a page must be activated to do this.

For example: we will test this page in Firefox and chrome. First, we start Robot Framework and run a “null” test. This will be a file called null_test.robot in the test/ folder:

test/null_test.robot

The typical file extension is .robot. Infact, the.robot script can be used by Robot Framework as either a resource file or a test suite. The main difference is the stated purpose of each file. Test suite’s contain test cases and are built to be executed by Robot Framework, whereas resource files contain only keywords for shared use in multiple test suites. Kind of like a set of pointers. Resource files can be referenced by other resource files, or by test suites, but test suites cannot be referenced by any other file.

Robot Framework uses the systems file structure as a way of organizing test suites. Each directory that contains test text files will be recognized as a test suite, with its own respective setup, teardown, or metadata. Robot Framework will treat test/ as a test suite (collection of tests), but initial_test.robot will also be a test suite.

Text written without indentation under Test Cases are treated as test case name just like similarly unindented text under the Keyword headers are keyword names. Test case content and Keyword content must be similarly indented. Using this knowledge, we can write the simplest test suite (test/first_test.robot) to get our setup going:

*** Test Cases ***
This is a null test case
   #This is how comments work in robot
   NoOp

No operation is a keyword from BuiltIn Library, and - as the name suggests - we don't have to import anything to use those keywords.

Breaking it Down

This is a null test caseis the test that we will be running. It has only one test case to run. This test case, NoOp, will do absolutely nothing (as we expect), just mark the test as passed.

How to run this test

To execute the test suite that we have just built we will leverage a docker image to create a container with our test execution. Let’s go the Docker way, simply because it is more repeatable for those reading along. Fortunately, most of the legwork has already been done for us in the robotframework/rfdocker project:

docker run \
 --rm \
 --volume "$PWD/tests":/home/robot/tests \
 --volume "$PWD/results":/home/robot/results \
 robotframework/rfdocker:latest \
 tests

This is quite a lot to unload, and I’ll refer you to docker documentation for all possible options, but here’s a rundown:

• docker run: use docker to execute a specified image as a container
• --rm: don't store the container after execution had ended
• --volume "$PWD/tests":/home/robot/tests: mount local directory tests/ to the container directory /home/robot/test. /home/robot is the default location where the Robot Framework will be invoked when container is run.
• --volume "$PWD/results":/home/robot/results: similar, mount local directory results/ to the container directory /home/robot/results. This is the default test log output directory. Since docker containers are ephemeral by their nature, a local drive mount will ensure logs do not get lost when the container is spun down.
• robotframework/rfdocker:latest: specifies the docker image for execution. latest is fine for development purposes, but you will want to link a specific image version for your production environment
• tests: specifies the name of the test suite to run. So the default Robot Framework invocation directory /home/robot concatenated with tests gives us /home/robot/test - which is exactly the directory we had linked with our local test suite content.

I hate to repeat myself, so let’s just pack it into a run.sh script to avoid early onset of RSI:

#!/usr/bin/env bash
docker run \
 --rm \
 --volume "$PWD/test":/home/robot/test \
 --volume "$PWD/results":/home/robot/results \
 robotframework/rfdocker:latest \
 tests

Now for last adjustments, let’s make it executable, and in case you were using git for your version control (which you should), we’ll add the results/ directory into our .gitignore file:

chmod +x run.sh
echo results/ >> .gitignore

We can finally call run.sh and enjoy the magnificent results of our first, unproductive test case:

> ./run.sh
============================================
Tests
============================================
Tests.Initial Test 
============================================
This is a null test                | PASS |
--------------------------------------------
Tests.Initial Test Suite            | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
============================================
Tests                               | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
============================================
Output:  /home/robot/results/output.xml
Log:     /home/robot/results/log.html
Report:  /home/robot/results/report.html

You will notice that the results/ directory has appeared in your working directory, and there are 3 files in it:

• output file -output.xml - contains machine-readable test results summary.
• report file — report.html - this is a high-level overview of your test execution. For ease of finding out if test run had been successful, it has either bright green or bright red background.
• log file — log.html - this is the interesting one, as it lets you dive deep into all the details of test suite execution.

Congratulations, you have successfully executed your first Robot Framework Test Case!

Review

• If you successfully followed this quick tutorial you will have implemented a basic Hello World-type test case in Robot Framework, a necessary building block in a proper modern software factory
• We've leveraged Robot Framework in a Docker container for easy repeatability
• We’ve codified the test runner as a standalone script, so it could be at this stage placed in CI/CD Pipeline using Jenkins and act as template for future tests.