Cucumber Testing: A Language Everyone Can Understand

The Test Team at Conductor is small — by design. Quality is not just the Test Team’s problem; everybody is responsible for the quality of our software.

The Test Team’s mission is to discover defects before customers do. But that’s easier said than done! Since we keep the Test Team lean, each engineering scrum team includes a single Test Engineer to help them get their test stories done. But those test stories are just like any other user story in that they can be worked on by any available engineer on the team.

Thus, testing at Conductor is all about communication and collaboration. Tests create a language shared between product owners, test engineers, developers, and customers. And since tests are shared across many teams, everyone contributing to our projects must understand the language we use to describe them.

For this reason, we chose Cucumber as our language for writing automated acceptance tests. Cucumber is a Behavior Driven Development (BDD) framework written in Ruby that runs automated acceptance tests. Cucumber tests interact directly with product code, but they’re written in a language that all stakeholders can understand. During one of the Test Team’s sprint demos, one of our colleagues from the marketing department looked at our test scenarios and said, “That’s plain English! That’s not even code! I can probably write these.” By writing these tests collaboratively, we learn how to describe our product’s behavior in a language that everyone understands.

We also treat Cucumber tests as a kind of living documentation — the source of truth for what our product does and how it does it. Rather than wasting time trying to maintain formal development / test / product specifications (which go stale faster than bread in your freezer) Cucumber tests keep specifications, tests, and code fresh and in sync.

If anyone changes the functionality of a feature in the product without first updating the Cucumber tests for that feature, the tests will fail, in order to signal that a behavioral contract has been broken. When the Cucumber tests are updated, they implicitly update the specification and document the current expected behavior of the affected feature.

Cucumber works by reading our specifications from plain English text files called feature files. It scans them for test scenarios, and runs those scenarios against our product. The feature files must follow a set of rules called Gherkin. Gherkin is a business-readable, domain-specific language that you use to provide test steps and expected outcomes to Cucumber. When you launch Cucumber, it parses the Gherkin scenarios, executes the steps, and generates a report that describes how closely the product’s behavior matches the expectations set forth in the scenario.

Each step in a Gherkin scenario is mapped to a step definition, which is usually a few lines of code — in the programming language of your choice — bundled into a library of re-usable support code specific to your product’s functionality. Cucumber itself is written in Ruby, but it can be used to run tests written many other languages, including Java (via cucumber-jvm), C#, Python, and even Perl.

Test scenarios all follow the same pattern: Put the product into a particular state, execute the test steps, and then evaluate the state of the product. To identify those three parts of the scenario, Gherkin requires you to use the keywords Given, When, and Then. For example:

Scenario Outline: Successful login to Searchlight as a legit Searchlight user Given I am on the "Login" page When I fill in "username" field with "foo@mydomain.com" And I fill in "password" field with "CucumberIsMagic" And I click "submit" button Then I should be logged in successfully as "foo@mydomain.com"

Conductor’s product is a web application called Searchlight. But Cucumber has no idea how to talk to a web application. For that matter, Cucumber doesn’t know how to talk to a database — or any external system — either. Cucumber is just a framework that parses Gherkin feature files and executes the steps defined in them.

To bridge the gap between Cucumber and the technologies that make up our product, we use frameworks like Watir-WebDriver and Ruby’s ActiveRecord to create step definitions and other code that handles communication with these components.

As an example, our user-facing application is written in Java, but is backed by a stack of other technologies in our infrastructure. We use Cucumber to test Searchlight’s user interface via a web browser, but we’ve also taught Cucumber to interact with RESTful APIs over HTTP, our data storage systems, and asynchronous message queues. These software systems are developed by different teams and designed by different product managers, which is why it’s essential that the tests for these systems share a language for expressing their requirements.

Every time we implement a step definition using a new framework, it’s like introducing a new noun or verb into our language. In this way, we share the ownership of this language; we improve it, and we maintain it.

We use Watir-WebDriver to handle interactions with our web applications in a way very similar to how a real user would access them. For example, the scenario above launches a real web browser that visits a login page, enters a username and password in the appropriate text fields, and clicks the submit button.

Watir, short for for Web Application Testing in R uby and pronounced ‘water’, is a framework for automating a web browser. Watir-WebDriver is Watir’s implementation on top of WebDriver’s Ruby bindings. Watir only supports Internet Explorer, but Watir-WebDriver supports Chrome, Firefox, Internet Explorer, and Opera; and can run in “headless” mode, which is important for us when we’re running tests in a virtual environment.

One of the nice features of Cucumber is that if you don’t have any step definitions yet, the framework will generate a helpful stub that you can fill out with additional steps. For the example above, our scenario needs the following step definition:

Given(/^I'm on the "(.*?)" page$/) do |page|
    @b = Watir::Browser.new :chrome
    @b.goto "https://searchlight.conductor.com"
end

With this definition, when you execute the scenario, Cucumber will launch a browser and visit the specified URL. Magic!

That page has fields for your username and password. We’ll need to tell Cucumber how to fill them in to make the second and third steps of the scenario work. No worries!

When(/^I fill in "(.*?)" field with "(.*?)"$/) do |field_name, value| case field_name when "username" @b.text_field(:name => 'j_username').set(value) when "password" puts "in password" @b.text_field(:name => 'j_password').set(value) else puts "invalid field name" end end

When(/^I fill in "(.*?)" field with "(.*?)"$/) do |field_name, value|
    case field_name
    when "username"
        @b.text_field(:name => 'j_username').set(value)
    when "password"
        puts "in password"
        @b.text_field(:name => 'j_password').set(value)
    else
        puts "invalid field name"
    end
end

Now let’s click that submit button.

When(/^I click "(.*?)" button$/) do |button|
    @b.element(:css => '#j_login').click
endWhen(/^I cick "(.*?)" button$/) do |button| @b.element(:css => '#j_login').click end

Boom! You’re logged in. Now let’s implement the final verification step, which asserts that you’ve logged in as the correct user.

Then(/^I should be logged in successfully as "(.*?)"$/) do |email|
    result = true if @b.text.include? email
    expect(result).to be true
endThen(/^I should be logged in successfully as "(.*?)"$/) do |email| result = true if @b.text.include? email expect(result).to be true end

That was easy!

You can go much further with Cucumber testing than we do in the example above. This is just a window into how we use Cucumber to interact with our web application. We love Cucumber! It allows multiple teams to collaborate on producing awesome test scenarios for our product. And it enables developers at Conductor to take greater responsibility for testing. In recent sprints, some of our teams used Cucumber to write their sprint’s acceptance tests without help from their test engineers. How ‘bout that?

If you’re interested in learning more about software testing, BDD, TDD, Cucumber — or just want to say hi — you can get in touch with me via email.

Oh, and we’re hiring!