My first (and last) Openstack test with behave

In the world of testing driven development, there is a special discipline. It’s called ‘behavior driven development’, aka BDD. I heard about it for awhile, and I’ve decided to play with it a bit.

My goal was to test Openstack infrastructure at operators level. That means there is a real working Openstack installation and I want to be sure it works. Defenition of ‘works’ is a subject of my research, and an idea of ‘scenarios’ in BDD sparked my interest. We have user scenarios and we want to test them.

There is a python library, behave. It’s available in many modern distributions, so apt-get install python-behave works just fine.

Quick intro

The process of writing tests for BDD is split into two parts. The first part uses some human-like language to describe ‘scenario’. A scenario is a list of initial conditions, actions and results. Each scenario can be parametrized. A second part is written in Python, and linked to ‘human text’ by special decorators. The second part may have setup/teardown/cleanup code, error handling, etc.

An intention for BDD is that there is a person who cares about a product (project manager, product owner, etc), but who have no deep understanding of the code. By ‘deep understanding’ I mean s/he does not know which function or API path should be called for a given function. And there are programmers, which are less interested in the global consumer view of the product but have a deeper understanding of implementation details. Project manager writes his expectations about the product in a human-readable format (which is not tied to any particular language, framework or API version), and programmers implements tests which translate those ‘human-readable verses’ into actual code (which is tied to language, framework, and API version). All newer-implemented features should pass those tests. If project migrates from one miracle framework into another brilliant one, text part will stay the same, the test would need to be adjusted and new code should pass those tests.

Sounds interesting, doesn’t it?

Openstack

I wrote a humble scenario:

Feature: nova boot and delete
Scenario: Create and delete instance
    Given we have valid credentials
     when we start instance
     then we can delete it.

(this is actual human-like text from nova.feature file, which is part of the test).

Then I wrote an implementation of these words:

(boilerplate part):

from behave import given, when, then
import novaclient.client
from keystoneauth1 import identity
from keystoneauth1 import session
import os

(Pay attention to ‘given’, ‘when’ and ‘then’ decorators I imported from behave).

Then I implemented first verse from feature file: Given we have valid credentials:

FLAVOR_ID = 'm1.tiny'
IMAGE_UUID = '02bdad7f-248b-4357-9d85-9e71fc71c7eb'
NET_ID = '89e03f44-d874-4c79-bbb5-30d5c576fef3'
@given('we have valid credentials')
def step_impl(context):
    auth = identity.v2.Password(
        auth_url=os.environ['OS_AUTH_URL'],
        tenant_name=os.environ['OS_TENANT_NAME'],
        username=os.environ['OS_USERNAME'],
        password=os.environ['OS_PASSWORD']
    )
    sess = session.Session(auth=auth, verify=False)
    assert sess
    context.nova = novaclient.client.Client('2', session=sess)
    assert sess.tenant_id

This example mostly contains openstack-specific code. I created session object (sess) and initialized nova client. I also checked that credentials are valid because tenant_id will be available only if authorization worked.

Next verse, we start instance:

@when('we start instance')
def step_impl(context):
    context.instance = context.nova.servers.create(
        'test',
        IMAGE_UUID,
        FLAVOR_ID,
        nics=[{'net-id': NET_ID}],
    )
    assert context.nova.servers.get(context.instance)

Again, it’s mostly Openstack code. I created an instance with given parameters and checked if it exists. Pay attention to the decorator: it contains exact wording from text ‘feature’ file.

And finally we implement our check, then we can delete it.:

@then('we can delete it.')
def step_impl(context):
    context.nova.servers.delete(context.instance)

Now we can run our test.

(N.B. python-behave package is slightly broken in Ubuntu, as it does not create entrypoints, so one need to call behave as a module)

$ python -m behave --no-capture
Feature: nova boot and delete # nova.feature:1
Scenario: Create and delete instance  # nova.feature:3
    Given we have valid credentials     # steps/nova_test.py:11 1.197s
    When we start instance              # steps/nova_test.py:24 0.670s
    Then we can delete it.              # steps/nova_test.py:35 0.349s
1 feature passed, 0 failed, 0 skipped
1 scenario passed, 0 failed, 0 skipped
3 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m2.217s

Looks beautiful, doesn’t it?

Grain of salt

If I change this test in less verbose pytest-style code with some fixture to provide me booted server, would I have less expressive code? I don’t think so. Would it be more concise? Yes, it will. I just wrote a text file (nova.feature), every line of which is sacred and should be preserved in pristine form across all my source files. I did same work, but with more letters on screen, with more keys pressed.

Well-written test code will have the same level of expressiveness.

Just look at this:

import novaclient.client
from keystoneauth1 import identity
from keystoneauth1 import session
import os
import pytest

@pytest.fixture(scope='module')
def sess(request):
    auth = identity.v2.Password(
        auth_url=os.environ['OS_AUTH_URL'],
        tenant_name=os.environ['OS_TENANT_NAME'],
        username=os.environ['OS_USERNAME'],
        password=os.environ['OS_PASSWORD']
    )
    sess = session.Session(auth=auth, verify=False)
    assert sess.get_token()
    return sess

@pytest.fixture(scope='module')
def nova(request, sess):
    return novaclient.client.Client('2', session=sess)

@pytest.fixture(scope='module')
def nova_boot_server(request, nova):
    FLAVOR_ID = '30'
    IMAGE_UUID = '02bdad7f-248b-4357-9d85-9e71fc71c7eb'
    NET_ID = '89e03f44-d874-4c79-bbb5-30d5c576fef3'
    instance = nova.servers.create(
        'test',
        IMAGE_UUID,
        FLAVOR_ID,
        nics=[{'net-id': NET_ID}],
    )
    return instance

def test_boot_and_delete(nova_boot_server, nova):
    nova.servers.delete(nova_boot_server)

It’s precisely the same, yet it has much less bureaucracy and less wording. It even has better code structure, as I have split one big fixture into three.

Therefore, I abandon BDD, as redundant. It does not help me in any way.

Conclusion

BDD is a management technique. It gives nothing to programmer, but it helps managers to stay in touch with programmers and provides a bit more formal (accountable) description of tasks.

If you are manager and you want to make more robust project with formal metrics, if you want to have control over results — BDD is your savior. Use is, force it, demand it.

If you are programmer or devops, ignore it until you are forced to use it. It wouldn’t raise your productivity, it wouldn’t save your time. It even wouldn’t give you any tools to work with a team (of guys of the same qualification). It solves other guys problems. But when you are forced to use it, use it without complaints, because manager with solved problem is better than a manager with unsolved problem.