BDD tests for Hadoop with Cucumber. Part I

Big Data is a processing of a huge amount of structured (more or less) data that should be filtered or sorted for future analysis or, actually made an analysis. So much data that you can’t process it on the one computer for a reasonable time. This can be: logs from services (like call service) or logs from web servers that contain billions of records, so you should process data in parallel on a bunch of computers. To do it so we use specific software based on MapReduce pattern. In our case, we use Hadoop with Cascading framework. Hadoop implements MapReduce and Cascading framework contains a lot of useful tools, for example, abstraction to Amazon S3, so we can use S3 buckets as input/output folders for processing jobs, but at the same time, we allow to use regular file systems on developer machines for development and test purposes.

Like any other piece of software, it should be tested before using in production, especially because of the high cost of logical mistakes in processing a lot amount of data. Unfortunately, there is no much information about BDD testing of Hadoop and Hive jobs. So I decided to write how we did it in our Cascading project in Intelliarts.

What do we have

Assume we have a lot of data about phone calls in a bunch of files. The file contains lines in the following format: Id,country,time,duration Where id is a call unique id, country is a caller country, time is a time when call started (timestamp) and duration is a call duration in milliseconds.

For example:

0000,UA,1433998201,60000 0001,US,1433998201,30000 0001,GB,1433998301,30000 ...

What do we want to do?

Let’s sort calls from these records by country (each country will be in a separate folder) and change the format of records — replace delimiters from , with tab symbol \t. From the data of the previous example so we should get the folders US, GB, UA and UA will contain a file with records looks like this:

id	country	time	duration 0000	UA	1433998201	60000

And, we want to be sure that we did not make any mistakes during implementation, so we want to test the results. What we will use:

• Hadoop + Cascading for data processing
• Gradle for build management
• Cucumber for tests
• Chunk Templates for test data fixture generation
• Docker for environment isolation

First of all, we should isolate our testing environment and make it portable between dev machines and CI. We use Docker for this propose. Inside the container we will:

1. Install Java
2. Install Hadoop
3. Install Gradle
4. Add codebase

Here is Dockerfile content:

Okay, now we have the environment. Let’s create a project Gradle config to download all dependencies for Hadoop application:

As you can see there are two custom tasks — cucumber and jar. First one runs cucumber tests and the second one compiles the source to the production jar-file. Now we can build the image:

$ docker build -t bdd_hadoop .

Time to build a processing application. Let’s create a directory structure: src > main > java > com > processing and add a file CallStream.java with content:

Let’s try to build a jar file:

$ docker run --rm bdd_hadoop gradle jar

Now we need to create some acceptance tests. This will be in Part II. Stay tuned!

Originally published at www.savvyclutch.com on October 7, 2016.