JBehave Vs Cucumber JVM: Comparison and Experience Sharing

Background

If you are a Java person like me and want to bring BDD to your Agile work environment, JBehave and Cucumber are the best options you will find useful. When I compare these tools, the best resource I found was the famous blog article written by Mykola Kolisnyk on the same topic. Although it was very informative, during the last 4 years, both JBehave and Cucumber have evolved differently and maybe we will have to do an extensive modification if we add all those information here. Therefore I decided to modify the Mykola’s scale a little bit and share my recent experiences with both tools.

Documentation and Community

The official documentation of a software tool is the major evidence available to public to have an idea on the capabilities and the best practices. Official web sites of both tools are very rich and organized with information. But in the practical environments, a general user needs more elaborated guidelines, blog articles, and community support. The major problem I understood was the JBehave related forums and blogs are written some time back and not up-to-date by now. When I got started implementing sample projects with both tools, I faced a lot of practical issues with JBehave and was hard to find the correct fixes. But for Cucumber, the implementation was straight-forward and less-painful in comparison to JBehave.

Conformity to BDD DSL (Gherkin)

Gherkin is the domain-specific language (DSL) used by BDD practitioners all-around the world as their business readable grammar. Gherkin has its own way of organizing the agile user stories using formatting rules like feature, scenario, steps, examples etc. Both JBehave and Cucumber support the Gherkin standard and its language rules. JBehave has its own grammar as well. But it is highly advised to stick into Gherkin syntax when you write your user stories because Gherkin is the accepted DSL in BDD.

Features

Although both are BDD tools, both tools have different implementations for various features to offer its users. I have selected the most useful features just for the comparison purposes.

Cucumber has built-in support for Java 8 features with its cucumber-java8 module. Developers who love using Lambda expressions instead of usual Java methods find this feature very attractive in Cucumber. Also since Lambda expressions are built-in feature in Java 8, probably its underlying implementation must be more efficient than the user-defined methods.

Most of the users make complaints on the JBehave IDE plugins (specially the IntelliJ IDEA users). But the Cucumber IDE plugins work very smoothly.

Composite Steps are not a feature of BDD/Gherkin Standard. Therefore, Cucumber and all other BDD tool do not support Composite Steps. But the JBehave users find this as a major disadvantage in Cucumber. Please note that the ‘composite steps’ feature was explicitly ignored by BDD practitioners and left out of Gherkin standard. The major reason was, people who made use of this feature frequently ended up with messy automation code that was fragile and difficult to maintain.

Source:

  1. https://github.com/cucumber/cucumber-jvm/issues/1052
  2. https://github.com/techtalk/SpecFlow/issues/813

Although there is a need for scaling our BDD test suites, having re-usability with ‘composite steps’ implementation may only help users in short run. When the systems evolve in long run, you will have to refactor your Gherkin scenarios, test methods, reusable test libraries, and the source code in ‘outside in’ manner. Having composite steps will be painful at this stage. So I suggest you to keep your scenarios and steps independent and decoupled as much as possible.

Reports

This is 2017 and our generation loves visual representations of data a lot. Therefore, reporting plays a major role towards the success of a BDD tool too. During my comparison, what I understood was, the report generating process in Cucumber is fairly straight-forward and the outcome is extremely nice. Cucumber event protocol is a data exchange format for components in the Cucumber ecosystem. Programs that support the protocol can be producers or consumers. If you are building a tool to analyse Cucumber events, this is definitely something you should look at.

On the other hand, the problem I noticed in JBehave is, its reports look a little-bit old-fashioned. I know, one can definitely implement a plugin to improve and customize its reports. But when Cucumber provides very nice reports, I am just wondering whether the time you spend on improving JBehave reports is a worthy thing to try.

Github Source Code Status

In the mean time, I checked the Github repositories of both tools as well. Above is the status as at 25–07–2017. As you can see, people interact more with Cucumber than JBehave at the moment. Also note that the code quality rankings and insights are calculated and provided by Lumnify. They vary from L1 to L5 with “L5” being the highest. (See more stats)

Stack Overflow Tag Analysis

The interactions of the community is one way to assure the long-lasting support for any software tool. Stack Overflow is the best place to study these interactions. Above are some of the stats I could find. People use different tags and I filtered my search results using different tag combinations. Every time, Cucumber had more community interactions. Also what I notice is, the discussions on Cucumber are relatively very new in comparison to JBehave. (Note that both Cucumber and Stack Overflow were launched in 2008, and JBehave was launched in 2003. So these stats can be taken as fair for both tools)

Final Words

Although I did not explicitly mention, JBehave has many additional configurations to fine-tune a BDD tool as per your preferences. But with those configurations come a great complexity too. At the end of the day, we are going Agile and we love being Lean in our work. The question I had was whether these complex configurations add any significant value to my workflow. The answer was a ‘no’. I have seen people doing BDD even without a powerful tool like Cucumber or JBehave. Some creates their own tools as well. My observation was, if you need a BDD tool, Cucumber has almost all the power you need :-)

According to my understanding, JBehave is in the maturity phase of its life cycle, whereas Cucumber is still reaching the maturity level with cool new features day-by-day. Cucumber will definitely have a promising future and I feel very safe to go with Cucumber ;-)

Common Issues

Both tools will have performance issues when the number of test scenarios are increasing in your project. Make sure you organize your BDD scenarios and features very neatly. Give the power to your test framework to effectively arrange and run your BDD tests. You will definitely need DevOp level optimizations and investments for this. There is no golden rule to solve such practical issues. Therefore, make sure you interact with the community and educate yourself with the best practices in the industry.


Be sure to follow me on Twitter @ashen_wgt and send me your feedback, any questions you have, and also let me know your experiences and I’ll check it out/retweet! You can also check my other articles on Agile Vision.