Developing Quality Open-Source

Based on Pyjama Talks, Episode 12 with Vasily Sozykin (Wrike), Shelley Lambert (Red Hat),hosted by Evgeny Kot (Wrike Prague)

Many software developers think testing is boring but it doesn’t have to be! Having discipline in QA is important, as testing is crucial for quality development. There are companies that require QA specialists to be able to write software and vice versa.

Every developer who writes code should have it tested to verify they won’t backslide in the process of work. There are good methods for automating the testing process but manual testing is still as important. At the end of the day, one can consider customer feedback a form of outsourced manual testing, which is a vital and common process for most products.

In this article we’ll cover the subject of open-source testing, QA practices, and how to achieve a quality open-source product.

Testing, shipping, and risks

It’s true that we live in the era of the shipping-oriented approach, where major companies operate similar to startups — ship, evaluate, and improve. The fact that it helps your product keep a finger on the pulse of the market and navigate customer demand, while making the adjustments process more effective is great. However, just because you can be fast doesn’t mean that you should be, especially if you introduce risk.

«Don’t mistake agility for haste»

You have to weigh in on all the pros and cons when setting the pace for your development process. For example, what’s the risk of upsetting customers vs. a severe security breach? Do you make them wait, or have them suffer the consequences that you’ll deal with later? You’ll have to make a choice and estimate the risks of releasing any piece of software of which there are still unknown variables.

That’s where the notion of technical debt comes into place. This debt is just a choice. You want to get something out quickly, but do you have a plan after release? Let’s say you have several features to roll out today. These are highly demanded and anticipated by your user base.The announcement is already out there. They’re all expecting the update. You run tests and find several bugs. One of them is security-related. You roll the dice to ship anyway, but you should also have a plan to fix all of that as soon as possible, and be ready to solve some of the collateral damage the bug may cause before it’s fixed.

Testing open-source products

It’s great when you can calculate risks before shipping, but open-source software like OpenJDK is a complex product and one can’t even imagine how customers will use it in their production. How should you test such products?

Shelley Lambert, Manager of Software Engineering at Red Hat, shares that the development team’s activity around OpenJDK is heavily focused on the testing process and the evolution of both tests and testing approaches. The OpenJDK team puts in more work into testing development than in some of the changes in the language itself.

«So we could use an approach that says, ‘We will vary and do pairwise testing and have a very high functional coverage.’ You know, 85% or 90%, but [with] a very minimal number of tests that are needed. And it’s not just the functional correctness. We also obviously have to look at performance, we have to look at scalability, we have to look at stress testing.»

One approach to also look for is to learn simply from what gets reported. Another is to apply, for example, a combinatorial test design. Based on research and how software is written, this approach allows us to find the most defects by combining two inputs.

Open-source responsibility

When you develop a product for purchase, you can prioritize issues. Users who didn’t pay go to the back of the queue, unless their issue is critical. But how should one prioritize bugs and testing in an open-source product?

The pillar idea here is you have to make sure your technical debt is constantly reducing.

1. Fix: Some bugs are just obvious: They’ll have multiple reports, or dozens of users sharing the same feedback.
2. Test: If there’s a new feature, make sure you’ve enabled all the testing. Check to see if it requires more tests.
3. Innovate: Are you able to apply a deep learning model to automate some triage? For example, identify the tasks that take the most time and energy, and see if those areas could be improved by applying new tech or new approaches, etc.
4. Be open-source-ready: Reduce your dependency on specific solutions you have to maintain. See if you can move to using open-source tools, or invite collaboration.

Testing open-source backwards compatibility

With OpenJDK, Lambert shares that the development team pulled together a QA program to be able to handle different versions out of one repository. The first step was to overlay filters on their test material, and overlay filters on their devops automation scripts.

«I can have a single repo with a filter on it so that if I want to test something in Java 8 vs Java 11 vs Java 16 — it’s the same script. It’s just a little piece of automation that auto detects you’re testing this version. The issue there, of course, is that as we have new releases and new features, [there] will be a lot more tests, so we do have to manage time and resources accordingly.»

Testing in a highly critical production environment (like a nuclear submarine) is, from Lambert’s experience, owned by both the producer of the product and the people who would take on that product. In her case, they had a set of test criteria and performed a lot of testing on their side.

Production environment and testing

Some parts of your software can be tested inside your production environment, but is it good?

Let’s take a look at it from the point of view of layers of testing. You start at the bottom of the stack and do the unit testing. You test each function as thoroughly as you can. You find a lot of materials at the op-code level, and as you go up the stack they’ll cost more and more.

Let’s take the nuclear submarine example: When you release an unarmed nuclear missile, does it come back and dive into the water near its launch point? That’s why you have to test thoroughly, because each production test costs a lot of money.

Now what happens if you don’t test in production? If the only thing that happens is that someone’s application doesn’t start properly or runs slowly, it’s not that critical. But if an application is controlling someone’s pacemaker and then stops? Now that’s critical. Mission critical software needs to have a solid discipline.

With unit vs. functional tests in microservices, Lambert says it’s a mistake for microservices not to use some form of unit or API test, because a microservice can be considered an API. So it’s a no-brainer and inexpensive to test an API. From a system-level perspective, microservices still have a need for system-level testing, or they won’t know the risk if something doesn’t work.

Why are QA and development separate?

Why do we still have separate departments, or even separate companies for quality assurance and software development? In some cases a software developer has to be focused. Let’s say they’re working on a specific piece of software and are specialists of a specific profile.

However, it’s all about perspective. It shouldn’t be about what skills we have now. Sometimes, a company may hire a number of manual QA employees who don’t have software development skills, because the job at the moment is clicking buttons, but that’s a short-sighted view. If you’re in QA, let’s get better next year in terms of your skills and approaches. Innovation and evolution is vital. QA should move even faster than the software development team. So make sure you’re sharing your knowledge and experience with others to lift them up. The same works for tooling.

Many companies don’t have tooling for specific problems and face a choice: an external solution vs. a homemade solution. Usually, they prefer to develop a homemade solution. Sometimes, it can even grow into a successful, homemade enterprise solution, but it’ll require some maintenance.

In fact, there’s a number of reasons to look for ways to take out proprietary solutions, or replace them with open-source solutions. The major reason: You’re not the only team maintaining it. You get the benefit of picking up other teams and other people’s improvements on the tool that you’ve selected. It’s always good to have a community of people working on the same projects.

Open-source/open-end

QA is a software development challenge and a collaborative effort on many levels.

Sometimes you have to make a collective choice with your team. But possible implications of that choice spread beyond the team itself. Even in terms of a devops pipeline, what CI system should you use? For example, Red Hat uses Jenkins to run their pipelines. But they also had to recognize that if they wanted to make something other people could utilize, they should allow their team the ability to run outside of that CI system and use a different one.

What’s even more important is to grow people’s skills, whether you’re leading a team or helping people. Make sure they can become their best. Once they reach their full potential, help lift their colleagues up. Reward the right things to make your team productive, engaged, and happy long term.

Please don’t hesitate to reach out to us with any questions or suggestions you might have.

Featured:

Evgeny Kot: Host, Director of Development at Wrike

Vasily Sozykin: Co-host, Backend Manager at Wrike

Shelley Lambert: Guest, Manager of Software Engineering at Red Hat