The importance of software development process
Blockchain isn’t rocket science — but it needs to be.
The ultimate goal of any software project is to produce high-quality, reliable, and maintainable software at the lowest cost and fastest pace possible. This is only possible with a well-designed development process established within the development team.
In today’s Medium article, I will explain why software development process matters and how it directly impacts the quality of the software we produce.
One of the biggest misconceptions with software engineering is that people often tend to think coding is the most important step. On the contrary, coding is far from that!
Aside from rapid prototyping, one of the worst habits a software engineer can pick up is to jump straight to coding without any software engineering. This type of practice quickly leads to poorly-designed software that inevitably will have an unusually high rate of bugs.
When I was working at JPL NASA, I once heard something along the lines of, “People usually take software for granted; as matter of fact, often times people don’t realize that software even exists on the hardware they are operating. There are really only two ways that people would care about your software. One, if they would die from the misbehavior of your software, or two, if they would lose money from the misbehavior of your software!”
The mentality of building high-reliability software is deeply instilled in the heads of JPL flight software engineers. While no human life is at risk in robotics space exploration if the software that controls the spacecraft misbehaves, there are billions of dollars and years of development at stake. Blockchain software, like Logos, also falls under the second category.
Unfortunately, we, as an industry, have yet to fully embrace a methodical approach to software engineering. While I do anticipate that this will change in the next couple of years, blockchain projects as a whole (with very few exceptions) have very poor software engineering practices that jeopardize the integrity of these systems.
At Logos, we embrace a methodical approach to software engineering — there is a rigorous software development process that the core team follows. We establish various gates to ensure the quality of the deliverable is met before proceeding to the next step.
In addition, we also enforce a well-balanced documentation requirement. Although we do allow rapid prototyping to bypass most of the gates, we still take all prototype code through the process to ensure it meets the production standard.
Step 1: Requirement Derivation
The Logos Network white paper serves as the genesis requirements for the projects.
A white paper’s purpose is to present a high-level description of the system with only some key attributes discussed. It serves as an idea to readers, but it is not sufficient to realize an implementation.
If anyone tried to code to the white paper without additional engineering, they will soon realize that they are now designing a complex software system as they are coding — as I discussed earlier in this article, this quickly leads to inferior software quality.
So, before we jump right to coding, we need to mitigate some risk by sorting out the ambiguities presented in the white paper. We call this step requirement derivation. Perhaps surprisingly, the vast majority of blockchain projects skip the requirements step and go straight from white paper to coding.
At Logos, we use a spiral-like model. During the requirement derivation phase, engineers decompose the white paper into various components of the software at a high level. We are not designing how the software is architected during this phase, as this is only a functional break down. Most of the ambiguities should be eliminated during this phase of engineering.
Engineers are encouraged to perform rapid prototyping to prove that some of the concepts depicted in the white paper are actually realizable by the underlying implementation. In some cases, we had to make updates to the white paper due to the results of some trade studies and prototyping.
In layman’s terms, this is the phase where the team gets to fully understand what is needed for the project. The diagram below depicts a generalized, spiral-like development process.
Note that depending on a project and team’s actual situation, development methodology needs to be heavily tailored to suit a project’s needs. For example, the consensus requirement derivation was not incremental at Logos and was done in one shot — we dug so deep until there was no ambiguity left.
At the end of this phase, we generally have a pretty good idea of what it takes to build the entire software system. The deliverable for this phase is a well-documented requirements document that details what the software system does.
In the aerospace industry, there are some extremely rigorous standards and guidelines, such as the DO-178B level A, that recommends to have requirements that are traceable to each and every line of the source code. While this may be overkill for the blockchain space, it’s still critical to derive clear and detailed requirements.
The requirements should only call out key attributes of the systems with sufficient granularity so that anyone that wants to have a custom implementation of Logos Network can do so without reading Logos code.
I remember reading an article in which a Bitcoin Cash developer expressed frustration that even the developers themselves do not know what is in the code base anymore due to lack of documentation. This is a serious risk, and is exactly what Logos is trying to prevent!
Step 2: Design Phase
Once we get a good understanding of what to build, the next step is the design phase, which involves figuring out how to build the software system. As with requirements derivation, trade studies and prototyping are highly encouraged through the process.
During the design phase, we take a deep dive into software architecture and design. We figure out how to break down the software logically and how to put the pieces together into a functional system.
C++ is a powerful but potentially dangerous language, so, with a good design process, we can mitigate most of the risks associated with this step. State diagrams, use case diagrams, sequence diagrams, and other visual aids are crafted during this phase to help understand exactly how we are going to build the system. Engineers are expected to have the whole component designed out before proceeding to coding. Data formats, resource utilization, and concurrency issues are also thoroughly discussed in this phase.
The deliverable at the end of this phase is a well-documented software design document, which enables readers to understand how our software system works at the implementation level.
Third party developers should be able to contribute to the ecosystem without reading your code! This means that, in addition to the design, we also need to document how messages are transmitted over the wire, and how they are formatted.
We firmly believe this is absolutely necessary if we want an active, open community and a sustainable ecosystem.
Step 3: Coding and Unit Test
After design is complete, this is when we are finally ready to code!
With a good design, coding should be the easiest part of the job! Why? Because coding should be nothing more than a realization of the design!
Realistically, we might still run into some design issues during implementation, but it is expected that the majority of the design is completed during the design phase, not the implementation phase.
The implementation phase should be relatively straight forward. I am not going to spend too much time on this phase, as virtually all projects are intimately familiar with it.
Step 4: Verification
Another issue I see with some blockchain projects is the lack of verification.
It would be naive to think that Travis CI or other common integration tools would serve as an adequate testing infrastructure before production deployment — but most projects do just this!
Can you imagine flying an aircraft without actually testing it? That is suicide! So why would it be acceptable in the blockchain industry to not verify your software system before deploying it?
Verification is absolutely critical to the success of a software project.
Continuous integration tools like Travis CI aren’t adequate because of exactly that — it is a continuous integration tool, not a system verification tool. It is not capable of verifying your system. When software units are put together, the dynamics change and complexities increase as more units (components) are integrated. This means that tests that span across multiple components need to be created to verify the entire system as a whole.
Distributed software systems are inherently difficult to debug and test — this just means we need to spend even more time verifying it. Nothing is free — and, often times, inexperienced teams cut corners on this exact step which leads to latent bugs that can cost users millions of dollars later on.
Remember, the earlier a bug is caught, the cheaper it is to fix. And it is almost cheaper to spend more money to catch bugs before they make it to the mainnet! Verification, as the last line of defense, should be explicitly designed to catch the corner cases that seem impossible to occur.
Conclusions
Blockchain software isn’t your daily social media type software — the ramifications of having critical bugs could easily cost consumers millions of dollars overnight.
If the blockchain industry wants adoption, then project quality needs to be the number one focus for the engineering teams out there! The development process for blockchain software, at least first-layer solutions where security is paramount, should be standardized across the industry.
Logos is leading the way by adhering to a best-in-class software development process, and we hope other projects will collaborate with us to change blockchain development for the better!
If you’d like to keep up with what we are doing:
Follow us: Twitter | Discord | Telegram | Reddit
Read: White paper | Logos website
