Responding to Bug Reports Efficiently

Isabella Dell
Aug 28, 2017 · 4 min read

During the last few days the Lisk Platform experienced an influx of patch releases one after the other. In the name of true transparency towards our stakeholders in this blog post I will describe the series of events which lead to the release of four patches last week. It has been an intense week for the Lisk Core team and we’re very thankful for the delegates for responding rapidly.

August 19th — 14:43 UTC

Community Member vekexasia contacted the Lisk Team with a bug report regarding blocks receipt processing. This bug had the potential to rollback a significant amount of blocks, and possibly irreversible damage the blockchain on the targeted nodes. The Lisk team mobilized immediately to tackle the problem with the bug report in hand.

August 20th — 01:00 UTC

Patch release Lisk Core 0.9.4 patch release was successfully deployed to the network resolving the potential threat of malicious block rollbacks. Due to the severity of the threat, this release went straight to Mainnet without proper regression testing. We quickly realized the patch update had introduced a new regression into the ecosystem.

August 20th — 07:12 UTC

Delegates on Lisk.Chat reported various failures in block processing and chain splits occurring due to double forging. As previously, the Lisk team sprang into action to investigate the root cause of the problem. During the refactor, some of the logic for fork detection was no longer working as expected, causing inconsistency in which block is chosen by the network when a delegate double forges.

August 20th — 09:13 UTC

Root cause of the regression was identified and a fix was implemented. However, this fix was not working in all cases, and further research has to be performed on the nature of the issue.

August 20th — 19:15 UTC

With new tests in hand for the failing logic, and a fix for the logic, the Lisk team released Lisk Core 0.9.5 patch release to the network. Again, while it was less than ideal to release directly to mainnet, the benefit of restoring stability to block receipt was far greater than the potential issues (i.e we skipped public testing in order to mitigate potential .

August 21st — 12:00 UTC

While reviewing the code to the Lisk Core 1.0.0 branch, the core team identified a missing validation in the blocks verification. While the rest of the verification checks would account for this particular validation, it was deemed critical to reinstate the validation. As previous critical threats, this release needed to be implemented as quickly as possible in the event a malicious actor decided to use it.

August 21st — 19:24 UTC

The Lisk Core team completed the implementation of the validation and subsequent tests that ensured the behavior was performing as expected. This lead to the release of Lisk Core 0.9.6 which changed the minVersion parameter of the network to remove all peers on Lisk Core 0.9.5 or lower. This network cut cleaned up the old, vulnerable versions of Lisk.

August 21st — 23:43 UTC

A buried bug in multi signatures code was discovered by the Lisk Core 1.0.0 release refactor process. This bug had the potential to crash nodes malicious and was therefore deemed critical enough to mobilize to patch immediately. With the fix and tests in hand, the Lisk team released Lisk Core 0.9.7 to Mainnet patching this potential crash exploit in the code.

Why Mainnet First?

It was vital to protect the security of the network, delegates, exchanges and all end-users. We greatly dislike releasing directly to mainnet; but in the circumstances of a potential network outage or attack, these measures have to be taken.

I personally want to thank all of the Delegates who stepped up through the issues and were available to update their nodes almost immediately. Its through their diligence and acceptance of our patches that we are able to move forward as an ecosystem.

LiskHQ as a whole would also like to thank vekexasia for reporting the block receipt issue to us, and for the proof of concept code that allowed us to fix the issue.


About the Author

Isabella Dell is System Architect for Lightcurve and Project Lead on Lisk Core and played a pivotal role in the development process since the launch of Lisk’s Mainnet. She brings significant enterprise experience with managing large scale systems and a deep understanding of blockchain technology. Her focus in the blockchain space is on the improvement of Lisk’s Blockchain technology and distilling all of the data into easily understood bytes.


Contact Details

Twitter: @Isabella_Lisk
Email: isabella@lisk.io


If you enjoyed reading this, please log in and show your support by clicking the “clapping” button below. This will help to share the story with others.

)

Isabella Dell

Written by

Contributor to the Lisk community and former System Architect for Lisk

Lisk Blog

Lisk Blog

Blockchain Application Platform

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade