Taming the monolithic database

Photo by j zamora on Unsplash

GoChain has been successful in increasing blockchain performance through block size reduction, significant code optimizations, and Proof of Reputation consensus, however, that causes another issue—more transactions per second means that the storage size grows at the same rate.

The storage layer originally used the go-ethereum implementation where all data is stored in a single goleveldb database. Because goleveldb is a log-structured merge tree, there wasn’t an easy way to offload old key/value pairs into cold storage because writes are spread across numerous files for the entire keyspace.

The GoChain testnet regularly needed to be reset because disks would fill and…


In the last post we looked at fast, primitive encoding using strconv but in this post we’ll take a higher level approach by using templates with the fmt package. The fmt package builds on top of our knowledge of primitive formatting options but packages it in a way that’s easier to use.

This post is part of a series of walkthroughs to help you understand the standard library better. While generated documentation provides a wealth of information, it can be difficult to understand packages in a real world context. This series aims to provide context of how standard library packages…


Formatting & parsing primitive values in Go is a common task. You probably first dipped your toes into the fmt package when you started Go, however, there’s a less commonly used package for basic formatting that’s more efficient and preserves compiler type checking.

The strconv package is built for speed. It’s great for when you need to handle primitive value formatting while minimizing allocations and CPU cycles. Understanding the package also gives you a better understanding of how the fmt package itself works.

This post is part of a series of walkthroughs to help you understand the standard library better…


When you need to squeeze every last bit or CPU cycle out of your protocol, one of the best tools is the encoding/binary package. It operates on the lowest level and is built for working with binary protocols. We previously looked at using encoding/json for text-based protocols but binary protocols can be a much better fit when two machines need to communicate quickly and efficiently.

Writing binary protocols seems like a daunting task but it is surprisingly simple once you understand a few concepts. …


For better or for worse, JSON is the encoding of the Internet. Its formal definition is small enough that you could write it on the back of a napkin but yet it can encode strings, numbers, booleans, nulls, maps, and arrays. Because of this simplicity, every language has a JSON parser.

Go’s implementation is a package called encoding/json and it allows us to seamlessly add JSON encoding to our Go objects. However, with its extensive use of reflection, encoding/json is one of the least understood packages even though it’s also one of the most used. …


So far we’ve covered working with raw byte streams and bounded byte slices but few applications simply shuttle bytes around. Bytes alone don’t convey much meaning, however, once we encode data structures on top of those bytes then we can build truly useful applications.

This post is part of a series of walkthroughs to help you understand the Go standard library better. While generated documentation provides a wealth of information, it can be difficult to understand packages in a real world context. This series aims to provide context of how standard library packages are used in every day applications. …


Photo by Tony Webster

Vendoring. Generics. These are seen as big issues in the Go community but there’s another issue that’s rarely mentioned — application package layout.

Every Go application I’ve ever worked on appears to have a different answer to the question, how should I organize my code? Some applications push everything into one package while others group by type or module. Without a good strategy applied across your team, you’ll find code scattered across various packages of your application. We need a better standard for Go application design.

I suggest a better approach. By following a few simple rules we can decouple…


In the previous post we covered byte streams but sometimes we need to work with bounded, in-memory byte slices instead. While working with a list of bytes seems simple enough, there are a lot of edge cases and common operations that make using the bytes package worthwhile. We’re also going to lump in the strings package in this post since its API is nearly identical although for use with strings.

This post is part of a series of walkthroughs to help you understand the Go standard library better. While generated documentation provides a wealth of information, it can be difficult…


Go is a programming language built for working with bytes. Whether you have lists of bytes, streams of bytes, or individual bytes, Go makes it easy to process. From these simple primitives we build our abstractions and services.

The io package is one of the most fundamental packages within the standard library. It provides a set of interfaces and helpers for working with streams of bytes.

This post is part of a series of walkthroughs to help you understand the standard library better. While generated documentation provides a wealth of information, it can be difficult to understand packages in a…


How I organize my tests in Go

The Purpose of Tests

People argue over testing style, whether to use TDD or BDD, or whether tests are even useful at all. Before I get into how I structure my tests in Go, I should explain how I see tests.

Tests should be 2 things:

  1. Self-contained
  2. Easily reproducible

That’s it. They should be self-contained so that changing one part of your test suite doesn’t drastically affect another part. They should be easily reproducible so that someone doesn’t have to go through multiple steps to get their test suite running the same as mine.

With that explanation, here are some of my own rules…

Ben Johnson

Writing databases and distributed systems in Go.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store