Encrypting Streams in Go

Michael Turner
May 7, 2019 · 3 min read
Image for post
Image for post

At Blend, we deal with highly sensitive consumer financial data. We use several data stores — Postgres, MongoDB, CockroachDB, and Etcd — all of which need to be backed up. While MongoDB and Postgres give us prebuilt tools for encrypting backups, Etcd and CockroachDB do not. Our standard practice is to encrypt these backups before storing them. This became more challenging as our backups grew.

Encrypting backups in memory

At the beginning the backups were small, and we were able to use ’s transit features to encrypt them. As usage increased, we stored more data, and these backups grew. For small files, we could encrypt the data in memory using the Go standard library. While these files don’t get huge (~2 GB at most), we didn’t allocate enough memory to the backup job containers to encrypt them — the containers ran out of memory and were killed. We needed a way to encrypt our backups without reading entire files into memory.

Encrypting large backups using streaming encryption

Instead of encrypting large backups in memory, we can do it over a stream. Thankfully, Go provides a stream cipher in the package. We decided to use an AES cipher with the stream cipher in counter mode. With this, we no longer needed to read all the data into memory — we could just read in data from an io.Reader and write to an io.Writer chunk-by-chunk using our cipher’s XORKeyStream method.

But why do the work of looping and copying data when we can take advantage of built-in functions like io.Copy? We thought it would be nice to wrap a plaintext stream with our encrypter also implementing io.Reader so we could just “read” our ciphertext immediately.

Skipping the initialization steps, our code ended up looking like this:

type StreamEncrypter struct {
Source io.Reader
Block cipher.Block
Stream cipher.Stream
Mac hash.Hash
IV []byte
func (s *StreamEncrypter) Read(p []byte) (int, error) {
n, readErr := s.Source.Read(p)
if n > 0 {
s.Stream.XORKeyStream(p[:n], p[:n])
err := writeHash(s.Mac, p[:n])
if err != nil {
return n, ex.New(err)
return n, readErr
return 0, io.EOF

What benefits does the StreamEncrypter type give us? For one, encrypting a file becomes:

encrypter, _ := NewStreamEncrypter(key, reader)
io.Copy(file, encrypter)

Not only is this simple, but it accomplishes our original goal of not having to read the entire file into memory to encrypt it. Looking at the line XORKeyStream(p[:n], p[:n]), we’re even reusing the space in our buffer. We never need to allocate more space!

Since we’re operating on interfaces instead of concrete types like os.File, our data can come from anywhere (e.g. an HTTP request) and be written anywhere (e.g. a file). This is particularly helpful since we send Etcd backups over the network to our backup job.

In addition to encrypting the data, we also need to verify that the ciphertext hasn’t been tampered with or corrupted. We do this using an HMAC, which computes a running hash of our cipher.


Encrypting data in memory worked for small backups, but it didn’t scale with our dataset. Stream ciphers are the best way to get around this to encrypt larger backups. Go provides useful library functions that should be taken advantage of, but they aren’t always easy to use . With a small amount of code, we can take advantage of these library functions and simplify the process. You can find our whole solution, as well as the decryptor, .

Interested in joining Blend in San Francisco or New York? and would love to talk to you!

Image for post
Image for post

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store