Using io.Reader to simulate an arbitrary-sized file in Golang

Level up your file tests

Photo by Mikołaj on Unsplash

Do you have any tests that involve processing a massive file? In a previous article, we explored how you can take advantage of io.TeeReader to send arbitrary-sized data to multiple places with constant memory usage. But how can we test this function without having to actually create a massive file?

We can implement our own implementation of io.Reader to instantly create a fake read-only file of any size, that occupies zero bytes of disk space.

First of all, how can we create a Reader? We just need to implement this method on a struct, and then you can pass that struct to any method that expects an io.Reader:

Read(p []byte) (n int, err error)

This says that we just need to make a Read method that will fill as many bytes of data as it can into p, and then return a value n saying how many bytes we read, and an error if there was one.

Knowing this, we can go ahead and tell our reader to simply cycle through the letters a-z forever, until there are no bytes left to read.

Great, now lets test it using our hashAndSend method from a previous article:

The following content of the file, and the hash of the file, is printed to standard output:

abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuv
wxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqr
stuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmn
opqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij

dba4a6315b76548b7a4dd079ef6aa29a7b34fa8b92c11668473441715c5f0af5

Great, what if we want to test a big file, will it crash my machine? Lets see if I can handle hashing a 100gb file: (I advise removing the fmt.Println calls if you try this)

If anyone was wondering, here’s the sha256 hash of 100GB of abcd...xyz:

162e7af51048e81bec4a0b20851f181ff83b999f718baa2d839aa2412ab34406
took 6m41.2434189s to hash!

We can also see that my machine did not crash. If you’re not using io.Reader in your code, try it out! It is extremely powerful.

Go Playground link: https://go.dev/play/p/DkCA1hdYopo