Explaining common I/O Patterns in Go

In this article, I like to introduce common I/O (Input & Output) patterns (a few advanced). Many beginners coming from Python or C++ backgrounds get confused with I/O handling in Go, mainly with the terminology of Readers & Writers. I want to clarify the concepts in plain English in this article so that one can use elegant Go I/O API in their applications.

Before you go ahead, are you using Google Docs to store your favourite ChatGPT prompts ? Then, checkout this cool prompt manager called Vidura for free:

Vidura AI - A prompt engineering platform to bake your prompts

Let’s see common use cases which a developer needs for their day-to-day life. Note: All the code snippets in this article are tested with Go version 1.18.

#1 Write to standard output

The most common example every Go programming tutorial teaches you:

But, no one tells you the above program is a simplified version of this one:

What’s going on here? We have an extra package import called osand usage of a method called Fprintln from `fmt` package. The Fprintln method takes an `io.Writer` type and a string to write into a writer. The `os.Stdout` satisfies `io.Writer` interface.

This example is excellent and can extend to any writer besides `os.Stdout`.

#2 Write to a custom writer

You learned how to write data to `os.Stdout`. Let’s create a custom writer and store some information there. One can do it by initializing an empty buffer and writing content to it.

This snippet instantiates an empty buffer and uses it as the first argument (io.Writer) for Fprintln method.

We can write data into an io.Writertype. Don’t confuse it with “A writer means the one who writes information, wrong in Go.”

#3 Write to multiple writers at once

Sometimes, one needs to write a string into multiple writers. We can easily do that using the MultiWriter method from the io package.

In the above snippet, we create two empty buffers called foo and bar. We are passing these writers into a method called `io.MultiWriter`to get a combined writer. The message Hello Medium will be written into both foo and bar at the same time internally.

#4 Create a simple reader

An I/O reader helps hold information that can retrieve with API

Go provides io.Readerinterface to implement an IO reader. A reader does not read but provides data for others. It is a temporary store for information with many methods like WriteTo, Seek, etc.

Let us see how to create a simple reader from a string.

This code creates a new reader using `strings.NewReader` method. The reader has all the methods of `io.Reader` interface. We use `io.ReadAll` to read content from the reader, and it returns a slice of bytes. Finally, we print it to the console.

Note: `os.Stdin` is a commonly used reader for collecting standard input.

#5 Read from multiple readers at once

Similar to io.MultiWriter, we can also create an io.MultiReader to read data from multiple readers. The data is collected sequentially in the order of readers passed to `io.MultiReader`. It is like gathering information from various data stores at once but in the given order.

The code is straightforward, creating two readers called foo and bar and trying to create a `MultiReader` from them. We can use `io.Readall` to read `MultiReader` content like a regular reader.

Now that we learned about readers and writers let us see an example of copying data from a reader into a writer. We see techniques for copying data next.

Note: Don’t use `io.ReadAll` for big buffers, as they can choke memory.

Copying data from a reader to writer

To refresh your understanding of definitions again:

Reader: From whom I can copy data

Writer: To whom I can write data to

These definitions make it easy to figure out that we need to load data from a reader (string reader) and dump it into a writer (like os.Stdout or a buffer). This copy process can happen in two ways:

1. Reader pushes data to a writer
2. The Writer pulls data from a reader

#6 Reader pushes data to a writer (copy variant 1)

This part explains the first copy variation, i.e., the reader pushes data into a writer. It uses the `reader.WriteTo(writer)` API.

In the code, we use the `WriteTo` method from a reader named rto write contents into the writer b. Finally, we verify the contents. In the following example, we see how a writer can proactively pull information from a reader.

#7 Writer pulls data from a reader (copy variant 2)

The method `writer.ReadFrom(reader)` is used by a writer to pull data from a given reader. Let’s see an example:

The code looks similar to the previous example. Whether you are working as a reader or a writer, you can pick variants 1 or 2 for copying data. Now comes the third variant, which is cleaner.

#8 Copy data from a reader to writer (copy variant 3, io.Copy)

The `io.Copy` is a utility function that allows one to move data from a reader to a writer without worrying about which variant to pick from above.

Let’s see how it works:

The first argument of `io.Copy` is Writer (destination), and the second is Reader (source) for copying data. The rest of the example looks similar to previous copy variants.

Whenever someone writes data into a writer, you want to have information available to read by the corresponding Reader. There comes the concept of piping.

#9 Create a data tunnel with io.Pipe

The `io.Pipe` returns a reader and a writer pair, where writing data into a writer automatically allows programs to consume data from the Reader. It is like a Unix pipe.

From go doc,

Pipe creates a synchronous in-memory pipe. It can be used to connect code expecting an io.Reader with code expecting an io.Writer.

You must put writing logic into a separate go-routine (from the main go-routine) because the pipe blocks the Writer until the data is read from the Reader, and the Reader is also blocked until the Writer is closed.

Here is what I am talking about:

The code creates a pipe reader and pipe writer. We spin up a go-routine to write some information into pipe writer and close it. We read data from the pipe reader on Line no: 19, using `io.ReadAll` method. The program will run into a deadlock if you don’t spin up a separate go-routine (to write or read, either for one operation).

We discuss a more practical use case of piping in the next section.

#10 Capture stdout of a function into a variable with io.Pipe, io.Copy and io.MultiWriter

Let’s say we are building a CLI application. As part of that process, we should tap into the standard output generated by a function(to console) and capture the same information into a variable. How can we do that? We can use the techniques discussed above to create a solution.

The above program works this way:

1. Create a pipe that gives a Reader and Writer. If you write something into a pipe writer, Go copies it to the pipe reader.
2. Create a io.MultiWriterwith os.Stdoutand custom buffer b
3. foo (as a go-routine) will write a string into pipe writer
4. The io.Copywill then copy content from the pipe reader into multi-writer
5. The os.Stdoutwill receive the output as well as your custom buffer b
6. The contents are now available in b

Using the iopackage in Go, we could manipulate data as we have seen in these patterns. I hope you got an easy explanation about why Go developers use Readers and Writers. I hope you enjoyed the article.

Furthermore, I also authored a book in Go for cooking REST API, if you are interested:

Hands-On RESTful Web Services with Go: Develop elegant RESTful APIs with Golang for microservices…

References:

io

In Go, how do I capture stdout of a function into a string?