Learn Go By Building a Command-Line App

Building a Redis sorted set copier in Go

Learning by doing is the best way to pick up any new technology. This tutorial will provide a quick start for you, this means that we will not be getting into the details regarding the syntax, installation, set up, etc. and instead focus on the important elements while building a real and useful app. There are a lot of tutorials already available for helping you set up everything, understand the syntax, etc, so links are provided to external resources wherever possible.

Why build a Command-Line App in Go

• Compiles to a single static binary: With Go, you can easily provide a single static binary that contains your whole application or CLI for your chosen platform. To target a different CPU or OS, you can just pass in an environmental override while building the application. Here’s a binary for Windows, 64-bit Linux, and Raspberry Pi:

GOOS=windows go build -o cli.exe
GOOS=linux go build -o cli
GOARCH=armv7 GOOS=linux go build -o cli-rpi

That’s it — and there are more platforms available too — like FreeBSD. You won’t need to install any dependencies and the final output binary can be tiny.

• Consistent style: Go is an opinionated language and while there may be some differences between which editors a project prefers — you will encounter a consistent standard for styling, formatting and build tools. Something like Node.js could involve any number of “task runners” or “transpilers” — or a more esoteric flavour of the language like TypeScript or CoffeeScript. Go has a consistent style and was deliberately designed to be unambiguous. This makes it attractive to contributors and easy for on-boarding.
• Fast on every platform: A statically compiled Go binary is super fast to load — compared to Node.js. For instance: Node.js on a single-core Raspberry Pi can take 1.5–3.0 seconds to load before executing any code.

Setup and Installation

Describing this is beyond the scope of this article, but here are some links to help you get ready with everything. In fact, the best tutorial is still on the official Golang website. You can either go through all of them straight away or skip to the next section in this tutorial after installing Go and come back later to study the remaining resources in detail:

1. https://golang.org/doc/install: Installation and basic configuration
2. https://tour.golang.org/welcome/1: A tour of Go, an interactive tutorial where you can type your code and execute the same within the browser.
3. https://golang.org/doc/editors.html: List of editor/IDE plugins. I use Visual Studio Code plugin, here is the link to it https://code.visualstudio.com/docs/languages/go

What are we going to build — The Problem

You must have worked on Redis or some kind of cache system. Sometimes, we want to replicate a scenario which happens in production in our local development system for the purpose of debugging or fixing any production issue.

A few days back, we noticed a weird behaviour in one of our production systems, digging deeper we found out that it had something to do with the Cache Management Service. The Cache Management Service was an application which used Redis as its backing store. After multiple attempts of trying to reproduce the issue with dummy data on our local systems, we were not able to simulate the same behaviour. So, the only choice left with us was to copy the data on our production Redis instance into our local instance and then reproduce the bug.

We just needed to copy a single Sorted Set from the production cluster which contained all the data related to the bug. This was a very simple problem and a many shell, python, etc scripts are already available on Github for this purpose. So, I could have easily used any of those scripts, modified them and used it for my purpose, or else, I could have easily written a script in Node.js myself. But we choose to Go instead, to build a tool which could be used by anyone, and who knows what next feature could be added to this tool in the future.

We are going to build a simple and most basic command-line tool which can copy a single sorted set from production Redis cluster to your local Redis instance.

Project Structure

We would try to organize our code just like most professional Go apps are built. We would be following Domain-Driven Design and Hexagonal Architecture in the most friendly and basic way possible for this tutorial. You may want to go through this blog later which explains what a Hexagonal Architecture is, once you are somewhat comfortable with Go.

Project Structure for our App

We just need 2 source files to perform this task, copy.go which is at the project root is the brain of the app, while main.go inside the cmd folder uses copy.go and wraps it inside a command-line binary interface. Each of these 2 files has a corresponding test case file. If you look at the Go standard library or most open source projects with a command-line interface, they would be organized in a similar manner. You can always go through the source code, it is available on Github.

Code — The Solution

In this section, we would explain each of the 2 source files and their corresponding test cases. Please read them in sequence, we will try to understand Go specific concepts on the way.

copy.go

As explained earlier, this is the brain of the app which is responsible for copying a sorted set from production to your local environment. Let us look at the code.

package rediscopy
import (
"github.com/go-redis/redis"
)
/*
Purpose of this project is to safely copy all keys from Redis running in production to your local one. This would be useful when you cannot take a dump from production for some reason. Even if you are able to get that dump, this tool can help you do it in the background in an automatic way.
Version 0.1 : Copy a sorted set, given a key in the command
*/

// Copier - Main struct which copies data
type Copier struct {
 server *redis.Client
 local  *redis.Client
}
/*
CopySortedSet - Copy a sorted set from production to local instance. Right now copy everything in one instance, later it should be done in batches so that we do not put a lot of load on our Redis instance */
func (cp *Copier) CopySortedSet(sortedSetSource, sortedSetTarget string) error {
  vals, err := cp.server.ZRangeByScoreWithScores(sortedSetSource,      &redis.ZRangeBy{
Min:    "-inf",
Max:    "+inf",
Offset: 0,
Count:  0,
}).Result()
  if err != nil {
   return err
  }
  valPtr := make([]*redis.Z, len(vals))
  for index, val := range vals {
   valPtr[index] = &val
   _, err := cp.local.ZAdd(sortedSetTarget, valPtr[0]).Result()
   if err != nil {
    return err
 }
}
return nil}
// Constructor
func NewCopier(server, client *redis.Client) *Copier {
return &Copier{server, client}
}

1. First, we define the name of the package, in this case, it is “rediscopy”
2. Then we import the necessary dependencies, a Redis library in our case.
3. We define our copier struct, structs in Go are analogous to classes in Java. Read more about structs here. This struct contains the connection to your production and local Redis instance. Later we would see how can we provide the connection to this via the constructor mentioned in the last section.
4. Then, we have a method attached to this struct which performs the actual work “CopySortedSet”. It receives the keys for production and local instances. Fetches the key from production instance and stores it inside a variable named “vals” , this is an array which contains all the values in our sorted set.
5. Then, we iterate over the array and store everything in our local instance.
6. We can observe some features of Go here, variable vars is dynamically initialized, read more on variables in Go here.
7. Idiomatic Error Handling in Go: if err!=nil {…..}
8. Pointers in Go
9. Inbuilt function: len
10. Inbuilt function: make
11. For Range loops in Go
12. In the end, we have a constructor for initializing our struct.

copy_test.go

Much like most of the things in Go, the test framework is inbuilt in the standard library, you do not need to look for any third-party library, although you can do that if you want.

package rediscopy
import (
"testing"
"github.com/go-redis/redis")
func getLocalConnection() *redis.Client {
return redis.NewClient(&redis.Options{
Addr:     "localhost:6379",
Password: "", // no password set
DB:       0,  // use default DB
})
}
func TestShouldCopy(t *testing.T) {
copier := NewCopier(getLocalConnection(), getLocalConnection())
err := copier.CopySortedSet("deltas", "deltas-copy")
if err != nil {
t.Error(err)
}
}

This is an actual integration test which would copy the elements in one key to another key within the same Redis instance(local). You should write mocks if you want to write unit tests. But just for the simplicity of this tutorial, I have written unit tests here. The other thing to notice here is that we have used the “testing” package from the standard library. Read more on testing in Go here.

main.go

package main
import (
"errors"
"flag"
"fmt"
"strings"
"github.com/go-redis/redis"
rediscopy "github.com/naqvijafar91/redis-copy"
)
func main() {
sourceAddress := flag.String("s", "localhost:6379", "source address")
sourcePassword := flag.String("sp", "", "source password")
sourceDb := flag.Int("sdb", 0, " Source Database number")
destinationAddress := flag.String("d", "localhost:6379", "destination address")
destinationPassword := flag.String("dp", "", "source password")
destinationDb := flag.Int("ddb", 0, " Source Database number")
sourceSetName := flag.String("skey", "", "Source Key")
destinationKeyName := flag.String("dkey", "", "Destination key")
flag.Parse()
// Now we need to make sure that we do not copy anything into production, we want a safety hook
// Therefore, we would have localhost and 127.0.0.1 as the only 2 allowed destination addresses
if !shouldAllowDestinationAddress(*destinationAddress) {
panic(errors.New(fmt.Sprint("Destination cannot be ", *destinationAddress, ". Make sure you are not copying anything to prod")))
}
copier := rediscopy.NewCopier(getConnection(*sourceAddress, *sourcePassword, *sourceDb),getConnection(*destinationAddress, *destinationPassword, *destinationDb))
// @Todo : Print and ask for confirmation before starting the work
err := copier.CopySortedSet(*sourceSetName, *destinationKeyName)
if err != nil {
panic(err)
}
fmt.Println("Key", *sourceSetName, "copied sucessfully")
}
func shouldAllowDestinationAddress(destinationAddress string) bool {
address := strings.Split(destinationAddress, ":")
if address[0] == "localhost" || address[0] == "127.0.0.1" {
return true
}
return false
}
func getConnection(address, password string, db int) *redis.Client {
return redis.NewClient(&redis.Options{
Addr:        address,
Password:    password, // no password set
DB:          db,       // use default DB
ReadTimeout: 9999999999999999,
})
}

Here are the key observations:

1. It belongs to package main
2. Imports the necessary dependencies along with our “rediscopy” package declared earlier.
3. Takes in the required parameters from the command line using the flag package.
4. After parsing the required parameters, creates a new copier instance defined in copy.go with 2 connections, one for source, one for the destination, using the constructor function. There are no constructors in Go, but it is considered a best practice to create a function beginning with New… to act as a constructor.
5. The most critical part of this application is preventing a copy from local to production, this can happen if you pass production parameters in the “destinationAddress” and local parameters in the “sourceAddress” via the command line. This is a mistake which could happen anytime by anyone, so our application should be intelligent enough to prevent this behaviour as this would result in corrupting our production cache. The function called “shouldAllowDestinationAddress” is responsible for performing this check. For now, we will only allow localhost or 127.0.0.1 in our destination port, we can add more intelligence according to the environment, but this is good enough for this tutorial.
6. Finally, we execute the copier’s “CopySortedSet” function and print the results.

main_test.go

package main
import "testing"
func TestDestinationSafetyHook(t *testing.T) {
if !shouldAllowDestinationAddress("localhost:6379") {
t.Error("Should have passed")
}
if shouldAllowDestinationAddress("xyz.bigactualprod.omega-cloud.com:6379") {
t.Error("Should have failed")
}
}

This contains the tests for our filter mentioned above, we would generally put these kinds of logics in a separate file, but for the sake of this tutorial, we can write this logic inside the main file and its corresponding test file.

Where to go from here? - Roadmap

Go is an amazing language that should be part of every developer toolset, and the best thing is that it has a very small learning curve, any developer can become productive within a week as there are few concepts to understand. To help you with your journey, I am listing down the next steps :

1. Go through the official website to get an overview of everything.
2. Follow some basic tutorials for understanding HTTP Web Services in Go.
3. Head for some deeper understanding in Go, try to understand interfaces, pointers, slices, and maps. Understand how slices are different from arrays.
4. Learn go-routines, channels, sync.WaitGroup, sync.Mutex.
5. Understand where to use channels and where to use sync.Mutex, both have separate use cases.
6. Understand testing and benchmarking using the standard library.
7. Read open source code bases to understand best practices.
8. Pick any project, build it, and learn the way of the Gopher.

Source code for this application is available here:

https://github.com/naqvijafar91/redis-copier