Writing Modular Go Programs with Plugins

Tucked among the multitude of significant features introduced in Go version 1.8 is a new Go plugin system. This feature let programmers build loosely coupled modular programs using packages compiled as shared object libraries that can be loaded and bound to dynamically at runtime.

This is a big deal! You see, developers of large system software in Go have, inevitably, come across the need to modularize their code. We have relied on multiple out-of-process designs such as OS exec calls, sockets, and RPC/gRPC (and so on) to achieve code modularization. While these approaches can work well, in many contexts however, they were ends to a mean to address Go’s previous lack of a native plugin system.

In this writeup I explore the implications of creating modular software using the Go plugin system. The text covers what you will need to get started, provides a fully functional example, and discussion on design and other implications.

The Go plugin

A Go plugin is a package compiled using the -buildmode=plugin build flag to produce a shared object (.so) library file. The exported functions and variables, in the Go package, are exposed as ELF symbols that can be looked up and be bound to at runtime using the plugin package.

The Go compiler is capable of creating C-style dynamic shared libraries using build flag -buildmode=c-shared as covered in my previous writeup.

Restrictions

As of version 1.8, the Go plugin only works on Linux. This most likely will change in future releases given the level of interest in this feature.

A simple pluggable program

The code presented in this section shows how to create a simple greeting program that uses plugins to expand its capabilities to print greetings in several languages. Each language is implemented as a Go plugin.

See Github repo — https://github.com/vladimirvivien/go-plugin-example

The greeting program, greeter.go, uses plugins in packages ./eng and ./chi to print English and Chinese greetings respectively. The following figure shows the directory layout of the program.

Greeter plugin example layout

First, let us examine source eng/greeter.go which, when executed, will prints a message in English.

File ./eng/greeter.go

package main

import "fmt"

type greeting string

func (g greeting) Greet() {
    fmt.Println("Hello Universe")
}

// exported as symbol named "Greeter"
var Greeter greeting

The source above represents the content of a plugin package. You should note the followings:

• A plugin package must be identified as main.
• Exported package functions and variables become shared library symbols. In the above, variableGreeter will be exported as a symbol in the compiled shared library.

Compiling the Plugins

The plugin packages are compiled using the following commands:

go build -buildmode=plugin -o eng/eng.so eng/greeter.go
go build -buildmode=plugin -o chi/chi.so chi/greeter.go

This step will create plugin shared library files ./eng/eng.so and ./chi/chi.so respectively. We can verify the type of the generated files as dynamic shared object files with the following command:

$> file chi/chi.so
chi/chi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=20c823671954b4716d8d945c3b333fc18cdbf7fe, not stripped

Using the Plugins

Plugins are loaded dynamically using Go’s plugin package. The driver (or client) program ./greeter.go makes use of the plugins, compiled earlier, as shown below.

File ./greeter.go

package main

import "plugin"; ...


type Greeter interface {
	Greet()
}

func main() {
	// determine plugin to load
	lang := "english"
	if len(os.Args) == 2 {
		lang = os.Args[1]
	}
	var mod string
	switch lang {
	case "english":
		mod = "./eng/eng.so"
	case "chinese":
		mod = "./chi/chi.so"
	default:
		fmt.Println("don't speak that language")
		os.Exit(1)
	}

	// load module
	// 1. open the so file to load the symbols
	plug, err := plugin.Open(mod)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	// 2. look up a symbol (an exported function or variable)
	// in this case, variable Greeter
	symGreeter, err := plug.Lookup("Greeter")
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	// 3. Assert that loaded symbol is of a desired type
	// in this case interface type Greeter (defined above)
	var greeter Greeter
	greeter, ok := symGreeter.(Greeter)
	if !ok {
		fmt.Println("unexpected type from module symbol")
		os.Exit(1)
	}

	// 4. use the module
	greeter.Greet()

}

As you can see from the previous source code, there are several steps involved in dynamically loading the plugin module to access its members:

• Select the plugin to load based on os.Args and a switch block.
• Open plugin file with plugin.Open().
• Lookup the exported symbol "Greeter" with plguin.Lookup("Greeter"). Note the symbol name matches the name of the exported package variable defined in the plugin module.
• Assert that the symbol is of interface type Greeter using symGreeter.(Greeter).
• Lastly, calls the Greet() method to display the message from the plugin.

You can find further detail about the source code here.

Running the program

When the program is executed, it prints a greeting in English or Chinese depending on the command-line parameter passed as shown below.

> go run greeter.go english
Hello Universe
> go run greeter.go chinese
你好宇宙

The exciting thing is that the capability of the driver program is expanded at runtime, by using the plugins, to display a greeting message in different languages without the need to recompile the program.

Modular Go Program Design

Creating modular programs with Go plugins requires the same rigorous software practices as you would apply to regular Go packages. However, plugins introduce new design concerns that are amplified given their decoupled nature.

Clear affordances

When building pluggable software system, it is important to establish clear component affordances. The system must provide clean a simple surfaces for plugin integration. Plugin developers, on the other hand, should consider the system as black box and make no assumptions other than the provided contracts.

Plugin Independence

A plugin should be considered an independent component that is decoupled from other components. This allows plugins to follow their own development and deployment life cycles independent of their consumers.

Apply Unix modularity principles

A plugin code should be designed to focus on one and only one functional concern.

Clearly documented

Since plugins are independent components that are loaded at runtime, it is imperative they are well-documented. For instance, the names of exported functions and variables should be clearly documented to avoid symbol lookup errors.

Use interface types as boundaries

Go plugins can export both package functions and variables of any type. You can design your plugin to bundle its functionalities as a set of loose functions. The downside is you have to lookup and bind each function symbol separately.

A tidier approach, however, is to use interface types. Creating an interface to export functionalities provides a uniform and concise interaction surface with clear functional demarcations. Looking up and binding to a symbol that resolves to an interface will provide access to the entire method set for the functionality, not just one.

New deployment paradigm

Plugins have the potential of impacting how software is built and distributed in Go. For instance, library authors could distribute their code as pre-built components that can be linked at runtime. This would represent a departure from the the traditional go get, build, and link cycle.

Trust and security

If the Go community gets in the habit of using pre-built plugin binaries as a way of distributing libraries, trust and security will naturally become a concern. Fortunately, there are already established communities coupled with reputable distribution infrastructures that can help here.

Versioning

Go plugins are opaque and independent entities that should be versioned to give its users hints of its supported functionalities. One suggestion here is to use semantic versioning when naming the shared object file. For instance, the file compiled plugin above could be named eng.so.1.0.0 where suffix 1.0.0 repents its semver.

Gosh: a pluggable command shell

I will go ahead and plug (no pun, really) this project I started recently. Since the plugin system was announced, I wanted to create a pluggable framework for creating interactive command shell programs where commands are implemented using Go plugins. So I created Gosh (Go shell).

Learn about Gosh in this blog post

Gosh uses a driver shell program to load the command plugins at runtime. When a user types a command at the prompt, the driver dispatches the plugin registered to handle the command. This is an early attempt, but it shows the potential power of the Go plugin systems.

Conclusion

I am excited about the addition of shared object plugin in Go. I think it adds an important dimension to the way Go programs can be assembled and built. Plugins will make it possible to create new types of Go systems that take advantage of the late binding nature of dynamic shared object binaries such as Gosh, or distributed systems where plugin binaries are pushed to nodes as needed, or containerized system that finalize assembly at runtime, etc.

Plugins are not perfect. They have their flaws (their rather large sizes for now) and can be abused just like anything in software development. If other languages are an indication, I can predict plugin versioning hell will be a pain point. Those issues aside, having a modular solutions in Go makes for a healthier platform that can strongly support both single-binary and modularized deployment models.

As always, if you find this writeup useful, please let me know by clicking on the ♡ icon to recommend this post.

Also, don’t forget to checkout my book on Go, titled Learning Go Programming from Packt Publishing.