Plugins with Go

Published in

ProFUSION Engineering

11 min readMar 18, 2024

How to use Go’s standard library to load plugins to your application and allow users to expand your app’s functionality.

What are plugins

In short, plugins are pieces of software that can be loaded by other pieces of software. They are normally used to expand an application’s functionality and look, even by developers who don’t necessarily work directly with the base application.

Most likely you have used plugins in your day-to-day life, maybe under a different name, such as extensions or add-ons. The most common example is VSCode extensions. You probably used VSCode at some point, right? After all, it’s the most popular text editor for programmers in the market. If you have, you must agree that VSCode, by itself, is a text editor, not an IDE. Its base functionality is very simple, with little support for functionality often seen in IDEs, such as debugging, autocomplete and test navigation. However, through the editor’s extension marketplace, you can find all kinds of plugins for supporting these features and more. Plugins, in fact, have become one of the main selling points for the editor, prompting tool developers to focus their efforts on making plugins exclusively for this editor, sometimes going beyond the realm of coding itself, like Figma has done.

For VSCode, your plugins are written in JavaScript, but there are usages of Go plugins in the wild. One such usage is Terraform, the Cloud Provider Infrastructure as Code service, which allows users to write plugins for its tooling, allowing them to interact with multiple Cloud Providers (AWS, GCP, Azure…).

Another example would be Kong, the API gateway service, which allows developers to tap into the request being received before it goes forward to the underlying services by using plugins written in different languages, including Go.

Disclaimer

In this article, we assume you have, at least, a basic understanding of the Go language. If you don’t have it, we recommend you first take a tour of Go, then come back.
Some features in this code require you are using, at least, Go version 1.21.0
The plugin feature of Go is not yet supported in Windows machines. If you are using one, we recommend you follow this tutorial using WSL.
The produced code from this article can be found on this Github Repository.

Infrastructure of a plugin

We can divide the plugin infrastructure into 3 parts: protocol/API, implementation and plugin loader. Please note that this division is not official or set on paper, is just what is normally seen in the wild.

Protocol/API

Protocols are definitions and defaults we arbitrarily set so we can communicate between parts cleanly. And just as with any protocol, we need to set how our plugins and the base application will communicate between them. For this, we can use different methods, from a simple document explaining what we’re expecting to a library of interfaces (the programming ones, as in class foo implements bar). As long as the plugin implementation follows these guidelines, the application will be able to call plugin code.

Implementation

We need some plugin code that implements what is set by the protocol. That is, we need to implement the expected functions and variables within our plugin code so they may be called by the host application.

Just a simple reminder that the plugin code is not limited to these implementations.

Plugin loader

This is the part that needs to be implemented by the host application. It has 2 responsibilities: find the plugins and load its functions in the code.

Plugins are external to the project of the host application, therefore we need a way to find all plugins for this application. We could simply define a fixed folder in the file system where we could dump all the plugins, but it is preferable to allow users of the application to point to their plugins, using configuration files, or even both.

After having all plugins installed, we need to access their API during the application. This is normally done with hooks: parts of the runtime where plugins (or parts of them) are called. Keeping VSCode as an example, one such hook would be “when a file loads”, so plugins can use this hook to catch the loading file and run based on that. Which hooks are implemented and when is intrinsically linked with the application’s logic and, therefore, very specific to it.

What are we building

Well, there is no better way to learn in programming than getting your hands dirty. So, let’s build a simple application that uses plugins.

What we’re going to build is a plugin-based HTTP redirection service. This is a simple HTTP service that listens to requests in a port and redirects them to another server, passing the response down to the original client. With this service, we can tap into the request and make changes to it. For this example, let’s get the request to print it, of course using a plugin.

As for the plugin loading part, we’re using a library as the protocol and a configuration file for locating the plugins.

Making the plugin protocol

First, let’s define our plugin protocol. For this, we’re going to define a go library module.

Before we define this module, let’s define the application module:

# From a folder you want to keep the project:
mkdir http-redirect
cd http-redirect
go work init
go mod init github.com/<your_github_username>/http-redirect
go work use .

The name of the application is, of course, open to your imagination. We’re going to use go workspace because we’re going to interact with multiple modules. To see more about this, check the docs.

Now we can actually create our library module:

# From http-redirect
mkdir protocol
cd protocol
go mod init github.com/<your_github_username>/http-redirect/protocol
go work use . # Add new module to workspace

Now let’s touch some files and your file tree should look something like this:

We’re going to be doing work in protocol.go. What we want in our protocol is a function to be called for every request. So we’re implementing a function called “PreRequestHook” for our plugins. This looks something like this:

// protocol.go
package protocol

import "net/http"

// Plugins should export a variable called "Plugin" which implements this interface
type HttpRedirectPlugin interface {
  PreRequestHook(*http.Request)
}

Simple enough, we just get a pointer (because we could change the request) to a http.Request type, which will be passed each HTTP request to our server. We’re using the standard library-defined types, but notice we could ask for different ones, depending on the application’s implementation.

That’s it! But do not be fooled by the simplicity of our example. For larger applications, this can be quite a large file, with different interfaces, default implementations, configurations and other shenanigans.

Implementing a plugin

Now that we have a protocol to follow, we can create our plugin and implement it.

Again, let’s create a new module for our plugin and a file for it.

# From http-redirect
mkdir log-plugin
cd log-plugin
go mod init github.com/<your_github_username>/http-redirect/log-plugin
go work use . # Add new module to workspace
touch plugin.go

And your file tree should now look like this:

And let’s write our plugin! First, let’s create a function to print our request.

// log-plugin/plugin.go
package main

import (
  "log/slog"
  "net/http"
  "net/http/httputil"
)

func logRequest(req *http.Request) {
  result, err := httputil.DumpRequest(req, true)
  if err != nil {
    slog.Error("Failed to print request", "err", err)
  }
  slog.Info("Request sent:", "req", result)
}

func logRequestLikeCUrl(req *http.Request) {
  panic("Unimplemented!")
}

func main() { /*empty because it does nothing*/ }

The unimplemented function here is made just to show we could add more functionality with more complex protocols, but we cannot configure it correctly as of now. We will not use it.

What we will use is the “logRequest” function. It simply dumps the request using golang’s stand library’s structured logging module. This does it for our functionality, but now we need to export our plugin so that it satisfies the protocol.

You may also notice we have a main function that does nothing. This is a requirement by the go compiler because of some of its features, which require an entry point. Although the main function exists in this compiled package, it may not be called as an executable.

We need to import our library. In a normal way, we could use `go get` to recover this library for us, but since we’re developing everything in our local machine, we just need to add the path to the lib in our “go.mod” file:

replace github.com/profusion/http-redirect/protocol => ../protocol

Let’s create a struct that implements the “HttpRedirectPlugin” interface, and calls our log function.

// log-plugin/plugin.go
package main

import (
  //…
  "github.com/<your_github_username>/http-redirect/protocol"
)

// … previous code …

type PluginStr struct{}

// Compile time check for
// PreRequestHook implements protocol.HttpRedirectPlugin.
var _ protocol.HttpRedirectPlugin = PluginStr{}

// PreRequestHook implements protocol.HttpRedirectPlugin.
func (p PluginStr) PreRequestHook(req *http.Request) {
  logRequest(req)
}

var Plugin = PluginStr{}

That’s all the code we need. We just need to build this declared as a plugin. To do this, we just pass the buildmode flag to the go compiler:

# From http-redirect/log-plugin
go build -buildmode=plugin -o plugin.so plugin.go

Voilà! We have a plugin! Now we just need to load it to our application.

Loading your plugin

We need an app to load these plugins. This is not the focus of this article, but here is the code for an HTTP redirection server in Go, which we’re going to change from now on.

// cmd/main.go
package main

import (
  "flag"
  "fmt"
  "io"
  "log/slog"
  "net/http"
  "strings"
)

var from int
var to string

func init() {
  flag.IntVar(&from, "from", 5555, "Local port to get requests")
  flag.StringVar(&to, "to", "", "Target server to redirect request to")
}

func main() {
  flag.Parse()
  Listen()
}

type proxy struct{}

func Listen() {
  p := &proxy{}
  srvr := http.Server{
    Addr: fmt.Sprintf(":%d", from),
    Handler: p,
  }
  if err := srvr.ListenAndServe(); err != nil {
    slog.Error("Server is down", "Error", err)
  }
}

// ServeHTTP implements http.Handler.
func (p *proxy) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
  // Remove original URL for redirect
  req.RequestURI = ""

  // Set URL accordingly
  req.URL.Host = to
  if req.TLS == nil {
    req.URL.Scheme = "http"
  } else {
    req.URL.Scheme = "https"
  }

  // Remove connection headers
  // (will be replaced by redirect client)
  DropHopHeaders(&req.Header)

  // Register Proxy Request
  SetProxyHeader(req)

  // Resend request
  client := &http.Client{}

  resp, err := client.Do(req)

  if err != nil {
    http.Error(rw, "Server Error: Redirect failed", http.StatusInternalServerError)
  }
  defer resp.Body.Close()

  // Once again, remove connection headers
  DropHopHeaders(&resp.Header)
  
  // Prepare and send response
  CopyHeaders(rw.Header(), &resp.Header)
  rw.WriteHeader(resp.StatusCode)
  if _, err = io.Copy(rw, resp.Body); err != nil {
    slog.Error("Error writing response", "error", err)
  }
}

func CopyHeaders(src http.Header, dst *http.Header) {
  for headingName, headingValues := range src {
    for _, value := range headingValues {
      dst.Add(headingName, value)
    }
  }
}

// Hop-by-hop headers. These are removed when sent to the backend.
// http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html
var hopHeaders = []string{
  "Connection",
  "Keep-Alive",
  "Proxy-Authenticate",
  "Proxy-Authorization",
  "Te", // canonicalized version of "TE"
  "Trailers",
  "Transfer-Encoding",
  "Upgrade",
}

func DropHopHeaders(head *http.Header) {
  for _, header := range hopHeaders {
    head.Del(header)
  }
}

func SetProxyHeader(req *http.Request) {
  headerName := "X-Forwarded-for"
  target := to
  if prior, ok := req.Header[headerName]; ok {
    // Not first proxy, append
    target = strings.Join(prior, ", ") + ", " + target
  }
  req.Header.Set(headerName, target)
}

Let’s first find where our plugins reside. For this, we’re going to define a configuration file in JSON. It will be just a list of paths, in this article a single item list at it, but notice this is an opportunity for defining configurations for the plugin.

// config.json
[
  "log-plugin/plugin.so"
]

Good enough for now. Let’s write our application to list this file’s contents. We’re doing plugin load in another file to keep things tidy.

// cmd/plugin.go
package main

import (
  "encoding/json"
  "os"
)

// global but private, safe usage here in this file
var pluginPathList []string

func LoadConfig() {
  f, err := os.ReadFile("config.json")
  if err != nil {
    // NOTE: in real cases, deal with this error
    panic(err)
  }
  json.Unmarshal(f, &pluginPathList)
}

Now let’s load the plugins themselves. For this we’re going to use golang’s plugin module from the stdlib.

// cmd/plugin.go
package main

import (
  //…
  "plugin"
)

// ...previous code...

var pluginList []*plugin.Plugin

func LoadPlugins() {
  // Allocate a list for storing all our plugins
  pluginList = make([]*plugin.Plugin, 0, len(pluginPathList))
  for _, p := range pluginPathList {
    // We use plugin.Open to load the plugin by path
    plg, err := plugin.Open(p)
    if err != nil {
      // NOTE: in real cases, deal with this error
      panic(err)
    }
    pluginList = append(pluginList, plg)
  }
}

// Let's throw this here so it loads the plugins as soon as we import this module
func init() {
  LoadConfig()
  LoadPlugins()
}

With the plugin loaded, we can access their symbols now, including the variable Plugin we defined in our protocol. Let’s change the previous code to actually store this variable instead of the whole plugin. Now, our file looks something like this:

// cmd/plugin.go

import (
  //…
  "protocol"
  "net/http"
)

//…

// Substitute previous code
var pluginList []*protocol.HttpRedirectPlugin

func LoadPlugins() {
  // Allocate a list for storing all our plugins
  pluginList = make([]*protocol.HttpRedirectPlugin, 0, len(pluginPathList))
  for _, p := range pluginPathList {
    // We use plugin.Open to load plugins by path
    plg, err := plugin.Open(p)
    if err != nil {
      // NOTE: in real cases, deal with this error
      panic(err)
    }
  
    // Search for variable named "Plugin"
    v, err := plg.Lookup("Plugin")
    if err != nil {
      // NOTE: in real cases, deal with this error
      panic(err)
    }
    
    // Cast symbol to protocol type
    castV, ok := v.(protocol.HttpRedirectPlugin)
    if !ok {
      // NOTE: in real cases, deal with this error
      panic("Could not cast plugin")
    }
    
    pluginList = append(pluginList, &castV)
  }
}

// …

Good, now all the variables in our pluginList are normal golang variables we can use as if they were part of our code from the start. Let’s then build the hook function to call all plugin hooks before we send the request ahead.

// cmd/plugin.go

//…

func PreRequestHook(req *http.Request) {
  for _, plg := range pluginList {
    // Plugin is a list of pointers, we need to dereference them
    // to use the proper function
    (*plg).PreRequestHook(req)
  }
}

And, finally, call the hook on the host code

// cmd/main.go

//…

// ServeHTTP implements http.Handler.
func (p *proxy) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
  PreRequestHook(req)
// …

That’s it! You created an application, a plugin, loaded that plugin into your application and ran your plugin code for each request we received, logging those requests.

Want to test? Just run:

# From http-redirect
go run cmd/*.go -from <port> -to <url>

Conclusion

In this article, we’ve discussed what are plugins, where are they used and how the amazingly simple (but extensive) Go standard library gives developers the power to create applications that accept them. In future endeavors, consider making your solutions extensible by using this infrastructure, which would allow other developers to make wide use of your tooling and applications.

Just remember, this is a simple introduction. A lot more can be done by plugins. We hope this article has given you the link needed to connect pieces of code and, from here, you’re able to expand the concepts to big and complex applications.