Go Plugin — Write it Once

Hello folks, welcome. In this article, we will go through one of Go’s most powerful features, Plugins.

Let’s assume we are developing a large scale application and we want to decouple each module so that it won’t fail if one of the moving parts stops working. We also want to be able to add new features to the existing system without downtime. Well, Plugins can help you in that regard. Let’s dig deep.

Let us develop a Job Aggregator utilising the power of Plugins.

We want to build a system as depicted in the image at the left. An aggregator will be fetching job info from different sources and based on the data received, markdown files will be generated and be saved to appropriate stores. After getting the aggregator ready, we need people to add different sources. Well we don’t know how to fetch data from these sources, so providers have to provide or let us know how to parse these data.

We will parse the data structure in the following format:

import "encoding/json"
type Empleo struct {
    Title    string
    Company  string
    Location string
    Link     string
    Tags     []string
    Time     string
}
func (e *Empleo) Serialize() (string, error) {
    b, err := json.Marshal(e)
    if err != nil {
        return "", err 
    }
    return string(b), err
}

An interface to tell providers what aggregator know about the source,

type EmpleoSource interface {
    Init() error
    Fetch()([]Empleo, bool)
}

If Fetch returns true aggregator assumes there are still data left to fetch so aggregator will call Fetch method again.

Now how the providers will provide source to aggregator ? Here comes the plugin. Providers have to write plugin satisfying EmpleoSource interface. Assume someone provided us a source which one parses the data from We Love Golang site. Notice that package name is main.

Notice that above source implemented our EmpleoSource interface. Also notice that at line 53 we have an instance of WeLoveGolang named INSTANCEWELOVEGOLANG (it will be required later).

What’s next now ? Well, it’s time to build the plugin. To do so we have to use the below command,

go build -v -buildmode=plugin -o output_file.so input_file.go

So once we build the plugin it will provide us .so file which is shared binary library. We got our first plugin. Now let’s look how the aggregator is using this .so file.

We have a configuration file which contains the sources. Notice that we have one source in the below configuration and key is same as instance variable we have declared in the plugin, followed by path to the .so file (in the example .so file is located in a folder named bin from root directory).

{
  "sources": {
    "INSTANCEWELOVEGOLANG": "./bin/we_love_golang.go.so"
  }
}

Below is the main code of our aggregator,

We have used viper to read configuration from json file. Lets go through line 28–62. We have declared an array of EmpleoSource which will hold all the sources. Later read sources from sources tag of json file. At line 32 Open function of plugin returns an instance of Plugin using .so file provided as parameter (v is the path to .so file), it will return error if .so file is invalid or not found. Later we used Lookup function which returns an instance of Symbol (note that we have provided instance variable name declared in plugin as symbol name, you can lookup for function as well). Now it’s time to cast Symbol to interface EmpleoSource. Once we have casted it successfully method Init() and method Fetch() is available to aggregator to use. Finally we have used fetched results after combining all of them to produce markdown file.

See aggregator doesn’t know how sources are working and you don’t need to do any change in core system. What you need to do is write a plugin satisfying the interface, build the plugin to generate .so file and finally let the aggregator know about the .so file through config. Aggregator will do the rest. Cool, isn’t it ?

In the way you can write as many sources as you want and plug them with aggregator without making any change to the aggregator.

Happy Coding.

Complete source code : https://github.com/pathao-eng/empleo