Use Go & gRPC to Create a Platform That Implements Modules In Other Languages
This article assumes you have decent knowledge of Go and rudimentary understanding of protocol buffers and NodeJS, which aptly describes me.
Here’s the contrived, hypothetical scenario — you’re writing an application that is a platform to be used by other developers. Your platform will include modules which do…something. These other developers will extend the platform by writing their own modules, in their native language.
There are probably many ways to do this, and possibly many better ones, but here’s my take using protocol buffers.
Platform (gRPC server + module runner)
First, in a new project (called “modularPlatform” here), create the platform directory. In it, we’ll create our proto file modularPlatform.proto:
syntax = “proto3”;
package platform;service Platform {
rpc RegisterModule(Module) returns (Details) {}
}message Module {
string name = 1;
string port = 2;
string location = 3;
string runCommand = 4;
}message Details {
string details = 1;
}
Generate the Go protocol buffer package in the same directory by running:
protoc -I . modularPlatform.proto — go_out=plugins=grpc:.from the command line. This will generate a file named modularPlatform.pb.go. Google’s protoc not installed? See https://github.com/google/protobuf. Next, create server.go:
package platformimport (
"context"
"fmt"
"net""google.golang.org/grpc"
)type platformServer struct {
server *grpc.Server
port string
}// NewServer returns a platformServer with port and server populated
func NewServer(port string) *platformServer {
return &platformServer{port: port, server: grpc.NewServer()}
}// Start Registers and runs the platformServer
func (p *platformServer) Start() error {
lis, err := net.Listen("tcp", fmt.Sprintf("localhost:%s", p.port))
if err != nil {
return err
} RegisterPlatformServer(p.server, p)
fmt.Printf("running on port %s\n", p.port)
go p.server.Serve(lis)
return nil
}// Stop stops the grpc server
func (p *platformServer) Stop() error {
p.server.Stop()
return nil
}// RegisterModule returns the Details associated with a running module
func (p *platformServer) RegisterModule(ctx context.Context, module *Module) (*Details, error) {
// TODO - any actual module setup required on the platform return &Details{Details: fmt.Sprintf("register %s on port %s", module.Name, module.Port)}, nil
}
The Start() and Stop() functions start and stop the grpc server, respectively. The RegisterModule() function means that our platformServer object satisfies the PlatformServer interface defined in the generated protobuf file, modularPlatform.pb.go.
Next, create module.go, still in our platform directory:
package platformimport (
"bufio"
"fmt"
"io"
"os"
"os/exec"
"strings" "github.com/stinkyfingers/modularPlatform/config"
)// GetModulesFromConfig returns the "modules" field from the config
func GetModulesFromConfig() ([]Module, error) {
configFields := make(map[string][]Module)
err := config.GetConfig(configFields)
return configFields["modules"], err
}func RunModules(modules []Module) error {
for _, module := range modules {
err := module.run()
if err != nil {
return err
}
}
return nil
}// run runs a grpc server for a single Module and executes the
// binary (or compiles/interprets+runs the code) for a Module
func (m *Module) run() error {
p := NewServer(m.Port)
err := p.Start()
if err != nil {
return err
} runCommandArr := strings.Split(m.RunCommand, " ")
commandArr := append(runCommandArr, m.Location)
cmd := exec.Command(commandArr[0], commandArr[1:]...) // write cmd errs to stdOut
err = pipeErr(cmd, os.Stdout)
if err != nil {
return err
} out, err := cmd.Output()
if err != nil {
return err
}
fmt.Println("module output: ", string(out)) err = p.Stop()
if err != nil {
return err
} return nil
}// pipeErr writes cmd stdErr to w
func pipeErr(cmd *exec.Cmd, w io.Writer) error {
errPipe, err := cmd.StderrPipe()
if err != nil {
return err
} scanner := bufio.NewScanner(errPipe)
go func() {
for scanner.Scan() {
_, err = w.Write(scanner.Bytes())
if err != nil {
return
}
}
}()
return nil
}
The above functions achieve two things: 1) Getting the modules from a user-supplied config file and 2) Running all the modules by way of os/exec's command. The pipeErr() function exists just to help direct the subprocesses’ (modules’) output to the current terminal.
Config
Next, create a directory in the project’s root called config and a file in it named config.go:
package configimport (
"os" "github.com/go-yaml/yaml"
)var (
defaultConfigLocation = "/etc/netanal.conf"
CONFIG_LOCATION = "CONFIG_LOCATION"
)// GetConfigLocation returns envvar CONFIG_LOCATION if set, otherwise the defaultConfigLocation
func getConfigFileLocation() string {
if os.Getenv("CONFIG_LOCATION") == "" {
return defaultConfigLocation
}
return os.Getenv("CONFIG_LOCATION")
}// GetConfig assigns values from the config to v
func GetConfig(v interface{}) error {
f, err := os.Open(getConfigFileLocation())
if err != nil {
return err
} decoder := yaml.NewDecoder(f)
return decoder.Decode(v)
}
This package looks for a config file at environment variable CONFIG_LOCATION and provides GetConfig() to parse the yaml-ized config into the provided interface.
Main/cmd
To make the whole shebang runnable, create a cmd directory at the project’s root and in it create main.go:
package mainimport (
"log""github.com/stinkyfingers/modularPlatform/platform"
)func main() {
modules, err := platform.GetModulesFromConfig()
if err != nil {
log.Fatal(err)
}err = platform.RunModules(modules)
if err != nil {
log.Fatal(err)
}
}
Let’s also create a config file dummy defining modules that we’ll mock up below. Create a file named modularPlatform.conf and fill it with yaml:
modules:
- name: go_example
runcommand: 'go run'
port: 9999
location: /Users/johnshenk/go/src/github.com/stinkyfingers/modularPlatform/modules/go_example/cmd/main.go
- name: js_example
runcommand: node
port: 10000
location: /Users/johnshenk/go/src/github.com/stinkyfingers/modularPlatform/modules/js_example/client.jsAt this point, we’ve created 1) a platform package which attempts to find modules, run a grpc server at each port that a module will utilize, and run the modules, 2) a config package that will parse our config file, informing the aforementioned platform package about expected modules, and 3) a cmd package which runs our platform and contains a config with two modules we’ve config-ed, but not actually written. Let’s write them.
Modules
Create a directory called modules at the project root and directories called go_example and js_example in the modules directory.
Sample Module in Go
In the go_example directory, copy the modularPlatform.proto file from the platform directory and run :
protoc -I . modularPlatform.proto --go_out=plugins=grpc:.which will generate (again) the same modularPlatform.pb.go file in this location (we could just import the pb.go file, but want this to be a standalone module). Then, create directory cmd in this directory (go_example) and main.go:
package mainimport (
"context"
"flag"
"fmt"
"log"pb "github.com/stinkyfingers/modularPlatform/modules/go_example"
"google.golang.org/grpc"
)var (
port = flag.String("p", "9999", "port that module runs on")
)func main() {
flag.Parse()
err := start()
if err != nil {
log.Fatal(err)
}
}func start() error {
conn, err := grpc.Dial(fmt.Sprintf("localhost:%s", *port), grpc.WithInsecure())
if err != nil {
return err
}
defer conn.Close() client := pb.NewPlatformClient(conn)
message, err := client.RegisterModule(context.Background(), &pb.Module{Name: "test_go_module", Port: "9999"})
if err != nil {
return err
}
fmt.Println("MESSAGE: ", message)
return nil
}
The above ^ is a stand alone Go program — a gRPC client, that will connect to the gRPC server in our platform’s platform.go package, call the RPC RegisterModule, and print the returned message.
Sample Module in Javascript
Now, return to <project_root>/modules/js_example and AGAIN copy modularPlatform.proto from the platform package (protobufs is all about passing .proto files around, right?) Then, create client.js:
var PROTO_PATH = __dirname + '/modularPlatform.proto';var grpc = require('grpc');
var protoLoader = require('@grpc/proto-loader');
var packageDefinition = protoLoader.loadSync(
PROTO_PATH,
{keepCase: true,
longs: String,
enums: String,
defaults: true,
oneofs: true
});
var platform = grpc.loadPackageDefinition(packageDefinition).platform;
var client = new platform.Platform('localhost:10000',
grpc.credentials.createInsecure());function register() {
var err = client.RegisterModule({ name:"test_js_module", port: "10000" }, (err, mo) => {
if (err) console.log("oh no", err);
console.log(mo)
});
}register();
This is a standalone NodeJS program which does the same as the Go program above: a client that connects to the platform’s gRPC server, calls RegisterModule() and stdouts the response. With this in place, you’ll have to npm init and npm install in good NodeJS fashion.
With ALL this in place, you can return to cmd package modularPlatform/cmd (at your <project root>) and run main.go. Don’t forget to set envvars, like so:
CONFIG_LOCATION=modularPlatform.conf go run main.goThis will run the platform which will extract the sample Go and JS modules from the yaml file modularPlatform.conf, run the modules, and the modules will connect to the platform’s gRPC server, calling the RPC RegisterModule(). Assuming life is good, you’ll see stdout confirming that each module has “registered.”
In the long run, the goal would be to create streaming RPC’s and have the server and client exchange whatever streaming data you like. And the module list can be expanded by developing RPC-friendly modules and specifying their location, port, and command in the config file (e.g. modularPlatform.conf).
You can see the source code at: https://github.com/stinkyfingers/modularPlatform
Much thank you.