Go Protocol Buffers!


Protocol Buffers?

Protocol buffers are a simple and compact way to structure data. Some of the features of protocol buffers include:

  • Structure data with types
  • Ability to serialize
  • Versioning
  • Fast
  • Logical data structure format

Another great feature is backwards compatibility. Protocol buffers allow developers to safely update data structures without breaking previous versions, and without complicated logic to decode version types.

Why?

Protobuf was created by Google for data-interchange between various systems at Google. At Namely, we are developing a datastore for keeping track of not only objects, but also the history of each object. This datastore, known as Arnoldb, will run as a microservice separate from the main Namely App. Our hope is to use Arnoldb as our main datastore and eliminate the need for ActiveRecord. To replace ActiveRecord, Arnoldb needs to be as fast as possible. This is one reason why we chose Protobuf: it is fast, as well as simple and compact. Protobuf also allows us to enforce a stricter standard between client and server: because both require a proto file to encode and decode communications.

Proto2 vs Proto3?

The current version (at the time of this post) of protocol buffers, is Proto2. The latest version, Proto3, includes several improvements over its predecessor. The biggest addition is the new map type. Proto3 also makes all fields optional, instead of allowing a developer to specify whether it is optional or required. Since Proto3 is still in alpha, some things aren’t fully baked — like the new map type.


Short Tutorial:

This protobuf tutorial runs through the creation of a quick protofile, Go server, and Go client. Tutorial repo. It will read from a csv file and then the client will send each line to the server.

1. Create the *.csv file.

Sample data

2. Create a *.proto file. Below you can see an example protofile.

//by default proto2 is used
//declares the package this protofile belongs to
package PbTest;
//declares the message TestMessage
message TestMessage {
required string clientName = 1;
required int32 clientId = 2;
optional string description = 3 [default = “NONE”];
repeated MsgItem messageItems = 4;
  enum TType {
CREATE = 0;
UPDATE = 1;
DELETE = 2;
}
  message MsgItem {
required int32 id = 1;
optional string itemName = 2;
optional int32 itemValue = 3;
optional TType transaction_type = 4;
}
}

3. Install the protocol buffer compiler. Go to the Downloads section for the version you want(See section: Proto2 vs Proto3? above).

4. Install the Go protocol buffer plugin.

$ go get -a github.com/golang/protobuf/protoc-gen-go

5. Run the protocol buffer compiler on the protofile to generate the files for Go.

$ protoc — go_out=. *.proto

6. Create the Go server.

package main
import (
“bytes”
“fmt”
“io”
“net”
“os”
“../PbTest”
“github.com/golang/protobuf/proto”
)

Setting up the server

func main() {
fmt.Println(“Staring Server..”)
c := make(chan *PbTest.TestMessage)
go func() {
for {
message := <-c
ReadReceivedData(message)
}
}()
  listener, err := net.Listen(“tcp”, “:8080”)
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
  for {
if conn, err := listener.Accept(); err == nil {
go handleProtoClient(conn, c)
} else {
continue
}
}
}

This function prints out each item received

func ReadReceivedData(data *PbTest.TestMessage) {
msgItems := data.GetMessageItems()
fmt.Println(“Receiving data…”)
for _, item := range msgItems {
fmt.Println(item)
}
}

This function handles any data which is sent through the connection the data is copied to memory and then unmarshalled using the proto file the unmarshalled data is then sent through the go channel.

func handleProtoClient(conn net.Conn, c chan *PbTest.TestMessage) {
fmt.Println(“Connected!”)
defer conn.Close()
var buf bytes.Buffer
  _, err := io.Copy(&buf, conn)
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
  pdata := new(PbTest.TestMessage)
err = proto.Unmarshal(buf.Bytes(), pdata)
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
c <- pdata
}

7. Create the Go client.

package main
import (
“encoding/csv”
“flag”
“fmt”
“io”
“net”
“os”
“strconv”
“../PbTest”
“github.com/golang/protobuf/proto”
)
type Headers []string

This function sets up the main flags and defaults.

func main() {
filename := flag.String(“f”, “example.csv”, “Enter the filename of CSV to read from”)
dest := flag.String(“d”, “localhost:8080”, “Enter the destination socket address”)
flag.Parse()
  data, err := retrieveDataFromFile(filename)
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
sendDataToDest(data, dest)
}

This function parses the Data from the CSV file to send to the server app.

func retrieveDataFromFile(fname *string) ([]byte, error) {
file, err := os.Open(*fname)
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
defer file.Close()
  csvreader := csv.NewReader(file)
var headers Headers
headers, err = csvreader.Read()
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
  itemIdIndex := headers.getHeaderIndex(“itemid”)
itemNameIndex := headers.getHeaderIndex(“itemname”)
itemValueIndex := headers.getHeaderIndex(“itemvalue”)
itemTypeIndex := headers.getHeaderIndex(“transactiontype”)
Testmessage := new(PbTest.TestMessage)
Testmessage.ClientName = proto.String(“Test Client”)
Testmessage.ClientId = proto.Int(191)
  for {
record, err := csvreader.Read()
if err != io.EOF {
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
} else {
break
}
    msgItem := new(PbTest.TestMessage_MsgItem)
itemid, err := strconv.Atoi(record[itemIdIndex])
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
    msgItem.Id = proto.Int(itemid)
msgItem.ItemName = &record[itemNameIndex]
itemValue, err := strconv.Atoi(record[itemValueIndex])
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
    msgItem.ItemValue = proto.Int(itemValue)
ttype, err := strconv.Atoi(record[itemTypeIndex])
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
    ctype := PbTest.TestMessage_TType(ttype)
msgItem.TransactionType = &ctype
Testmessage.MessageItems = append(Testmessage.MessageItems, msgItem)
fmt.Println(record)
}
return proto.Marshal(Testmessage)
}

This function gets a header index value

func (h Headers) getHeaderIndex(headername string) int {
if len(headername) >= 2 {
for index, s := range h {
if s == headername {
return index
}
}
}
return -1
}

This function sends the data to the server.

func sendDataToDest(data []byte, dst *string) {
conn, err := net.Dial(“tcp”, *dst)
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
  n, err := conn.Write(data)
if err != nil {
fmt.Fprintf(os.Stderr, “Fatal error: %s”, err.Error())
os.Exit(1)
}
fmt.Println(“Sent “ + strconv.Itoa(n) + “ bytes”)
}

8. Run the App.

$ go run server.go &
$ go run client.go