New official MongoDB Go Driver and Google Protobuf — making them work together

Update: Article has been updated for MongoDB Go language SQL Driver version Beta 2.

MongoDB is one of the most popular NoSQL databases in the world. MongoDB has started developing new official driver for Go language few months ago:

It is Beta 2 version when I write this article. I believe MongoDB will issue production-ready driver in few months.

Lets imagine the following scenario for this article: we need to develop (using Go language) gRPC CRUD service which is using MongoDB database as persistent engine. We have to write/read Google protobuf structures in/from MongoDB database.

I am not going to focus on gRPC service development. You can see detail tutorial here. I am going to focus on how to persist data in Google protobuf format in MongoDB database.

Full source code for this article is available here:

Requirements

  • Install MongoDB 3.2 or higher instance. You can get it for free from Atlas cloud service.
  • Create database “experiments” and collection “proto” inside.
  • Install MongoDB Compass (GUI for MongoDB) from here for free.
  • Install Go 1.11 or higher (for Go modules support).
  • Download Proto compiler binaries here:

Extract package to any folder on your PC and add “bin” to PATH environment variable.

  • Install Go language code generator plugin for Proto compiler:
go get -u github.com/golang/protobuf/protoc-gen-go

Create and compile protobuf data message

Lets create sample proto file data.proto with message structure we are going to write/read to/from MondoDB:

Note: protobuf wrappers (google.protobuf.BoolValue, google.protobuf.Int64Value, etc.) are very usefull. You can use them, for example, instead of optional attribute that is not supported for proto3 version.

Next step is to compile proto file to Go:

protoc data.proto --proto_path=. --proto_path=third_party --go_out=plugins=grpc:.

Note: third_party folder contains included proto files. See full source code here.

The result of compilation is data.pb.go file.

Write/read protobuf data to/from MongoDB

Next is create main.go file that includes write/read functionality:

As we see the code above is very simple. Lets look at the “proto” collection after compilation and execution of the code. Here is screenshot from MongoDB Compass GUI tool:

Data inserted successfully but the result looks terrible:

  • Timestamp stores as binary integer values instead of as MongoDB native Date type
  • Protobuf wrappers are stored as objects instead of just values
  • A lot of xxx_ protobuf internal useless fields

Lets fix all issues one by one.

How to fix issues with Timestamp and protobuf wrappers

MongoDB driver stores data in BSON (binary JSON)format. Luckily BSON engine gives us way to implement custom marshaling/unmarshalling for any Go type. Like Go language JSON engine does. It allows us to create custom marshaling/unmarshalling for protobuf Timestamp and wrapper types.

I create very simple library:

So just add it to our code (lines 23–41):

Lets look at the “proto” collection into database after compilation and execution of the code:

It works. Looks much more better. There is only one problem — internal xxx_ fields.

How to remove XXX_ internal useless fields

Sri Krishna Paritala has developed great plugin for protobuf compile. It allows to add custom Go language tags to the proto message fields and XXX_ fields. I has forked it, updated to the latest version of protobuf compiler, fixed bugs and added some new functionality:

Download it:

go get -u github.com/amsokol/protoc-gen-gotag

So lets remove XXX_ internal fields from BSON data. I want share example how to change BSON field name also. Modify data.proto file as following:

Then run protobuf compiler with protoc-gen-gotag plugin:

protoc data.proto --proto_path=. --proto_path=third_party --go_out=plugins=grpc:.protoc data.proto --proto_path=. --proto_path=third_party --gotag_out=xxx="bson+\"-\"",output_path=.:.

Note: we have to set location of generated Go file using output_path parameter. The only “.” is right after “:”. See details here.

So lets look at the “proto” collection into database after compilation and execution of the code:

That is much more better. No more xxx_ fields.

Bonus: timestamp field has name date instead of timestampvalue.

That’s all for now. Full source code is available here.

Thanks!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade