New official MongoDB Go Driver and Google Protobuf — making them work together
Update: Article has been updated for MongoDB Go language SQL Driver version Beta 2.
MongoDB is one of the most popular NoSQL databases in the world. MongoDB has started developing new official driver for Go language few months ago:
The Go driver for MongoDB. Contribute to mongodb/mongo-go-driver development by creating an account on GitHub.
It is Beta 2 version when I write this article. I believe MongoDB will issue production-ready driver in few months.
Lets imagine the following scenario for this article: we need to develop (using Go language) gRPC CRUD service which is using MongoDB database as persistent engine. We have to write/read Google protobuf structures in/from MongoDB database.
I am not going to focus on gRPC service development. You can see detail tutorial here. I am going to focus on how to persist data in Google protobuf format in MongoDB database.
Full source code for this article is available here:
New official MongoDB Go Driver and Google Protobuf - making them work together - amsokol/mongo-db-go-protobuf-tutorial
- Install MongoDB 3.2 or higher instance. You can get it for free from Atlas cloud service.
- Create database “experiments” and collection “proto” inside.
- Install MongoDB Compass (GUI for MongoDB) from here for free.
- Install Go 1.11 or higher (for Go modules support).
- Download Proto compiler binaries here:
Protocol Buffers - Google's data interchange format - protocolbuffers/protobuf
Extract package to any folder on your PC and add “bin” to PATH environment variable.
- Install Go language code generator plugin for Proto compiler:
go get -u github.com/golang/protobuf/protoc-gen-go
Create and compile protobuf data message
Lets create sample proto file data.proto with message structure we are going to write/read to/from MondoDB:
Note: protobuf wrappers (google.protobuf.BoolValue, google.protobuf.Int64Value, etc.) are very usefull. You can use them, for example, instead of optional attribute that is not supported for proto3 version.
Next step is to compile proto file to Go:
protoc data.proto --proto_path=. --proto_path=third_party --go_out=plugins=grpc:.
Note: third_party folder contains included proto files. See full source code here.
The result of compilation is data.pb.go file.
Write/read protobuf data to/from MongoDB
Next is create main.go file that includes write/read functionality:
As we see the code above is very simple. Lets look at the “proto” collection after compilation and execution of the code. Here is screenshot from MongoDB Compass GUI tool:
Data inserted successfully but the result looks terrible:
- Timestamp stores as binary integer values instead of as MongoDB native Date type
- Protobuf wrappers are stored as objects instead of just values
- A lot of xxx_ protobuf internal useless fields
Lets fix all issues one by one.
How to fix issues with Timestamp and protobuf wrappers
MongoDB driver stores data in BSON (binary JSON)format. Luckily BSON engine gives us way to implement custom marshaling/unmarshalling for any Go type. Like Go language JSON engine does. It allows us to create custom marshaling/unmarshalling for protobuf Timestamp and wrapper types.
I create very simple library:
This is extension for MongoDB Go driver adds support for Google protocol buffers types …
So just add it to our code (lines 23–41):
Lets look at the “proto” collection into database after compilation and execution of the code:
It works. Looks much more better. There is only one problem — internal xxx_ fields.
How to remove XXX_ internal useless fields
Sri Krishna Paritala has developed great plugin for protobuf compile. It allows to add custom Go language tags to the proto message fields and XXX_ fields. I has forked it, updated to the latest version of protobuf compiler, fixed bugs and added some new functionality:
Add custom struct tags to protobuf generated structs - amsokol/protoc-gen-gotag
go get -u github.com/amsokol/protoc-gen-gotag
So lets remove XXX_ internal fields from BSON data. I want share example how to change BSON field name also. Modify data.proto file as following:
Then run protobuf compiler with protoc-gen-gotag plugin:
protoc data.proto --proto_path=. --proto_path=third_party --go_out=plugins=grpc:.protoc data.proto --proto_path=. --proto_path=third_party --gotag_out=xxx="bson+\"-\"",output_path=.:.
Note: we have to set location of generated Go file using output_path parameter. The only “.” is right after “:”. See details here.
So lets look at the “proto” collection into database after compilation and execution of the code:
That is much more better. No more xxx_ fields.
Bonus: timestamp field has name date instead of timestampvalue.
That’s all for now. Full source code is available here.