What’s all the buff about PROTOBUF — Part2

So this article will cover the HOW part of the last article.

Protobuf provides lots of flexibility in defining schema. You can use almost every common types of datatype while defining the schema.

An example of a simple schema could be like below

syntax = "proto3";

package example.simple;

message SimpleMessage {
   int32 id=1;
   bool is_simple=2;
   string name=3;
   repeated int32 sample_list=4;
}

The name of the file should end with .proto suffix, for ex: simple.proto.

First line of any proto file starts with systax, which indicates which version of protobuf we are using (latest is proto3)

package is being used to better arrange/order the files as per the standards.

Main schema comes under the message section . We define the messageName and all the attributes or fields inside this section. Each field has a datatype and a tag associated to it. For ex: in above is_simple field, bool is the type and 2 is the tag attached to it. Tag plays an important role, which decides the order and encoding of the fields. It also helps in maintaining backward compatibility very easily.

A more complex schema can be something like below :

syntax="proto3";

package example.third;

message ChildObject {
    int32 id = 1;
    string name = 2;

}

enum Brands {
    UNKNOWN_BRAND = 0;
    LOW_BRAND =1;
    HIGH_BRAND =2;
}

message ComplexObject {
    int32 id = 1;
    Brands brand = 2;
    ChildObject childObject=3 ;

}

You can see, in the above .proto file we have defined enums and child objects.

These enums and objects are similar to standard enums in high level languages like Java. We can have multiple permutations and are only limited by our imagination for any kind of schema.

Sample code in Python

I am using PyCharm as the IDE for developing this demo code. Feel free to use any Python code editor.

Before staring, you need to install protobuf on your system. Mac users can install protobuf by :

brew install protobuf
//You can do protoc --version, to check if protobuf has been installed properly

My project structure looks like below :

So this has Python module simpleMessage for the .proto files explained above.

Let’s start with the simple case.

1.Create a simple.proto file under simpleMessage module as below :

syntax = "proto3";

package example.simple;

message SimpleMessage {
   int32 id=1;
   bool is_simple=2;
   string name=3;
   repeated int32 sample_list=4;
}

2. Open the terminal, and using protoc compile generate the code for the above .proto file

protoc -I=simpleMessage/ --python_out=simpleMessage/ simpleMessage/simple.proto

You need to provide the path of the .proto file for protoc , and the place where generated Python code will be placed. Once you run, you can see simple_pb2.py file will be created under simpleMessage/ .

Protoc generated all the boiler place code for you, and you don’t need to write any code for handling defined schema.

3. Create a new python file as below, and in that import the simple_pb2.py file

import simpleMessage.simple_pb2 as simple_pb2

simple_message = simple_pb2.SimpleMessage()

simple_message.id=1
simple_message.is_simple = True
simple_message.name = "Sample Name"
#List can't be directly added. It has first to be taken out and then added.
simple_list = simple_message.sample_list
simple_list.append(1)
simple_list.append(2)

print simple_message


with open("simple.bin" , "wb")  as f :
    print "writing data to the file"
    bytesAsString = simple_message.SerializeToString()
    f.write(bytesAsString)

with open("simple.bin", "rb") as f :
    print "Reading data from the file"
    simple_message_from_file = simple_pb2.SimpleMessage().FromString(f.read())
    print simple_message_from_file

In the above code, we are just trying to create the SimpleMessage object based on the schema defined in the simple.proto file . Just to check/confirm it’s working, I am storing the same in a file and then reading it back. This is just a simple use case. These can even be used to transfer information/data between two talking services/systems.

I hope this small exercise was helpful in getting you started and excited to try all new opportunities that Protobuf provides !

Detailed code can be found here -> https://github.com/Vishwa07dev/ProtoBufPOCInPython