Serializing objects with Protobuf in .NET 7
Efficient Binary Serialization in .NET with Protobuf
Serialization is the process of converting an object or data structure into a format suitable for storage or transmission. This conversion allows for data to be stored in files, databases, or sent across networks with ease.
Applications often require data persistence or inter-system communication. Serialization provides a consistent methodology to package and handle this data.
Common serialization formats include:
XML: A text-based format known for its human-readable structure. It has extensive compatibility but can be larger in size compared to more concise formats.JSON: Another text-based format, lighter than XML. It has become a standard for web-based APIs due to its readability and ease of use.Binary: Formats likeProtobufare binary and aim to optimize for size and speed. These are machine-optimized and may not be readily interpretable by humans.
The choice of serialization format often depends on the application’s specific requirements. Some applications may prioritize human-readability, while others may prioritize performance.
Protobuf
Protobuf is a serialization and deserialization standard provided by Google (also used in gRPC). The main highlight of Protobuf is that it's lighter and more performant in operations compared to JSON and XML.
Protocol Buffers are an efficient and portable structured data storage format. Messages are serialized into a binary wire format that is very compact, making it extensible as a communication and data storage protocol.
Advantages
- Very fast processing
- Very useful for communication between APIs
- Easy interoperability between languages
Disadvantages
- Not human-readable:
JSON, being in text format with a simple structure, is easy to read and analyze by humans, but this is not the case withProtobuf. - Not very popular in the community.
In this article, we will learn how to set up and use Protobuf in .NET 7.0, following best practices and tips to keep the code performant.
Prerequisites
- Visual Studio 2022 (
.NET 7.0) - Nuget package
protobuf-net
API
In the program.cs of the API, configure as per the code below:
try
{
var builder = WebApplication.CreateBuilder(args);
builder.Host.AddSerilog(builder.Configuration, "API Protobuf");
Log.Information("Getting the motors running...");
builder.Services.AddRouting(options => options.LowercaseUrls = true);
builder.Services.AddSwaggerApi(builder.Configuration);
builder.Services.AddControllers();
var app = builder.Build();
if (app.Environment.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
app.UseMiddleware<ErrorHandlingMiddleware>();
app.UseSwaggerDocApi();
app.UseHttpsRedirection();
app.UseRouting();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
});
app.Run();
}
catch (Exception ex)
{
Log.Fatal(ex, "Host terminated unexpectedly");
}
finally
{
Log.Information("Server Shutting down...");
Log.CloseAndFlush();
}The next step is to create the Person class with the Protobuf mappings:
[ProtoContract]
public class Person
{
[ProtoMember(1)]
public int Id { get; set; }
[ProtoMember(2)]
public string Name { get; set; }
[ProtoMember(3)]
public Address Address { get; set; }
[ProtoIgnore]
public bool Active { get; set; }
}
[ProtoContract]
public class Address
{
[ProtoMember(1)]
public string Line1 { get; set; }
[ProtoMember(2)]
public string Line2 { get; set; }
}[ProtoContract]: Indicates that the class will be serialized usingProtobuf.[ProtoMember]: Indicates that the property will be serialized usingProtobuf.[ProtoIgnore]: Indicates that the property will be ignored when serialization occurs.
To make the calls to Protobuf easier, create an extension class named ProtoBufExtensions and insert the following code:
using System;
using System.IO;
public static class ProtoBufExtensions
{
public static string SerializeToStringProtobuf<T>(this T obj) where T : class
{
using var ms = new MemoryStream();
ProtoBuf.Serializer.Serialize(ms, obj);
return Convert.ToBase64String(ms.GetBuffer(), 0, (int)ms.Length);
}
public static T DeserializeFromStringProtobuf<T>(this string txt) where T : class
{
var arr = Convert.FromBase64String(txt);
using var ms = new MemoryStream(arr);
return ProtoBuf.Serializer.Deserialize<T>(ms);
}
public static byte[] SerializeToByteArrayProtobuf<T>(this T obj) where T : class
{
using var ms = new MemoryStream();
ProtoBuf.Serializer.Serialize(ms, obj);
return ms.ToArray();
}
public static T DeserializeFromByteArrayProtobuf<T>(this byte[] arr) where T : class
{
using var ms = new MemoryStream(arr);
return ProtoBuf.Serializer.Deserialize<T>(ms);
}
public static void SerializeToFileProtobuf<T>(this T obj, string path) where T : class
{
using var file = File.Create(path);
ProtoBuf.Serializer.Serialize(file, obj);
}
public static T DeserializeFromFileProtobuf<T>(this string path) where T : class
{
using var file = File.OpenRead(path);
return ProtoBuf.Serializer.Deserialize<T>(file);
}
}Then, create a controller named ProtobufController and insert the code below for testing:
[Route("api/[controller]")]
public class ProtobufController : Controller
{
[HttpPost("serialize")]
public IActionResult Serialize([FromBody] Person model)
{
var result = model.SerializeToStringProtobuf();
Serilog.Log.Information($"Protobuf serialized: {result}");
return Ok(result);
}
[HttpPost("deserialize")]
public IActionResult Deserialize([FromBody] string protobuf)
{
var result = protobuf.DeserializeFromStringProtobuf<Person>();
return Ok(result);
}
}Testing
To perform the tests, run the POST /serialize endpoint in Swagger and check the log collection result. Then run the POST /deserialize endpoint passing the Protobuf serialization text to convert to a C# object.
Case Study: Streamlining Real-time Gaming Communication
In a real-world scenario, an imaginary gaming company was in the process of developing an online multiplayer game. The game’s architecture relied on sending a significant amount of data between the client and server in real-time. Initial prototypes used JSON for data exchange, but the developers noticed latency issues, especially during peak gameplay moments.
After some research, the team decided to give Protobuf a try. Here's what they found:
- Reduced Data Payload: By moving from
JSONtoProtobuf, the data payload size reduced by approximately 40%. This reduction meant faster data transmission, especially critical for real-time games where every millisecond counts. - Performance Enhancements: Deserializing data on the client-side was noticeably faster with
Protobuf. The game's response time improved, providing a smoother gameplay experience to users. - Forward and Backward Compatibility: As the game evolved, there were updates and changes to the data structure.
Protobufmade it easy to introduce new fields without breaking existing clients, ensuring seamless updates. - Cost Savings: With reduced data size, the game used less bandwidth, leading to cost savings in data transfer, especially important given the large user base.
- Multi-platform Support:
Protobufoffered easy interoperability between different languages, making it easier for the team to develop cross-platform clients.
This transition to Protobuf was instrumental in NexaGames successfully launching their game and scaling it to millions of users. The case highlights the tangible benefits of choosing the right serialization protocol based on specific project requirements.
Final Thoughts
Protobuf was designed to be extremely fast, lighter, and consequently more performant than other protocols. Even though it's not very popular, always use it when developing internal communication APIs and performing serialization/deserialization.
Future and Evolution
Looking ahead, the importance of efficient data serialization is poised to grow, especially with the proliferation of distributed systems, IoT devices, and high-performance applications. Protobuf, with its emphasis on efficiency and compactness, has a significant role to play in this landscape.
As software development continues to move towards microservices and real-time data exchange, the need for fast, lightweight serialization methods will only increase. Given its foundational ties with gRPC, a rising RPC framework, Protobuf is well-positioned to be a central tool in the toolkit of modern developers. It's worth keeping an eye on its evolution and the emergence of any new features or optimizations that cater to the evolving demands of modern software architectures.
