Versioning of messages in message brokers

In the world of modern software development, API versioning has long been the norm. This practice allows developers to ensure backward compatibility and uninterrupted operation of services during updates. If you are not doing it yet — start now.

But what about message versioning in event processing systems like RabbitMQ? This issue becomes especially relevant when changes are required, and messages from one sender (producer) are read by many consumers.

We started thinking about how to do it right. The first thing they did was, of course, Google. Here are the most common approaches to message versioning:

• Duplicate messages for different versions: This method involves simultaneously sending messages of both versions to the queue, which are then filtered on the handler side. This approach is not effective for highly loaded systems due to excessive use of resources. With this approach, we duplicate messages, which is very inefficient in a highly loaded system.
• Sending new versions of messages to a new queue: new versions of messages are sent to a separate queue that consumers bind when they are ready to work with a new version. This complicates queue management and requires additional monitoring. And also from problems — see point one.

For now, we have come to another way. I have not found whether it is used in other companies or not. But I think that this is something that will suit us (at least — we will try it with him).

So, our approach. It is based on the principles of semantic versioning (https://semver.org/) and aims to optimize the use of resources while maintaining flexibility and backward compatibility:

• Using minor updates for non-breaking changes: We plan to add new fields to messages without removing old ones to ensure backward compatibility with existing handlers. That is, our final minor version will contain the first and second versions. You can add the identifier of the pre-release version — maybe we will use this as well.
• Contract testing to determine processor compatibility. Before implementing the changes in production, we plan to use contract testing (I won’t describe it here, it’s a topic for a separate article) to test the ability of handlers to work with new versions of messages.

Implementation and monitoring

• Design and Implementation: First, we will add new fields to messages, keeping the old ones for backwards compatibility.
• Consumers notification: After implementation of changes, we will inform consumers about the new version and provide a changelog.
• Monitoring: Through contract testing, we will know if all consumers have moved to the latest minor version. If so — we can delete the old version and create a new major — it will contain only the deletion of the old version.

Conclusion

Message versioning in highly loaded event systems requires careful planning and optimization. Our approach based on semantic versioning and contract testing is our attempt to solve this challenge while maintaining maximum efficiency and compatibility.

This is currently a plan and not fully implemented. But I think sharing plans is also very useful. And for those who read. And maybe you will give your feedback and suggest an improvement/another more efficient version. Write, I will be glad to hear your opinion.