Switching to Protobuf (from Avro) on Kafka

Tanishq Dubey
Clear Street
Published in
2 min readJan 27, 2023

Clear Street is a fintech building modern infrastructure for capital markets and reimagining the legacy workflows and silos that are essential to increasing access in the financial markets, all while transparently decreasing risks and costs.

Our platform is built on a modern foundation designed around Kafka and tightly integrated into the AWS cloud, giving us horizontal and vertical scalability. Our architecture lets us easily decompose complex finance business logic into manageable chunks, or microservices, and Kafka’s democratization of data access and data flows make it easy for us to create an event-sourced architecture.

However, we’ve learned along the way that even modern technology, like Kafka, sometimes needs a new approach and that the most common approaches aren’t always the most effective. This is why we decided to move from Avro to Protobuf on Kafka.

Keep reading on our blog:

https://www.clearstreet.io/insights/switching-to-protobuf-from-avro-on-kafka

Clear Street does not provide investment, legal, regulatory, tax, or compliance advice. Consult professionals in these fields to address your specific circumstances. These materials are: (i) solely an overview of Clear Street’s products and services; (ii) provided for informational purposes only; and (iii) subject to change without notice or obligation to replace any information contained therein.

Copyright © 2022 Clear Street LLC. All rights reserved. CLEAR STREET and the Shield Logo are Registered Trademarks of Clear Street LLC.

--

--

Tanishq Dubey
Clear Street

Currently @clear-street, UIUC Grad, Airplane nerd, talk RISC-V to me