Streamlining Kafka client security using the power of open-source software — Problem Security is a principal concern of Expedia Group’s Stream Platform — we aim to secure all of our engineer’s stream resources. Obviously, this starts with the Kafka topics that persist their events — but once you scratch the surface you find many other resource types that are involved in typical…

Beyond the specification — applying schema evolution to real-world situations — I’ve been fortunate enough to be involved with the design and implementation of a number of large-scale data platforms that are used by 1000s of customers, be they analysts, engineers, or scientists. A common trait shared by these platforms is that they used Apache Avro to provide strong schema-on-write data…

What should you do when you need to make a breaking change to your data model? — Apache Avro has the notion of schema compatibility that allows us to determine whether or not a schema is compatible with one or more earlier or newer schemas with respect to some compatibility constraint. That we can have compatible changes necessarily implies that we can also have incompatible changes. In…

The use of enums is usually a good practice, but this is not always the case when using Apache Avro — Apache Avro is commonly used in both batch and real-time data systems to describe extensible and defendable data schemas. Avro enables the creation of data systems where data types can be mutated without impacting producers or consumers. However, it is not without its issues. …

Circus Train is an open source tool developed by Hotels.com for migrating and replicating very large Apache Hive datasets between clusters and clouds. The tool has become a key component of Expedia Group data platforms, enabling off-premises migrations, hybrid cloud architectures, and disaster recovery strategies. Circus Train was created when…

When I had my UK home refurbished a few years ago, I had Cat6 installed to the front door in anticipation of future developments in internet connected doorbells. I expected that PoE would be a likely choice to power such a device, and it was already supported by Doorbird, a…

Hotels.com has recently contributed the Waggle Dance project to the open-source community. The unusual name is taken from a dance that bees perform for the rest of their colony, helpfully directing them to sources of food. …