Parquet: Columnar Storage for Hadoop Data

At AppNexus, over 2MM log events are ingested into our data pipeline every second. Log records are sent from upstream systems in the form of protobuf messages. Raw logs are compressed in Snappy when stored on HDFS. That said, even with compression, this…

pool_party — Transactional allocation for short-lived requests

Let me start off by saying that one really isn’t supposed to write custom memory allocators. Berger, Zorn, and McKinley thorougly debunked the need for them by the vast majority of the population in [1]. If you’re looking…

an_message: Format Agnostic Data Transfer

Every distributed RESTful system has a communication problem. How does Service A communicate with Service B? Does it pass data via multipart/form-data? Does it pass individual fields on the query string? Does it POST a blob of JSON?

Introduction to the Actor Model for Concurrent Computation

x Eighty Swift

Understanding the low level behavior of our applications is one of the most useful skills for working on high throughput systems at AppNexus. We currently process around 4 million requests per second with extremely strict latency requirements, so being able to identify and correct inefficiencies at…