Tagged in

Big Data

Xandr-Tech

Our latest thoughts, challenges, triumphs, try-again’s, most snarky and profound commit messages. Our proudest achievements, deepest darkest technical debt regrets (just kidding, maybe). All the humbling yet informative things you learn when you try to do things with computers.

More information

Followers

602

Elsewhere

More, on Medium

Big Data

Harichandana in Xandr-Tech

Sep 4, 2020

How I spent my summer at Xandr: Building a Data Analytics Pipeline

Avi Lissack in Xandr-Tech

Jul 10, 2020

Securing our Big Data Platform (Part 2)

When last we saw our heroes, they were battling Hadoop…

Avi Lissack in Xandr-Tech

Jun 12, 2020

Securing our Big Data Platform (Part 1)

The video we posted on June 23rd, 2015 (link here) introduced…

Xandr Engineering in Xandr-Tech

Nov 3, 2017

Meet Van Gogh: The Algorithm that Makes Data Segmenting Simple

Xandr Engineering in Xandr-Tech

Mar 30, 2015

Parquet: Columnar Storage for Hadoop Data

At AppNexus, over 2MM log events are ingested into our data pipeline every second. Log records are sent from upstream systems in the form of protobuf messages. Raw logs are compressed in Snappy when stored on HDFS. That said, even with compression, this…

Xandr Engineering in Xandr-Tech

Oct 27, 2015

Stopping Invalid Traffic using Spark Streaming, Kafka, and Science!

The Signals Intelligence group (SIGINT) within AppNexus Data Science has the responsibility of identifying and stopping invalid traffic as quickly as possible. One technique they use to achieve this very quickly…

Xandr Engineering in Xandr-Tech

Aug 11, 2015

Securing our Big Data Platform Part 1

The video we posted on June 23rd, 2015 (link here) introduced our efforts to open up our big data platform to other teams within AppNexus. Data Platform as a Service (DPaaS) is our internal offering that allows other teams at AppNexus to run analytics upon our…

Xandr Engineering in Xandr-Tech

Jun 22, 2015

Taming Big Data

The Data Platform Engineering Team at AppNexus has been utilizing Hadoop in production for the last 4+ years. Growth of data volume as well as the number of customer use cases supported by Hadoop’s infrastructure has grown exponentially since we adopted the Hadoop stack. More specifically, in 2012…

Xandr Engineering in Xandr-Tech

Jan 20, 2016

Securing our Big Data Platform Part 2

When last we saw our heroes, they were battling Hadoop authentication in DPaaS. But our intrepid knights were able to overcome that crafty foe and move on to the next obstacle: authorization. And that’s where our story continues…

Xandr Engineering in Xandr-Tech

Jan 31, 2016

Real Time Big Data

Lambda architectures for Big Data sprung out of the concept that stream processing is fast but inaccurate. So you needed to have a fast but less accurate streaming path along with a slower, accurate batch path. With the advent of technologies like Ratsoda and Samza, this is no longer the case.