How we use and abuse Redshift to serve our data

In part one, we described our Analytics data ingestion pipeline, with BigQuery sitting as our data warehouse. However, having our analytics events in BigQuery is not enough. Most importantly, data needs to be served to our end-users.

Image for post
Image for post
TL;DR — Teads Analytics big picture

In this article, we will detail:

  • Why we chose Redshift to store our data marts,
  • How it fits into our serving layer,
  • Key learnings and optimization tips to make the most out of it,
  • Orchestration workflows,
  • How our data visualization apps (Chartio, web apps) benefit from this data.

Data is in BigQuery, now what?

Back to where we stopped in the previous article. Our data is stored in raw and aggregated formats in BigQuery. Our data analysts can query it directly for exploration or debugging purpose, but SQL has its limit. That’s where some Scala code takes over in our architecture, in a component we call the Analytics Service. …


Analytics pipeline at Teads

In this article, we describe how we orchestrate Kafka, Dataflow and BigQuery together to ingest and transform a large stream of events. When adding scale and latency constraints, reconciling and reordering them becomes a challenge, here is how we tackle it.

Part II of this article focuses on how we use and abuse Redshift to serve this data to our end users.

Image for post
Image for post
Teads for Publisher, one of the webapps powered by Analytics

In digital advertising, day-to-day operations generate a lot of events we need to track in order to transparently report campaign’s performances. These events come from:

  • Users’ interactions with the ads, sent by the browser. These events are called tracking events and can be standard (start, complete, pause, resume, etc.) or custom events coming from interactive creatives built with Teads Studio. …

About

Alban Perillat-Merceroz

Software Engineering Manager @ Teads

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store