LiveRamp Engineering Blog

Joining Petabytes of Data Per Day: How LiveRamp Powers its Matching Product

Distributed Tracing at Massive Scale

When a developer thinks about monitoring and observability of their production application, two things generally come to mind: metrics and logs. While those are really useful for debugging and monitoring purposes, there is still a critical monitoring element that…

Initial Release of Workflow2 — LiveRamp’s Big-Data Workflow Orchestrator

LiveRamp Hackweek: Serializing and Transmitting Bytecode Between JVMs

Migrating a Big Data Environment to the Cloud, Part 5

What comes next? How are we going to re-engineer…

Migrating a Big Data Environment to the Cloud, Part 4

Copying to the cloud

Migrating a Big Data Environment to the Cloud, Part 3

How do we get to the cloud?

Migrating a Big Data Environment to the Cloud, Part 2

Starting the journey

Migrating a Big Data Environment to the Cloud, Part 1

[Opinion] Our Exciting Journey as Data Scientists Onwards to Higher Levels of Abstraction

Introduction

Over the last few years, I’ve been consistently excited by the ever-increasing pace of new developments in the cloud computing…

[Opinion] Report Guidelines

Reports are our chief vehicle for influencing organizations, especially as they grow to a larger number of people. Further, they are also how we assemble our collective knowledge about the problems we work on and the solutions to these problems. Because reports are so important…

LiveRamp Is Migrating to Google Cloud Platform

Have you ever thought of what it takes to move 97 petabytes to the cloud? Now think about those 97 petabytes of stored data flanked with 300 terabytes of memory and 90,000 CPU cores, and you have the exact size of our Hadoop environment. Over the…

[Opinion] Guidelines for Effective Meetings

Meetings can be an effective way to discuss projects and reach consensus on decisions. But I believe when ran improperly, they can also be a huge waste of our valuable time. Hence, we should have some guidelines on how to create and run effective…

Using Machine Learning to Auto Detect Column Types in Customer Files

Introduction

Utilizing MapReduce Combiners and HyperLogLog++ to process millions of queries over datasets with billions of records

Friday Thoughts: strive for phoenixes, not snowflakes

tl;dr: Avoid surprises, restart your workloads often to ensure your app starts as expected.

Server outages can produce some interesting surprises. As we continue to strive for dependable interfaces and services…

Friday Thoughts: test properties when you can

tl;dr: You can often get a lot more mileage out of time spent writing tests if you can find properties to test using randomized data instead of (or as well as) cases to test with specific data.

Efficiently Analyzing a 600 Billion Edge Graph in Real-Time

LiveRamp for Sociologists: Exploring the correlations between demographic segments

For Hackweek XLIV, our data science team explored how LiveRamp’s demographic data and our identity resolution technologies could be used by sociologists to study human behavior. For…

Hackweek XLIV

For many LiveRampers, hackweek is one of their favorite parts of working here. It’s a time when you can work on whatever you think will bring the most value to the company, whether it is fixing a process that’s been making you miserable, cleaning up some old code that’s dragging down the system, or…

Friday thoughts: fail, fast and furiously

tl;dr: When implementing a service or API, if you get a request you don’t quite understand, the kindest thing you can do is to return a noisy error.

Let’s consider an API like:

GET /mySum?num=3&num=42