We recently open-sourced a number of internal tools we’ve built to help our engineers write high-performance Cascading code as the cascading_ext project. Today I’m going to to talk about a tool we use to improve the performance of asymmetric joins — joins where one data…

At LiveRamp, we constantly face scaling challenges as the volume of data that our infrastructure must deal with continues to grow. One such challenge involves the logging system. At present we use Scribe as the transport mechanism to get logs from our webapp servers into our…

When using Cascading to run MapReduce jobs in production, the most common exception we find in our job logs look like this:

Caused by: cascading.flow.FlowException: step failed: (1/1), with job id: job_201307251526_37599…

While Hadoop has a number of tools available to make sure cluster resources are allocated intelligently (such as the fair scheduler and the YARN project), we’ve found that for time-critical issues these tools don’t always give enough granularity of…