Taming the events: How we regained petabyte-scale Hive query power

In the beginning, four long years ago, simply having every data set in Hadoop was good enough. We were taking baby steps into the big data world and had a limited user base migrating from other sytems into the big data clusters, enabling them to execute heavy queries on large data sets in a timely manner. As the number of users and size of the data sets grew in size, we were challenged by new performance problems.

Jul 18, 2016 · 10 min read
Image for post
Image for post

Cool and simple ideas

What our users were looking for

user ID
user name
job name
number of maps and reducers
resources used in CPU and MB-seconds
the query (in the case of Hive jobs)

We’d give them everything they wanted

A handful of simple tricks

Image for post
Image for post

Was it worth it?

CREATE TEMPORARY FUNCTION event_parser AS 'com.booking.hive.udtf.EventParser';

create table mydb.sample_raw AS
SELECT event.epoch,
FROM default.raw_events as event
LATERAL VIEW event_parser( event.json,
) response
yyyy_mm_dd = '2016-04-18'
AND hh = 21
AND is_frontend = 1
AND action_name = 'explorer';
create table mydb.sample_flat AS
SELECT epoch,
FROM default.events_flat_web
yyyy_mm_dd = '2016-04-18'
AND hh = 21
AND is_frontend = 1
AND `action/name` = 'explorer';
1 days 1 hours 25 minutes 10 seconds 20 msec
0 days 2 hours 41 minutes 12 seconds 420 msec

We may want to keep it

Booking.com Development

Software development at Booking.com

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store