Ro’s Looker Code Generation System: Motivation and Key Features

Sami Yabroudi
Feb 20, 2019 · 3 min read

(article #1 in series)

Ro uses Looker as its BI tool. Looker is based on the idea of analysts building Looker “models” written in its LookML language; these models describe a graphical point-and-click interface that a business-side individual can use to query the specified data sets without needing to write code.

Operations, data models additions/modifications, product offerings, and medical condition offerings are scaling far too quickly at Ro for the standard Looker development model to be sustainable without having a small army of a data team to provide the labor required. Applying our general analytic tendencies to the situation, we found that much of the creation and maintenance labor required for LookML models was repetitive and error prone. Separately, LookML lacked certain features that we were having trouble doing without, particularly around documentation and the idea of view blocks. So, now we automatically generate most of our LookML code instead.

Basic Architecture

Completely independently of Looker, we code Looker views, explores, and models in our own Ro Looker templating language. A Python engine that we’ve built turns these templates into LookML and writes the LookML files to our Looker GitHub repo. A web hook pushes the LookML from GitHub to our Looker instance. The number of lines written in our Ro Looker templating language is less than 10% of the number of lines in the LookML that is generated.

Key Features

Key features of the Ro LookML code generation system include:

Automatically add dimensions to a Looker view when a new column is added to the underlying table

We couple this with our automatic pivot table generator — when Ro rolls out a new supported health condition, for example, this means that in most cases all of the fields for the new conditions appear first in our derived tables then in Looker automatically.

Reducing our custom permissions system from 15 lines per Looker field to a single word

We do permissions on a field-by-field basis instead of model-by-model.

Explore-level documentation via Sphinx

We provide documentation specific to each explore in Looker, complete with examples and technical notes. Our documentation (written in Markdown) files are wrapped in with our Looker templating, and our documentation-generation engine uses our Looker templating files to autogenerate portions of the documentation site.

New looker concept: view blocks

Groups of Looker views can appear together in several different Looker explores. For example, our “member” block contains many views with various types of member-level information; this block appears in the majority of our Looker explores. Looker does not have any existing structure that encapsulates this idea of multiple views being grouped.

Automate organization of explore views

We use some tricks to organize views in a large explore:

Remove the need to specify dimension type

We automatically deduce this from the underlying table column’s type. We also have automated levels for timestamp dimensions with different levels of time breakouts (date, day of week, month, hour, minute, etc).

Make group labels easier/automatic

In Looker currently, each dimension/measure needs to have its group label (if any) specified explicitly. With the Ro system, we can assign group labels to field name prefixes, and any field with one of these prefixes will be added to the corresponding group automatically.

Automatic parameter lists

Currently, Looker requires that you manually specify parameters. Our system can auto-generate parameter lists from unique values in a column or unique dimensions in a dimension group.

A variety of other tricks

  • Ex: all measures use the first defined drillset by default
  • Ex: removing the need for things like “${TABLE}”

Further discussion

This article was cross-posted on Looker Discourse.

Ro Data Team Blog

Ro Data Team Blog: data analytics, data engineering, data science

Sami Yabroudi

Written by

Ro Data Team Blog

Ro Data Team Blog: data analytics, data engineering, data science

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade