Building a Serverless Analytics App to Capture and Query Clickstream Data

The best way to answer questions about user behavior is often to gather data. A common pattern is to track user clicks throughout a product, then perform analytical queries on the resulting data, getting a holistic understanding of user behavior.

In my case, I was curious to get a pulse of developer preferences on several divisive questions. So, I built a simple survey and gathered tens of thousands of data points from developers on the Internet. In this post, I will walk through how I built a web app that:

• collects free-form JSON data
• queries live data with SQL
• has no backend servers

To stay focused on collecting click data, we’ll keep the app’s design simple: a single page presenting a series of binary options, on which clicking will record the visitor’s response and then display live aggregate results. (Live version here. Spoiler alert: you can view the results here.)

Creating the static page

Keeping with the spirit of simplicity, we’ll use vanilla HTML/CSS/JS with a bit of jQuery to build the app’s frontend. Let’s start by laying out the HTML structure of the page.

Note that we left the #body element empty-we'll add the questions here using Javascript:

By adding the questions with Javascript, we only have to write the HTML and event handlers once. We can even adjust the list of questions at any time by just editing the global variable QUESTIONS.

Collecting custom JSON data

Now, we have a webpage where we want to track user clicks-a classic case of product analytics. In fact, if we were instrumenting an existing web app instead of building from scratch, we would just start at this step.

First, we’ll figure out how to model the data we want to collect as JSON objects, and then we can store them in a data backend. For our data layer we will use Rockset, a service that accepts JSON data and serves SQL queries, all over a REST API.

Data model

Since our survey has questions with only two choices, we can model each response as a boolean-false for the left-side choice and true for the right-side choice. A visitor may respond to any number of questions, so a visitor who prefers spaces and uses vim should generate a record that looks like:

With this model, we can implement the click handlers from above to create and send this custom JSON object to Rockset:

In practice, ROCKSET_APIKEY should be set to a value obtained by logging into the Rockset console. The Rockset collection which will store the documents (in this case demo.binary_survey) can also be created and managed in the console.

Updating existing responses

Our code so far has a shortcoming: consider what happens when a visitor clicks “spaces” then clicks “vim.” First, we will send a document with the response for the first question. Then we’ll send another document with responses for two questions. These get stored as two separate documents! Instead we want the second document to be an update on the first.

With Rockset, we can solve this by giving our documents a consistent , which is treated as the primary key of a document in Rockset. We’ll generate this field as a random identifier for the visitor on page load:

Now let’s run through the previous scenario again. When the web page loads, the “vote” object gets seeded with an ID:

When the visitor clicks a choice for one of the questions, a boolean field is added:

The visitor can continue to add more responses:

Or even update previous responses:

Every time the response changes, the JSON is stored as a Rockset document and, because the _id field matches, any previous response for the current visitor is overwritten.

Saving state across sessions

We’ll add one more enhancement to this: for visitors who leave the page and come back later, we want to keep their responses. In a full-blown app we may have an authentication service to establish sessions, a users table to persist IDs in, or even a global frontend state to manage the ID. For a splash page that anyone can visit, such as the survey we’re building, we may not have any previous context for the user. In this case, we’ll just use the browser’s local storage to maintain the visitor’s ID.

Let’s modify our Javascript code to implement this mechanism:

Data-driven app: aggregations on the fly

At this point, we’ve created a static page and instrumented it to collect custom click data. Now let’s put it to use! This generally takes one of two forms:

• an internal dashboard informing product decisions or triggering alerts around unusual behavior
• a user-facing feature to enhance a data-driven product

Our survey’s use case falls under the latter: as an incentive to answer questions for curious visitors, we’ll reveal the live results of each question upon clicking a choice.

To implement this, we’ll write Javascript code to call Rockset’s query API. We want to send a SQL query that looks like:

The response will be a JSON object with counts for each question (count of “true” responses and total count of responses), along with a count of unique visitors.

We can parse this data and set attributes on HTML elements to relay the results to the visitor. Let’s write this out in Javascript:

Even with tens of thousands of data points, this AJAX call returns in around 20ms, so there is no concern executing the query in real time. In fact, we can update the results, say every second, to give the numbers a live feel:

Finishing touches

Access control

We’ve written all the logic for sending data to and retrieving data from Rockset on the client side of our app. However, this exposes our fully privileged Rockset API key publicly, which of course is a big no-no. It would give anyone full access to our Rockset account and also possibly allow a DoS attack. We can achieve scoped permissions and request throttling in one of two ways:

• use a restricted Rockset API key
• use a lambda function as a proxy

The first is a feature still-in-development at Rockset, so for this app we’ll have to use the second.

Let’s move the list of questions and the logic that interacts with Rockset to a simple handler in Python, which we’ll deploy as a lambda on AWS:

Our client-side Javascript can now just make calls to the lambda endpoints, which will act as a relay with the Rockset API.

Adding more questions

A benefit of the way we’ve build the app is we can arbitrarily add more questions, and everything else will just work!

Similarly, if a visitor only answers a subset of the questions, no problem-the client-side app and Rockset can handle missing values gracefully.

In fact, these circumstances are generally common with product analytics, where you may want to start tracking an additional attribute on an existing event or if a user is missing certain attributes. Since we’ve built this app using a schemaless approach, we have the flexibility to handle those situations.

Rendering and styling

We haven’t fully covered the logic yet for rendering and styling elements on the DOM. You can see the full completed source code here if you’re curious, but here’s a summary of what’s left to do:

• add some JS to show/hide results and prompts as the visitor progresses through the survey
• add some CSS to make the app look nice and adapt the layout for mobile visitors
• add in a post-survey-completion congratulatory message

And voila, there we have it! End to end, this app took just a few hours to set up. It required no spinning up servers or pre-configuring databases, and it was easy to adapt while developing as there was it was just recording free-form JSON. So far over 2,500 developers have submitted responses and the results are, if nothing else, interesting to look at.

The survey is currently live here-feel free to check it out and poke around! Results, as of the writing of this blog, are here. And the source code is available here.

Originally published at https://rockset.com on May 17, 2019.