Simple Impression Tracking With Crystal and PostgREST

by Itamar Nabriski, Client Team Leader

After repeatedly bumping into the Crystal Programming Language as of late, I finally got curious and started reading about it and was pleasantly surprised. Having grown tired of callbacks, promises and their workaround — async/await in JS/Node, I especially liked Crystal’s fibers concurrency model.

I began contemplating which project to try Crystal on.

Ad Impressions

In online advertising, an impression is a URL, called when an ad is displayed (a.k.a “impression pixel” or simply “impression”). An app developer only gets paid for displaying an ad if the ad’s impression URL was fired and counted on the advertiser’s server.

In real life, this is more complicated as the ad goes through several middlemen each paying the next until reaching the aforementioned app. The middlemen need to agree on payments — so each one appends his own impression URL to the ad. If all goes well, all middlemen should count the same number of impressions but this rarely happens. When the difference in impressions is not reasonable, it’s called having an “impression discrepancy” and one of the parties involved will certainly lose money. It’s important to understand who’s responsible for the problem and agree on how to split the difference.

After a few puzzling discrepancies as of late, where both parties appeared innocent, I decided to add a new faux-middleman to count the impressions and understand which party got the correct count.

This looked like a good opportunity to give Crystal a spin.

Installing Crystal

Crystal is a young language and it definitely shows when you try to install it.

I was able to install it on my Mac but running Kemal for a web framework, I got the following error:

crystal run src/ 
Undefined symbols for architecture x86_64:
"_iconv", referenced from:
_*Iconv#convert<Pointer(Pointer(UInt8)), Pointer(UInt64), Pointer(Pointer(UInt8)), Pointer(UInt64)>:UInt64 in I-conv.o
"_iconv_close", referenced from:
_*Iconv#close:Nil in I-conv.o
"_iconv_open", referenced from:
_*Iconv#initialize<String, String, (Symbol | Nil)>:Nil in I-conv.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `cc "${@}" -o '/Users/nabriski/.cache/crystal/crystal-run-sanger_client.tmp' -rdynamic -lz `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libssl || printf %s '-lssl -lcrypto'` `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libcrypto || printf %s '-lcrypto'` -lpcre -lgc -lpthread /usr/local/Cellar/crystal/0.26.1_1/src/ext/libcrystal.a -levent -liconv -ldl -L/usr/lib -L/usr/local/lib`

Searching for it didn’t bring up any helpful solution. Instead, I installed Ubuntu as a virtual machine and worked there. I also deployed to an Ubuntu server.

Installing PostgREST

PostgREST it a RESTful server for the Postgres DB. For simple interactions with an SQL db, I prefer this interface as I find SQL statements or using an ORM in the code to be clunky. In PostgREST you simply POST your SQL query as JSON to a URL end point and get back another JSON as the result.

Installing it was really neat as the project provides a docker-compose that installs both the Postgres DB and PostgREST — as long as you have Docker running, you just configure the docker-compose yaml, run it and then create your schema in the DB. I ignored any security considerations for now as this server will not be initially in a production environment.

My Crystal App

My app’s aim is to be an arbiter between Mobfox and our publishers regarding impression discrepancy. Thus, if on a certain day, Mobfox has counted 10,000 impressions and the publisher has counted 15,000 impressions we can go to the app and see how many impressions were counted. If it sides with Mobfox then we assume the publisher is wrong and vice versa.

The app is very simple with 3 endpoints:

  1. Serve a VAST wrapper to Mobfox’s video API.
  2. Record an impression pixel when the video ad is played.
  3. Record an error pixel if the video ad was not played correctly.

Endpoint 1— VAST Wrapper

Our first endpoint takes the request parameters for the VAST and runs them through the VAST wrapper template, returning the wrapper:

This should look pretty familiar from many other web frameworks. Our endpoint matches GET requests on /vast.xml. The rest is also straightforward — we set a header, read a few query params and then pipe these into the VastTempl class which we use to ‘render’ the template.

By calling the new method to create an instance, we call the class’s initialize method. The @q, @s and @pub arguments, by virtue of the @ prefix, are automagically turned into properties of our instance.

We then use the ECR templating module included in Crystal to render the VAST file with the aforementioned properties. There are a couple of interesting things here:

  1. Our instance properties are visible in the template engine’s context.
  2. As Crystal is a compiled language, I used the def_to_s method to embed the template in the compiled code to prevent loading from disk on each call.

Our template is as follows:

A video player will parse this file, follow the VastAdTagURI, retrieve the video file and play it. It will subsequently call our impression pixel, which can be found in the <Impression> element.

Endpoint 2— Impression Pixel

This endpoint receives and records the impression pixel.

We parse the inv and pub parameters from the query string, create our JSON ‘payload’ and then POST it to the PostgREST server listening on port 3000. We use the /imps PostgREST endpoint to indicate we wish to insert to the imps table. This table coincidently has two columns named inv and pub (in addition to other auto-generated ones).

The third endpoint is almost identical and can be omitted for brevity.

Compile and Deploy

Compiling a Crystal project for deployment is pretty simple:

crystal build --release src/

Deploying the resulting executable was more challenging, I could not find a simple service for running a Kemal app in production and eventually settled for using the Immortal supervisor.

Closing Thoughts

Except for installation and deployment, working with Crystal is so far enjoyable. Getting compile time errors but still working with an interpreted-like language is nice. Not having callbacks is nice as well. Performance was also impressive although I didn’t really look at the metrics — it just worked.

As for PostgREST, I really like the concept. I don’t like having my code peppered with SQL or using complicated ORMs for simple CRUD stuff.

Will definitely consider using Crystal and PostgREST on a larger project.