Implementing custom parsers

Before we begin

This tutorial assumes you are comfortable with query syntax and some basic parsing operations, if you are not, here are some recommended reading:

Let’s build Youtube!

For this tutorial we are going to build a subset of some Youtube features, implementing a subset of their API into graph style.

Starting a new Parser

We will start with a very basic parser, a “hello world” parser you might say:

{:one "Hello World"
:two "Hello World"}
  • :parser this contains the parser itself, making easy to call the parser again recursively, a typical case is when dealing with a join field (that includes children items to be parsed)
  • :query only present when a sub-query exists for the current item, same as (get-in env [:ast :query])
  • :ast is the AST for the current entry, I recommend you play around with>ast to get familiar with how the AST is structured, here I’ll post some examples, so you get exposure to it, but please try some for yourself until you are comfortable with the format:

Project Setup

Before we go on let’s create a new project and add the dependencies that we are going to use:

Thinking on a composable graph

In our data is represented as a graph, and this can’t be different for our parser on the server, but while the client reads from a map in memory; our server must read in parts, lazily, while it navigates through the query. So it’s about implementing the nodes on the graph, and how each node can navigate, ideally we want those nodes to be composable pieces; parts that we can move around and reuse as we wish, from now on, we will call these parts Readers.

Starting the Root Reader

Now we are going to start our real parser, implementing our root node. The only thing we need to handle at the root now is the [video/by-id "SOME-ID"]:

Youtube API

To make the requests, we are going to need a Youtube Key, you can find/create one at Go in “Credentials” and create an API Key.

(video-by-id #:video{:id "oyLBGkS5ICk" :parts #{"snippet"}})

Smart Reading

It’s time to figure out how to get from our query to the Youtube request. So we have to determine the arguments to video-by-id from our parser environment.

Coercing results with spec

You might catch one problem on our return, the numbers on statistics are strings instead of numbers, and the published-at is also a string instead of a date. Let’s solve that with spec!

Fetching associated data

So far we did handle all client needs making a single request to Youtube, now it’s time to load more that on the fly, how much that changes for us? Not much actually.

Placeholder nodes

There is one issue that some people stumbled upon while using; the problem happens when you need to display two or more different views of the same item as siblings (regarding query arrangement, not necessarily DOM siblings), how do you make this query?

The code so far

Here is the cumulative code we wrote on this article (also added specs for the used attributes):


I hope by now you have an idea on how to write your parser, on this article we took advantages of Youtube conventions to write tiny amount of code and got a lot of coverage. We did wrapping of an external library, but you can use the same techniques to wrap a SQL database or a Datomic database, or your microservices architecture, pretty much whatever you want.

  • Implement related videos, here you can find how to fetch related videos.
  • Make the Youtube Key a parameter on the calls, since we using maps you can add an extra key, then fetch the key from the env on the parser, so users can inject this information when calling the parser.
  • Error handling
  • Add specs for all used keys to ensure correct coercion.
  • Implement comment replies.
  • Use spec keys to figure the possible children on each node, so you can alert users when they try to fetch an invalid key.
  • Implement pagination on comments using query params.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store