How ChiselStrike generates its TypeScript client API

Two weeks ago, ChiselStrike announced a new TypeScript client SDK generator that handles the details of invoking the automatically generated REST API for your data model. You can use this SDK in your web and Node.js applications to make it very easy to work with entity data. The best part is that it preserves the types of all data coming across the wire, so have greater assurance that your code will do what you expect.

Performing a query for all instances of an entity in a type-safe way is as easy as writing this:

const client = createChiselClient({ serverUrl })
const posts: BlogPost[] = await chiselClient.posts.getAll()

It’s just one command to run (chisel generate) to create the client source code, and you’re ready to go. However, hidden behind that simplicity is a bunch of complexity that analyzes all your entity and route information, provides the right types for your client app, issues the correct requests, and deserializes response data from the backend API.

In this series of blog posts, I’ll walk you through the process of implementing chisel generate and share some of the interesting aspects of this endeavor that I discovered along the way. And even if you’re not a code generation enthusiast, I hope you take away one thing from all this: code generation doesn’t have to be intimidating!

What are we generating?

To illustrate what is the goal, let’s consider a ChiselStrike project with a single (heavily simplified) entity:

// models/blog_post.ts
export class BlogPost extends ChiselEntity {
    text: string;
    tags: string[];
    publishedAt: Date;
    authorName: string;
    karma: number;
}

And a routing map exposing CRUD endpoint for BlogPost:

// routes/index.ts
export default new RouteMap()
    .prefix("/blog/posts", BlogPost.crud());

Above, note the use of a static method crud() which returns a RouteMap containing all CRUD endpoints for the BlogPost entity, and makes that route available at the path /blog/posts. Once this code is running in the ChiselStrike daemon (chiseld), you can immediately start invoking those endpoints using any HTTP client. But what most client apps would rather do instead is use a type-safe client API.

When designing a new API, it’s often helpful to start with the desired usage and worry about the implementation later. Here’s what it looks like, given the above entity and route.

Firstly, a factory function to create the client object given some configuration (minimally, a URL endpoint):

const client = createChiselClient({ serverUrl });

Next, a way to create a new blog post using that client object by issuing a POST request:

const post: BlogPost = await client.blog.posts.post({
    text: "To like or not to like Trains, that is the question!",
    tags: ["lifeStyle"],
    publishedAt: new Date(),
    authorName: "I Like Trains",
    karma: 0,
});

Notice how URL path segments map onto the client properties. The path /blog/posts becomes nested properties .blog.posts. This is a good place to start, but we also need to account for the fact that the CRUD paths use placeholders to capture variables. For example, to PATCH a specific blog post, you would issue a PATCH HTTP request against /blog/posts/:id, where :id specifies where the unique ID of the post goes in the path. This functionality surfaces in the API as an id() method in the call chain. Here’s what we want it to look like for that PATCH on a BlogPost instance:

const updatedPost: BlogPost = await client.blog.posts.id(“[YOUR-ID]”).patch({
    tags: ["lifeStyle", "philosophy"],
});

Generating the entity types (in Rust)

In the above code, you can see that both post() and patch() accept a BlogPost object as an argument, and also return a BlogPost — that’s the type-safety in this API! What we need to do is generate that type for the client to use (the client app cannot share entity code with the backend — we can’t simply reuse the original BlogPost entity class as you might hope).

What we’re going to do to get that BlogPost type for the client is query chiseld for the model data, turn it into a similar-looking TypeScript type, and store it in a dedicated file called models.ts.

The generated entity type will be slightly different from the original entity that subclasses ChiselEntity, as we don’t need anything as complex as a class for the client. We also need to add an explicit id field as all ChiselEntities inherit one. This is what the final type needs to look like for the client:

// models.ts
export type BlogPost = {
    id: string;
    text: string;
    tags: string[];
    publishedAt: Date;
    authorName: string;
    karma: number;
};

To generate this, we can use the following code (written in Rust). VersionDefinition contains all of the entity and route information obtained from chiseld:

fn generate_models(version_def: &VersionDefinition) -> Result<String> {
    let mut output = String::new();
    for def in &version_def.type_defs {
         writeln!(output, "export type {} = {{", def.name)?;
         for field in &def.field_defs {
             let field_type = field.field_type()?;
             writeln!(
                 output,
                 " {}{}: {};",
                 field.name,
                 if field.is_optional { "?" } else { "" },
                 type_enum_to_code(field_type)?
             )?;
         }
         writeln!(output, "}}")?;
    }
    Ok(output)
}

fn type_enum_to_code(type_enum: &TypeEnum) -> Result<String> {
    let ty_str = match &type_enum {
         TypeEnum::ArrayBuffer(_) => "ArrayBuffer".to_owned(),
         TypeEnum::Bool(_) => "boolean".to_owned(),
         TypeEnum::JsDate(_) => "Date".to_owned(),
         TypeEnum::Number(_) => "number".to_owned(),
         TypeEnum::String(_) | TypeEnum::EntityId(_) => "string".to_owned(),
         TypeEnum::Array(element_type) => {
             format!("{}[]", type_enum_to_code(element_type)?)
         }
         TypeEnum::Entity(entity_name) => entity_name.to_owned(),
    };
    Ok(ty_str)
}

Now, I’m sure you came here looking for TypeScript, but I just delivered you a bunch of Rust, mostly out of context! If you don’t get all that right now, that’s OK. The reason why we’re using Rust here is because chiseld is implemented in Rust, and we’re reaching into its internals to get a hold of the VersionDefinition object that has all the data we need.

If you dig in there, you can see that the code emits an export type statement with a placeholder for the type name, and then it iteratively emits individual properties (fields) with their corresponding type.

So that gets us from the original entity class definition to the generated type for the client. Pretty easy so far. Admittedly, we saved a lot of work by pulling the already parsed types from the ChiselStrike backend, but in the future, we hope to drop that dependency.

Generating the client object

Now that we have our types, it’s time to focus on the client object itself. To make it work as illustrated in our earlier examples, we can leverage TypeScript’s object types and nest them to create the required structure. It’s simple, it’s typed, and it will be easy to generate compared to a bunch of classes. Here’s what the client object looks like:

const client = {
  blog: {
    posts: {
      post: (blogPost: Omit<BlogPost, "id">) => Promise<BlogPost> {...},
      // put, get, delete functions missing here
      id: (id: string) => {
        return {
          patch: (blogPost: Partial<BlogPost>) => Promise<BlogPost> {...}, …
          // get, delete missing here
        }
      }
    }
  }
}

Definitely a lot more code here! We haven’t even implemented any of those functions yet, and that’s not even all of the CRUD methods. We’re still missing GET, DELETE, and PUT for both entity instance (with an id) and entity group (without an id) variants. We’ll probably also want to have some convenience methods for bulk retrieval, such as getAll().

Even if we’re planning on generating the code, it needs more structure. Okay, let’s start by replacing post: (user: Omit<User, “id”>) => Promise<User> {…}, with something more complete.

Generating the POST handler function (in TypeScript)

How about instead of generating the function in Rust, we wrote a TypeScript function that would generate the handler for us? It’s very easy to do using arrow functions:

function makePostOne<Entity extends Record<string, unknown>>(
    url: URL,
    entityType: reflect.Entity,
): (entity: OmitRecursively<Entity, "id">) => Promise<Entity> {
      return async (entity: OmitRecursively<Entity, "id">) => {
           const entityJson = entityToJson(entityType, entity);
           const resp = await sendJson(url, "POST", entityJson);
           await throwOnError(resp);
           return entityFromJson<Entity>(entityType, await resp.json());
      };
  }

If you’re thinking “What the heck does this code do?”, don’t worry, I’ll explain.

The makePostOne() function takes url and entityType parameters. The url tells us where the request will be sent, and the entityType reflection parameter is used to convert our entity to and from JSON. More on that later.

The makePostOne() function returns an async arrow function with the signature (entity: OmitRecursively<Entity, “id”>) => Promise<Entity>. The entity parameter must be OmitRecursively<Entity, “id”> because when we’re POSTing (creating) a new entity, we don’t have an ID yet. The database will provide it for us. Since all of our generated entity types contain the id field, we need to omit it, and do so in a recursive manner because entities can be nested. (To learn more about the implementation of OmitRecursively, read here.) The return type doesn’t need such a restriction because the function returns the newly created entity including the id field.

Now to the body of the function. The ChiselStrike backend accepts entities as JSON, so first we do that conversion using entityToJson.

const entityJson = entityToJson(entityType, entity);

This is one of those funny little details that look innocent but boy are they complicated. The reason why we can’t simply do JSON.stringify is that we support types such as Date and ArrayBuffer that aren’t directly compatible with JSON. To do the conversion, we need a reflection object entityType describing the type. The function will also do some basic sanity checking to make sure that the user hasn’t bypassed the type system and smuggled in some type mismatching values.

This process gives us a JSON-compatible object which we simply send down the wire using sendJson.

const resp = await sendJson(url, "POST", entityJson);

Then we check the response for errors:

await throwOnError(resp);

All this code so far just handles the HTTP request, so now we have to do the reverse for dealing with the HTTP response using entityFromJson().

return entityFromJson<Entity>(entityType, await resp.json());

Using this approach for makePostOne() we can add similar handler functions for PUT, PATCH and DELETE. Assuming that they already exist, let’s use them to compose the full client factory:

function createClient(serverUrl: string) {
  const url = (url: string) => {
    return urlJoin(serverUrl, url);
  };
  return  {
    blog: {
     posts: {
        post: makePostOne<BlogPost>(url(`/blog/posts`), reflection.BlogPost),
        delete: makeDeleteMany<BlogPost>(url(`/blog/posts`)),
        id: (id: string) => {
          return {
            delete: makeDeleteOne(url(`/blog/posts/${id}`)),
            patch: makePatchOne<BlogPost>(
              url(`/blog/posts/${id}`),
              reflection.BlogPost,
            ),
            put: makePutOne<BlogPost>(
              url(`/blog/posts/${id}`),
              reflection.BlogPost,
            ),
          };
        },
      },
    },
  };
}

Awesome! This will save us a lot of generation as we can put the functions makePostOne(), etc. to a client_lib.ts library file which is a regular TypeScript file which can be linted, analyzed or anything that we fancy. If you’re interested, you can see the full Rust source code for generating these functions.

What’s next

You may have noticed that our client is still missing the ability to ‘get’ things. ChiselStrike’s CRUD GET endpoints provide a rich set of filtering options and additional parameters. To support all of that in a type-safe manner is not trivial, and I’ll talk about it in a future post. After that, we will discuss the code behind entityToJson() and entityFromJson() functions and related reflection mechanisms.

To get notified of these future posts, consider following us here on Medium, Twitter or Discord. Hope to see you there!