Why Dovetail depends on code generators to make our engineers more efficient
By Jack Lian — Software Engineer
At Dovetail, we rely heavily on the auto-generation of code and types within our development environments and architecture. In a nutshell, the primary value of this strategy is to save developer time. It does this by requiring less code to be written and preventing fewer errors from being introduced.
Another strong benefit of Dovetail’s approach is that by using a combination of tools such as graphql-codegen, PostGraphile, and PgTyped, we can generate interfaces and types for both our front-end and back-end, directly from PostgreSQL. Fundamentally, this makes our database schema the source of truth for much of our code-base, eliminating entire classes of errors.
What is code/type generation and why does it matter?
Code generators are used extensively in modern software engineering. Many developers will be familiar with HTML generation, which almost all web applications use via template engines to create pages and markup.
There are a number of categories of generators. Model-driven development is particularly powerful as we use it to generate application types and interfaces using our DB as the source model. As we make changes to our schema, we won’t need to manually make updates to our types and interfaces. Our generators can simply recompile.
As previously mentioned, one of the other key advantages of code generation is that it minimizes errors in our software. In the book ”Code Complete, second edition”, Steve McConnell highlights that the number of bugs and errors scales with the number of lines of code written:
The product that’s twice as large is likely to have more than twice as many errors.
Therefore, by having software auto-generate certain code categories for us, we cut down on the surface area where errors can be introduced.
The code generators we rely on at Dovetail
PgTyped
At Dovetail, we love TypeScript and use it throughout our stack, including for our Infrastructure as Code with Pulumi. Therefore we rely heavily on PgTyped to generate our types and interfaces for our database queries. With PgTyped handling our query templating, we don’t have to spend time writing query construction in TypeScript.
Traditionally, engineers defined their data model in the database layer and again in the application layer, in classes or structs. These can be handwritten or done using Object Relational Mapping (ORM), such as Sequelize or TypeORM. However, at Dovetail, we believe that handwriting these models are error-prone. We also wanted to avoid the abstraction that ORMs provide to be closer to the SQL and to better leverage the power of SQL for performance reasons.
With PgTyped we no longer need to map our DB model to TypeScript. PgTyped will automatically generate these types for us using our SQL queries as the source input.
Given a postgres `file` table and a query like this:
/* @name getFileById */
SELECT *
FROM dovetail.file
WHERE id = :id!;It generates the types for us like this:
import { PreparedQuery } from '@pgtyped/query';
export type file_retention_status = 'DELETED' | 'EXPIRED' | 'NOTIFIED' | 'PROCESSED';
export type file_upload_status = 'CANCELLED' | 'DONE' | 'FAILED' | 'UPLOADING';
export interface IGetFileByIdParams {
id: string;
}
export interface IGetFileByIdResult {
created_at: Date;
id: string;
size: number | null
type: string | null;
updated_at: Date;
upload_status: file_upload_status;
}
export interface IGetFileByIdQuery {
params: IGetFileByIdParams;
result: IGetFileByIdResult;
}
const getFileByIdIR: any = {"usedParamSet":{"id":true},"params":[{"name":"id","required":true,"transform":{"type":"scalar"},"locs":[{"a":39,"b":42}]}],"statement":"SELECT *\nFROM dovetail.file\nWHERE id = :id!"};
/**
* Query generated from SQL:
* ```
* SELECT *
* FROM dovetail.file
* WHERE id = :id!
* ```
*/
export const getFileById = new PreparedQuery<IGetFileByIdParams,IGetFileByIdResult>(getFileByIdIR);graphql-codegen
Another area where we generate code is on our React web application. Instead of manually writing the types and interfaces for the GraphQL request, we rely on graphql-codegen to generate them for us, based on the content of the specific GraphQL request and the schema. This ensures that the interfaces always match the query.
If you wrote a GraphQL query like thisL
const EXPORT_TAGS_QUERY = gql`
query ExportTagsDataQuery($id: UUID!, $offset: Int, $limit: Int!) {
project: projectById(id: $id) {
id
tags: tagsByProjectId(condition: { deleted: false, deletedCascade: false }, offset: $offset, first: $limit) {
totalCount
nodes {
id
title
color
__typename
}
}
}
}
`;tYou might write your types like this:
export type TagsDataQuery = {
readonly project: {
readonly __typename: "Project";
readonly id: string;
readonly tags: {
readonly __typename: "TagsConnection";
readonly totalCount: number;
readonly nodes: ReadonlyArray<{
readonly __typename: "Tag";
readonly id: string;
readonly title: string;
readonly color: string | null;
} | null>;
};
} | null;
};
export type TagsDataQueryVariables = Exact<{
id: Scalars["UUID"];
offset?: InputMaybe<Scalars["Int"]>;
limit: Scalars["Int"];
}>;
const {
data: { project },
} = await apolloClient.query<TagsDataQuery, TagsDataQueryVariables>({
query: PROJECT_EXPORT_TAGS_QUERY,
variables: { id: projectId, limit: 25, offset },
fetchPolicy: "network-only",
});With graphql-codegen we get generated types like this:
const {
data: { project },
} = await apolloClient.query<graphqlTypes.TagsDataQuery, graphqlTypes.TagsDataQueryVariables>({
query: PROJECT_EXPORT_TAGS_QUERY,
variables: { id: projectId, limit: 25, offset },
fetchPolicy: "network-only",
});PostGraphile
We don’t only apply code generation to generating types, we also make use of PostGraphile to avoid handwriting GraphQL resolvers. Handwriting GraphQL resolvers not only expose a situation where a bug can hide but also introduce the infamous n+1 problem. PostGraphile basically reads your PostgreSQL schema and generates for you a GraphQL API.
Previously we mentioned how our front-end uses graphql-codegen to generate interfaces from our GraphQL queries and GraphQL schema, which is itself generated by PostGraphile from our PostgreSQL schema. The types on our back-end are also generated by PgTyped from our PostgreSQL schema.
This means our PostgreSQL schema is the source of truth for a significant portion of our codebase, from the back-end to the front-end, which uniquely demonstrates the power of this approach.
Summarising the benefits of code generation at Dovetail
Save engineer time and improve efficiency
- Our code generator approach eliminates entire classes of errors, including basic human errors such as typos, saving even more time. The surface area where bugs may occur is also smaller.
- No inconsistency between models, which can occur with human-coded solutions.
- No longer need to write GraphQL query resolvers. We can avoid writing SQL queries, unless for a mutation. We also solve the GraphQL n+1 problem.
- Reduce the negative impact on customer experience by reducing the errors that make it to production.
- As an additional benefit, prevent SQL injections.
Database model/schemas as the source of truth
- We depend on our DB as the source of truth, rather than independently writing models for each. We no longer need to maintain the mapping between our DB model and our application interfaces.
- The application model is always in sync with the database model and we won’t run the risk of having multiple application models for the same database model.
Closing
At Dovetail, we ideally write code that writes other code. Auto-generation of code and types has been extremely effective in enabling that for us. It’s made a significant impact on overall engineering efficiency and the reduction of wasted effort.
If you’re interested in code generation practices and would like the opportunity to work with our stack, then come join us!
