From manual to magical: Automating API docs and code autocomplete with TypeScript and OpenAPI

Creating and maintaining API docs is not easy. What if you could automate it so your docs are up-to-date without any manual work?

Types demo

Streamlining API documentation at Pipedrive

Like many companies, Pipedrive used to maintain its internal services API documentation in the repository README file. This meant that any changes developers made to the API also required a manual update to the documentation. Developers sometimes overlook this step, causing a drift between server implementation and documentation.

Fortunately for Pipedrive, most of our internal services that expose APIs are written in TypeScript (hereafter referred to as “TS”) and use the Fastify framework. These two key factors allow us to take advantage of the following process:

• Fastify lets you validate your API parameters using JSON Schemas.
• Tools like TypeBox enable you to generate both JSON Schemas and TS types.
• Fastify has a core plugin (fastify-swagger) that transforms these JSON Schemas into the OpenAPI format.
• Finally, you can use a tool like Swagger to display the generated API documentation in a consistent and automated way.

By now, you can probably guess the type of solution I will discuss. Indeed, we could stop at transforming our API documentation into the OpenAPI format. But why stop there? Let’s take advantage of libraries like openapi-typescript. We can transform those OpenAPI schemas back into TS types, which can then be leveraged through a unified internal API client library. In doing so, we could harness the power of module augmentation and introduce two exciting features for API consumers:

• Automatic type checking for correctness and safety, ensuring that your API calls adhere to the expected types, reducing runtime errors.
• Autocomplete for API calls to the provider service, enhancing developer productivity by providing intelligent suggestions and reducing the likelihood of mistakes.

With these enhancements, we could maintain up-to-date API documentation and improve the overall development experience by directly integrating type safety and code assistance into our workflow. It might sound too good to be true — I felt the same when Diogo first presented the idea. But let’s dive into the journey of the system we developed, and hopefully by the end, you will be just as amazed about this innovation as I am!

How to ensure accuracy and consistency in API documentation

Before we start digging into the solution, let’s begin by highlighting the different problems we encountered.

In the fast-paced world of modern web development, maintaining accurate and up-to-date API documentation and consistent schema definitions is crucial for ensuring smooth communication between services. Here’s a closer look at the issues commonly faced by developers.

• Duplicated implementations of server schemas and client services: separate implementations of server and client schemas often lead to inconsistencies and increased maintenance efforts since each update must be manually replicated on both sides.
• Zero test guarantees for client-server compatibility: even with well-written tests, you can’t be sure that clients and servers will interact correctly. Tests usually focus on isolated components rather than the actual API contracts end-to-end.
• Maintaining Updated API Documentation: keeping API documentation updated with every change takes a considerable amount of work and is prone to human error, making it challenging to ensure the documentation accurately reflects the true state of the API.
• Low trust in API documentation accuracy: due to the challenge of maintaining updated documentation, developers are reluctant to trust the accuracy of API docs, leading to redundant manual checks and inefficiency.

All these challenges reduce developer efficiency. The time spent managing and checking API documentation and schemas takes away from building new features and improving existing ones, slowing down the development process.

By automating the generation of TS types and integrating them with tools like OpenAPI and Swagger UI, we can address these issues, ensuring accurate documentation, reliable type checking, and enhanced developer productivity.

Building an automated API documentation system

In this section, we’ll dive into the architecture and workflow of our automated system. Our solution comprises two main components: the Types Service and the Types Library.

Types Service

The Types Service is the backbone of our system and is responsible for:

1. Receiving schemas and accepting the name of the API provider and the OpenAPI schema via HTTP requests.
2. Schema validation, utilizing AJV to ensure the received schema conforms to the OpenAPI specification.
3. Storing schemas. Once validated, the schema is stored in a database for easy retrieval and use.
4. Serving Swagger UI. The service also hosts Swagger UI, enabling developers to view and interact with the stored OpenAPI schemas through a user-friendly interface.

Types Library

The Types Library works as a versatile tool for both API providers and consumers.

1. For API Providers: it facilitates the submission of OpenAPI JSON schemas to the Types Service, making the necessary HTTP requests and schema transmission process.
2. For API Consumers: it enables the retrieval of stored schemas from the Types Service and converts them into TS types.

Inside our automated documentation process

Now that we know the challenges we are facing and how to address them, let’s discuss the solution we chose to implement. Our approach is structured around three main flows:

1. Push Flow, where API providers submit their schemas to our central repository.
2. Documentation, where developers can consult up-to-date documentation for providers’ services.
3. Pull Flow, where API consumers retrieve the stored schemas to generate TS types, ensuring correctness and safety.

We will explore each one of these flows in detail, demonstrating how they work together to create a seamless and efficient system for managing API schemas and types.

Push Flow

The Push Flow is designed to streamline how API providers submit their OpenAPI schemas to the central repository. Here’s how it works:

1. Schemas Provider. The API provider merges changes to the main branch of their repository.
2. Push Schemas GHA. A GitHub Action (GHA) triggers, fetching the configuration file and initiating the push of schemas.
3. Types Library. This reads the provided schema files.
4. Validation. Each schema file is validated using a validator (AJV).
5. Types Service. Valid schemas are pushed to the Types Service.
6. Database. The Types Service stores the schemas in the database.

Documentation

After the schemas are pushed into the central repository, developers can access the most recent documentation through the Swagger UI. This page is served by the Types Service, which ensures that any changes to the schemas are immediately reflected in the documentation, providing developers with the latest API information at all times.

Here you can see what the documentation UI looks like:

Pull Flow

The Pull Flow allows API consumers to fetch the stored schemas and convert them into TS types. The Types Library leverages the openapi-typescript tool to do this, ensuring API provider implementations are consistent with their schemas, providing robust type checking and reducing code duplication and potential runtime errors. Additionally, we have an OpenAPI client that’s enhanced by the generated types, allowing developers to benefit from autocomplete and type safety when calling internal services.

Here’s how it works:

1. Backend/frontend services use the Types Library to build TS types for OpenAPI schemas and call internal services with type checks.
2. The Types Library fetches the necessary OpenAPI schemas from the Types Service.
3. The Types Service retrieves schemas from the database.
4. The OpenAPI client communicates with internal services, ensuring type safety through the generated TS types.

Automating the schema submission and retrieval processes ensures that API documentation is always up-to-date and that TS types are consistent across services. This process enhances developer efficiency and confidence in the accuracy of the documentation. Additionally, using the OpenAPI client with type safety improves the developer experience by providing autocomplete features and reducing the likelihood of runtime errors.

Why automated API documentation and TypeScript types matter

Implementing our automated API documentation and TypeScript type generation system has brought several advantages, significantly improving both the development process and overall project reliability. Let’s now take a look at the most outstanding benefits.

Consistency and accuracy

Automated schema validation and storage ensure that API documentation is always accurate and consistent with the actual implementation, eliminating possible errors from manual updates. Swagger UI served through the Types Service is always up-to-date, reflecting any schema changes immediately and reducing the risk of using outdated API specifications.

Enhanced developer efficiency and confidence

Developers save time by not manually updating documentation, allowing them to focus on more productive tasks. This speeds up development cycles and releases. Automation ensures reliable and up-to-date API documentation, reducing the need for manual checks and fostering smoother team collaboration.

Improved type safety and developer experience

Automatically generated TypeScript types ensure that providers and consumers comply with the same contract, reducing runtime errors and enhancing code reliability. Developers benefit from autocomplete and type-checking features in their IDEs, improving their overall experience.

Simplified schema management and robust validation

A centralized repository simplifies the management and retrieval of API schemas, providing a single source of truth for both providers and consumers. Using AJV for schema validation ensures all schemas conform to the OpenAPI standard, catching errors early and preventing invalid schemas from being used.

Scalability

The system efficiently handles multiple schemas from different providers, scaling well as the number of services grows without compromising performance. As we add more services, this scalability ensures that our documentation and type generation processes remain fast and reliable. This is crucial for supporting the needs of our development teams and maintaining a high standard of service quality.

Impact and adoption

We developed the solution in 4 weeks and made it available to teams in October 2023. Since then, the adoption has been remarkable. In the 8 months following its release, 15 teams have integrated the system into their workflows, and nearly 60 services currently provide automated API schemas. To facilitate this adoption, we integrated our automated schema generation and documentation process into an existing service template used by teams to create new services. This integration made it easier for teams to adopt the system right from the start. These developments showcase the system’s effectiveness and the substantial improvement it brings to our development process.

Integrating these benefits into our development workflow has significantly improved the accuracy, efficiency, and reliability of our API documentation and type definitions. This solution accelerates the development process and enhances the overall quality of our services, leading to a more robust and maintainable codebase. With widespread adoption across teams and numerous services, our automation system’s impact is evident and substantial.

What’s next?

As we continue to improve and expand the internal adoption of our automated API documentation and TypeScript type generation system, we have identified several key areas for future enhancements.

Dependencies tracking and testing

Enhancing our system to track dependencies between services and libraries will bring significant benefits, allowing us to:

• Track which services and libraries rely on specific schemas, providing better visibility into the impact of schema changes.
• Alert service and library owners when dependent schemas are modified, ensuring they are aware of potential breaking changes.
• Run consumer tests against new schemas before merging changes to the main branch, catching issues early and improving system stability.

Support for other programming languages

While the current system works with Fastify and TypeScript, the publishing mechanism is designed to be language-agnostic. Services only need a configuration file that points to valid OpenAPI Schema definitions. With this in mind, future enhancements will include:

• Enabling type publishing workflows for a wider range of programming languages, making our system more versatile.
• Providing better integration and support for various development environments, ensuring that teams using different technology stacks can easily adopt automated type generation.

Transforming development by increasing trust and usability

The journey of automating API documentation and TypeScript type generation has been hugely transformative for our development workflow. By leveraging Fastify, TypeBox, and OpenAPI schemas, we have built a system that keeps our API documentation up-to-date and consistent with implementation. This automation saves time, reduces human error, and boosts reliability within our development teams.

Our setup efficiently automates schema submission and retrieval, making sure type definitions are accurate and consistent. The automated documentation served through Swagger UI provides developers with always up-to-date API specs, increasing trust and usability. Integrating type safety and code assistance into our development process has significantly improved the developer experience, reducing runtime errors and boosting productivity.

Looking ahead, we plan to enhance our system with dependency tracking, expanded testing and support for additional programming languages. These improvements will help us manage schema changes more effectively, support a wider range of environments, and maintain our commitment to high-quality, reliable services.

By adopting these practices, we have optimized our development process and improved the accuracy and efficiency of our API documentation and type definitions. This puts us in a strong position to handle the evolving challenges of modern web development, ensuring our APIs remain a strong foundation for innovation and growth.