How To Plan and Execute an Event-Driven API Project Within Your Organization?
How to leverage the API-first approach to plan and deliver an event-driven, asynchronous API project?
Unlike their RESTful counterparts, you don’t call and wait to get a response from event-driven APIs. They work asynchronously, meaning that you subscribe with an API server to get notified whenever something interesting happens at the API server. These notifications are delivered to you as asynchronous events. The API server can be a message broker, an event streaming platform like Kafka, or a web server supporting protocols like WebSockets or SSE.
Well, that’s the 33000 feet view of event-driven APIs. I’m sure you can easily find many materials on the Internet explaining them. If you are new to this space, you might as well read my article to get an idea.
We can find good content highlighting the “what” and “how-to” parts of the event-driven APIs. What I felt missing there was the content that focuses on the developer workflow of building an event-driven API from scratch. Hence, I came up with this post explaining the bare-minimum process to kick-off an event-driven API project within your organization.
This post discusses the different lifecycles of an event-driven API project: inception, design, collaboration, implementation, testing, and maintenance.
The Dogewatch API
Imagine your company is planning to launch a new API to publish a real-time feed of Dogecoin prices. Feed consumers would be the trading systems, mobile apps, crypto wallets, and media outlets such as TV stations and websites.
The feed will be sourced from a currency exchange like Coindesk, followed by some processing in the middle, and published into a message broker topic afterward. The consumers are expected to subscribe to that topic with proper credentials to consume the feed.
The initial architecture looks like this:
You are tasked as the product manager for this feed. Right now, you don’t have anything in your hand. So, how would to plan to kick things off and deliver this in production?
Let’s find that out.
Step 1: Building a team
Now that you have a clear use case and a delivery target ahead of you. What’s pending is assembling a team to deliver the price feed.
Before you begin the recruitments, let’s see what roles we need in a typical API team.
API product manager
The product manager (or PM for short) oversees the entire lifecycle of the API, from inception to sunset.
The PM often sits down with the business stakeholders to figure out the business expectations of the API, converts them to the API model, and leads the engineering team to build that. Oftentimes, PM owns the API design specification and governance activities such as versioning, lifecycle management, etc.
In our case, you are playing this role.
API Engineer/architect
The API engineer/developer builds the API according to the design provided by the PM. In simple terms, they power the API backend.
For larger API projects, consider an engineering team led by an architect.
In our example, this team’s responsibility is to extract Dogecoin prices from Coindesk, add some value, and publish it to your message broker. So it is essential to decide on your API infrastructure before hiring the engineer because the skillset matters.
Implementation and performing various tests, including unit, integration, performance, security testing, code reviews, and DevOps, are responsibilities you can assign to this team.
API tester/QA engineer
API testers ensure that the API delivers according to the API specification. That includes schema conformance, performance testing, usability testing, etc.
Well, these are the bare-minimum roles that you can consider adding to your API project team. Assuming API engineers can multi-task at the beginning for coding and DevOps, you can always scale out later when the project is getting more funding.
Step 2: Choosing a development methodology
Once you figure out the project team, the next task is to select a development strategy.
Out of many software development methodologies, we can pick the API-first approach for the implementation based on its advantages, such as:
- Parallel software development
- Reduced risk of failure
- Improved developer experience
- Optimized development costs
In the API-first approach, we first create an API specification that describes the various aspects of the API. That specification is shared among various project stakeholders to collaborate over the design. Once an MVP version is agreed upon, teams will start the actual implementation.
That is in stark contrast to the traditional model, where implementation starts first, and API specification follows afterward.
In our example, we need to create a specification to document the interfaces of the Dogecoin price feed, which should at least include:
- What are the URLs and ports of the broker(s) that consumers should subscribe to receive the feed?
- What topics/queues will be available on the broker?
- For each topic, what are the expected input/output message formats/schemas?
- Which security scheme should be followed to get connected to the broker(s)
We will select the Asyncapi specification to document the above information as we advance.
Step 3: Creating the AsyncAPI specification
Now that your team is ready to kick off the project. But, from where would you start?
Let’s create an AsyncApi specification to document the price feed API.
AsyncAPI specification has been derived from its RESTful counterpart, OpenAPI, to document asynchronous, event-driven APIs. The specification provides a rich set of language features to describe and document an event-driven API, including the broker information, event schema, event channels, operations, etc.
The file format must be JSON or YAML; however, only the subset of YAML that matches the JSON capabilities is allowed. For our example, the specification file looks like this:
Typically, this document is initiated by the person who owns the API design, which is the API product manager. You can use a tool like AsyncAPI Studio, Apicurio Studio, or a text editor to author the specification.
To understand the AsyncAPI specification further, you can read my article below.
Step 4: Host the AsyncApi specification and collaborate on the API design.
Now that you have just created the API specification document. The next step is to share that with your team members to gather feedback.
For that, you can’t just email them the file and ask for feedback. Instead, it should be hosted in a central place where everyone gets the necessary rights to view, comment, and edit over the design. Each change to the specification must be tracked and versioned. Apicurio is an open-source API and schema registry that facilitates collaborative editing for AsyncAPI documents.
Alternatively, you can commit the specification file into a Git repo and allow team members to give their feedback via PRs.
No matter what logistical approach you use, the collaboration should happen for multiple iterations, and you should invite all the business stakeholders for their feedback. For instance, you could invite a potential feed consumer from a media outlet and show him the early version of the API design.
This stage aims to gather consumer feedback as early as possible to refine the design and reduce the development cost.
Step 5: API mocking and testing until the backend is ready
At this stage, you have finalized the MVP version of the price feed’s design. All stakeholders have agreed upon which broker to use, the security schemes, and the input and output event formats.
Now let’s go ahead and build this!
Building the API backend will take some time due to the complexity. But, the testing and the consumer teams can’t be blocked until it is ready in full.
The beauty of the API-first approach is that the downstream teams can work parallel when you have a solid API specification in place. Because they exactly know the “contract” they should work on. No teams will be blocked or depend on upstream teams. That drastically reduces the time to market.
In our example, the API tester can start writing API test cases soon after the AsyncAPI is finalized. Tools like Microks allow developers to generate API mocks and tests based on an AsyncAPI specification.
Once the API mocks are in place, the early adopters can build the consumer applications against the mock API. They don’t wait until the backend is delivered. To speed things up, AsyncAPI provides code generators to generate the client-side SDK as well.
Step 6: Deliver the API backend.
In the meantime, the API engineers can extract Dogecoin prices from Coindesk and publish them to the broker. That might take some time to cover the implementation, testing, and setup of the DevOps pipeline.
The beauty here is that the backend team is independent of what others are doing. They block nobody. That gives them the flexibility to choose the best tech stack they are familiar with but deliver the result strictly according to the initial AsyncAPI specification.
Step 7: Putting it all together
Once the backend is ready, the team can switch from the mocks to the actual production backend. Guess what? The transition will be smooth as everybody has been playing their part according to the initial API specification.
Ideally, switching API integration tests to production would only require changing the backend URL and security credentials. The same goes for the consumer applications that depend on the backend.
Takeaways
Here, we barely scratched the surface of a typical event-driven API implementation process. The process follows the API-first model to reap the benefits of parallel development, testing, and incremental design iterations.
Your mileage may vary based on the use case, complexity of the business requirements, and the technology. But you can still consider this post as a guiding star whenever you come up with idea to build the next great real-time data feed.
