Trips, Traps and APIs
Navigating legacy app integrations in government
In Digital Services at MAPC, we commonly integrate our apps with other software through application programming interfaces (APIs). Being able to integrate with other software allows us to avoid duplicating work and deliver software quicker and cheaper. Since we simply send an API a message and process its response, we do not need to know how it does what it does. The API can change and improve and as long as it conforms to our expectations, and we get the value of updates to it for free. Unfortunately we have learned there are two types of APIs: easy and hard to use.
The challenging APIs often arrive from software marketed to businesses like CRMs and GIS software. These APIs often feel like afterthoughts bolted on to the main product so that the folks in marketing can check off the “we have an API” checkbox. Challenges from these APIs include poor documentation, bad connections, and un-intuitive responses. After tackling these challenges with our past few projects I have developed some patterns to more easily handle them.
Bad Connections
In the case of bad connections we often use a background job queue, such as Sidekiq, to decouple requests from the rest of our application and allow for automatic retries. We build the application to throw an error if the background worker fails to retrieve the information it needs from the external API. Then we have the background worker log its errors and send a report to us so we can individually debug problematic requests. A background job processor also is important for speeding up a large number of API requests by introducing concurrency and tools for respecting API limits. Sidekiq allowed us to cut the amount of time to import data from a vendor product by 50%.
Poor documentation
This often requires an empirical analysis of how an API works to confirm behavior. A testing tool like Paw makes verifying and documenting API behavior faster. Typically I will try to find the closest request case in the documentation and then build examples in Paw before writing the code for my background worker class. I will also use linters to catch syntax errors in JSON and other parts of the request. When coding in Ruby I will use the %Q notation to wrap JSON requests to avoid parsing errors while keeping JSON readable. These three techniques have made working with APIs faster and easier.
General Quirks
Unfortunately quirks from an API that is not widely used often require direct conversation with the vendor to resolve. The lines between unexpected, undocumented, and unexplained can feel blurry when dealing with a product that many people do not use. Instead of being explained in blog posts or StackOverflow questions, the knowledge you seek is often inside someone’s head. Often an engineer or support person that works for the vendor. Fortunately, the vendor also has access to logs on its side of the request so they can see what is going wrong in more detail if the API does not return useful error messages.
API integrations can save a lot of time and money when they go smoothly. However the tougher ones can eat up a large amount of time as unexpected problems reveal themselves. The techniques above can help minimize the amount of time you spend fighting with tough external APIs. However the best way to stop fighting with external APIs is to stick with ones that are used by a large number of people and well documented. It is always easier to walk the well worn path.
