When You Integrate with Third-Party APIs: Some Best Practices

Integrating with third-party APIs is common for software developers when building a web application. By leveraging external services, we are able to add new features, expand functionality, and streamline workflows. However, integrating with third-party APIs comes with its own set of challenges. In this article, I’ll share my experiences to keep in mind when integrating with third-party APIs.

API Doc is Not the Bible

API documentation is essential for any integration. However, it’s important to remember that documentation is not always complete or up-to-date, and sometimes it might even be incorrect. APIs often change over time, which means that relying solely on documentation can lead to issues. For example, I once encountered a situation where I wanted to use the “updated_at” field in a response to determine the latest transaction. According to the API documentation, this field was specified as a timestamp. However, when I tested it in the production environment, it unexpectedly returned “None” instead of a timestamp. The fact that the field was nullable was not clearly stated in the API documentation. To avoid these problems, we should take a proactive approach to understanding the API. This may include reaching out to the API provider for more information or testing the API in a canary testing.

Performing Canary Testing

Before integrating with a new API, it’s important to test it thoroughly. In addition to unit testing and integration testing, incorporating canary testing is a great approach for early issue identification prior to the official launch. At iCHEF, we consistently select a group of clients to participate in testing new features, guaranteeing that the API aligns with our requirements and operates smoothly. Despite the thoroughness of our integrated testing, canary testing remains invaluable in uncovering unforeseen bugs or errors.

Logging is Never Too Much

When integrating with third-party APIs, it’s essential to log everything. This includes API requests, responses, and any errors that occur. Logging is crucial for debugging issues and identifying trends over time. By keeping detailed logs, we can quickly pinpoint issues and troubleshoot them. If there’s an issue reported by a customer, you can always find the logs to figure out whether it’s caused by your product or if you should report the issue to the third-party company and respond to the customer accordingly.

Catch All Kinds of Errors

Usually, we remember to catch all the errors mentioned in the API documentation, but there are also network errors, server errors, and application errors to consider. By catching all errors and using an error-tracking platform (such as Sentry) to notify us in Slack channels, we can proactively identify unexpected errors and provide better error messages to improve the user experience. It’s also important to have a plan in place for handling errors. This may include retrying failed requests or providing fallback options.

Be Mindful of Rate Limits and Allowed Lists

When integrating with third-party APIs, it’s important to be aware of rate limits and allowed lists. Rate limits are often in place to prevent abuse of the API. However, sometimes we might encounter rate limits that are not in line with our expectations. For example, if we retry failed requests due to a timeout error, we might end up with multiple retries in a short amount of time, triggering restrictions on our service’s ability to call the API for a given period. Developers should be aware of these limits and plan their integrations accordingly. Allowed lists are often used to restrict access to the API to specific IP addresses or domains. Developers should ensure that their applications are on the allowed list before going live.

Advanced: System Design for a Third-party Payment Gateway

In this advanced discussion, let’s consider a third-party payment system as an example. Assume we have two basic APIs: Pay and Refund. At first glance, it might seem simple to just call these APIs when a customer requests a payment or a refund. However, real-world scenarios often present various complexities that need to be addressed. Let’s explore some of these situations and the system designs that can help us handle them effectively.

1. Handling Failures during the Process: What if a request to the third-party payment API times out? It’s important to determine whether the payment was actually completed or not. If the third-party payment provider offers a Check API, we can use it to verify the current status before retrying. However, if there is no Check API available, we might consider refunding the last transaction and retrying the payment. This approach depends on the specific user story and the capabilities of the third-party payment system.
2. Asynchronous Jobs for Payment and Result Verification: Depending on the user story and the nature of the third-party payment system, it might be beneficial to handle payments and result verification as asynchronous jobs. Instead of waiting for an immediate response after calling the Pay or Refund APIs, we can initiate these actions as background jobs and check the results independently. This approach allows for better separation of concerns and can improve system performance by freeing up resources for other tasks.
3. Callbacks and Webhooks: Sometimes you need to configure and handle callbacks or webhooks provided by the payment gateway. These mechanisms allow the payment gateway to notify your system of transaction updates, such as successful payments or chargebacks. First, design your system to handle webhook callbacks in an idempotent manner by using unique identifiers, such as transaction IDs or request tokens, to identify and track the status of each callback. Second, Implement a robust error handling mechanism. This can include retrying the webhook delivery for a specific number of times with appropriate backoff intervals. If all retries fail, store the failed webhook details in a persistent storage or queue for later processing or investigation.
4. Comprehensive Logging of API Calls and Results: Logging all attempts and results of API calls can be immensely helpful for monitoring and troubleshooting. By logging the details of each transaction, including the API endpoint, parameters, response, and any errors encountered, we gain valuable insights into the integration process. This comprehensive logging allows us to trace the flow of transactions and identify any issues or inconsistencies that may arise.
5. Monitoring and Alerting: Implement monitoring and alerting mechanisms to track the status and health of webhook deliveries. Set up notifications or alerts to be triggered when webhook failures occur or when the number of failed deliveries exceeds a certain threshold. This helps ensure timely awareness of any issues and facilitates prompt action.
6. Data Encryption and Security: Implement appropriate encryption protocols to secure the communication between your system and the payment gateway. Ensure that sensitive data, such as credit card details, are securely transmitted and stored. Follow best practices for data protection, such as tokenization or encryption at rest.
7. Designing for Extensibility: Considering extensibility is vital, especially when there is a possibility of integrating another third-party payment system in the future. To accommodate future integrations seamlessly, it’s important to design the system with modularity and flexibility in mind. This can involve using well-defined interfaces, abstracting away implementation details, and applying design patterns that promote extensibility. By adopting a modular and extensible architecture, we can easily integrate additional payment systems without disrupting the existing functionality.

Conclusion

Integrating with third-party APIs can be a powerful tool for software developers. By following some best practices like those outlined above, developers can avoid common pitfalls and ensure a smooth integration process. Whether you’re incorporating a new API or managing an existing integration, hope these best practices can greatly benefit your applications.