GETTING STARTED | REST API | KNIME ANALYTICS PLATFORM
You can REST easy with KNIME Analytics Platform
Graceful handling of HTTP errors in KNIME Analytics Platform 4.4
Introduction
As data scientists and data engineers, we commonly run into scenarios in which we want to consume RESTful web services (or “REST APIs” for short). We might want to retrieve some social media data from the Twitter API or Reddit API, utilize pre-built machine learning services from AWS or Google, or integrate with private APIs that are hosted within our organizations.
The challenge is consuming REST APIs isn’t as easy as we’d like. Perhaps it’s straightforward enough to call a web service while we’re testing locally, but a production-ready integration is another story.
Vendors often implement cost-savings and security measures such as rate limiting on their APIs to prevent their resources from being over-utilized. Some APIs might fail intermittently for one reason or another. In order to seamlessly integrate with such APIs, we need solutions that can handle the occasional (or frequent) error.
Fortunately, there are new updates to the REST Client Extension for KNIME Analytics Platform which help us integrate REST APIs into our data flows in a production-ready fashion.
What’s new with the REST nodes in KNIME Analytics Platform 4.4?
The example workflow below demonstrates the significance of the error handling updates to the REST nodes in KNIME.
In KNIME Analytics Platform 4.3 or earlier, some fairly advanced techniques would be required for handling the occasional HTTP error. We’d have to implement a recursive loop to re-attempt any failed requests, and we’d also have to use the Wait… node to pause for a moment between each request (to avoid any rate-limiting errors).
In KNIME Analytics Platform 4.4, robust error-handling capabilities have been added to each REST Request node (GET, PUT, POST, DELETE, etc.).
Configuring the error handling features
Below is the new Error Handling configuration tab for the REST Request nodes.
The Server-side errors (HTTP 5XX) section allows us to control how the REST nodes will handle server errors from an API (i.e., server downtime or inability to fulfill requests).
The Rate-Limiting error (HTTP 429) section allows us to determine how the REST nodes will behave if an API we’re calling begins to throttle our requests due to overuse. We can pause execution before continuing to send requests, and the node will also re-attempt the original failed request.
The other sections in the Error handling tab determine the behavior for other errors which wouldn’t typically merit a retry, such as improper authentication or bad requests. We can either set the node to fail in such instances or continue executing (failed requests would display the status code error message in the output table).
Conclusion
The updates to the REST Client Extension in KNIME Analytics Platform 4.4 provide an appropriate level of abstraction over API integrations for us. We no longer need to worry about how to gracefully handle HTTP errors — we can instead focus on how we want to utilize various APIs in our workflows.
Workflow examples and additional documentation for the REST Client Extension are available on the KNIME Hub.
