Developer’s Guide — streaming data between API layers in MuleSoft
Understanding Streaming Data
Before you figure out what the streaming of data is, I have to pinpoint that this will be a developers guide on how to implement streaming in Mule 4.
Streaming data involves processing and transmitting data in a continuous flow, rather than in discrete blocks. This real-time approach is ideal for scenarios where low latency and efficient resource utilization are crucial.
Implementation Scenarios
Requesting the data
In the first scenario, we will have the following setup: Experience API, Process API, and System API.
System API unlocks data from SAP S/4HANA. By default, the SAP S/4HANA Query component in Mule 4 supports a Repeatable file store iterable streaming strategy. It means that you do not have to adjust the SAP S/4HANA component, but you have to adjust everything that is connecting our APIs into one functional cluster.
Starting from Experience API, if you request a Process API by HTTP Request component you have to do the following steps:
- Manually adjust HTTP Request configuration and add property streamResponse=”true” through Configuration XML view
<http:request-config name="papiRequestConfig" doc:name="HTTP Request configuration" doc:id="80a409b5-123g-40e7-81f9-524b7f3ba840" basePath="papi-api-v1" responseTimeout="1000">
<http:request-connection protocol="HTTP" host="0.0.0.0" port="8092" streamResponse="true"/>
</http:request-config>2. Set the MIME Type and the parameter “streaming=true” in the HTTP Request component itself and HTTP Listener in the main flow as well
3. If you consume or transform the data anywhere after retrieving it from the source, then you need one more thing. Inside e.g. Transform message, add a property “deferred = true”. The property must be in the same line as an output.
4. Repeat step two for each HTTP Request component which is on the path of streamed data
Posting the data
In the second scenario, we will change the setup a little bit. Besides having Experience API, Process API, and System API we will add one more System API. Another System API is working with Database. In this scenario, we will request the data from SAP S/4HANA in the first System API and POST the data into Database through the second System API.
HTTP Request component with method POST demands only one add-on on top of everything that has been written above — define “Content-Type” inside the Headers tab.
output application/java
---
{
"Content-Type": "application/json; streaming=true",
}*If the streaming does not work with the given setup when POSTing, add “deferred=true” to the parameters inside the MIME Type tab.
Conclusion
In this developer’s guide, we’ve explored the intricacies of streaming data between API layers in the context of MuleSoft’s Mule 4 platform. We delved into the fundamental concept of streaming data, which involves the continuous processing and transmission of data, as opposed to the traditional block-by-block approach. This real-time paradigm proves invaluable in scenarios where minimizing latency and optimizing resource utilization are paramount.
