WSO2 Enterprise Integrator — How to stream a large file to an HTTP server
Sometimes when we integrate different systems together using enterprise integrator software, we need to handle large files. In some use cases, there is no need to “interpret” this file but we just need to transfer it to another location. Legacy systems are still based on files. Features like file streaming is essential to expose such existing functionality using newer technologies and achieve legacy modernisation.
Let us see how WSO2 Enterprise Integrator can be used to stream a file to an HTTP service stepwise.
Mock HTTP Service
I have come up with a mock HTTP service which accept a message as a POST request and stream it into a file on the server itself. Following is the code snippet that shows how it streams received bytes to the file.
The server code is hosted here. You can download the project and run it using your preferred IDE. It will start a HTTP server listening on port 9000. It will accept POST requests with the context /sendfile.
WSO2 EI Mediation Logic
- WSO2 EI has a rich connector ecosystem. To deal with files, it has “File Connector” extension. You can use it to read the file.
- Using WSO2 Integration Studio you can design your logic as below.
- Define an API to invoke file reading process.
2. Read the file using file connector read operation. Specify streaming =true parameter. (To know how to use file connector please refer this documentation). Please note how contentType parameter is specified.
3. Forward the message to HTTP endpoint
All above logic can be exported as a carbon application that is deployable on WSO2 EI 6.x or WSO2 EI 7.x using WSO2 Integration Studio itself.
WSO2 EI Runtime Configuration
Download WSO2 EI and edit [EI_HOME]/conf/axis2/axis2.xml file as below adding message builder and formatter for the new content type application/file.
Testing
Run WSO2 EI server using integrator.sh (or integrator.bat on windows). Start the mock service as well. Invoke the API with some payload using Postman. Here, we are not using this payload in the mediation logic. As API just to invokes the flow, any payload will work.
I am using one of my favourite movie file here as the source file to read and stream using WSO2 EI. As specified, only .mkv files will be read from file connector. I place it in the location specified at the read operation.
After forwarding it to the HTTP endpoint you will see the file has been streamed to the location hard coded at the server side code. Make sure to play it to make sure it is not corrupted!
If we observer memory usage, there is only a slight increase in memory even if 2.5G movie file is streamed.
How to Improve
- You do not need to hard code location at file connector read operation. You can read it from input message to the API.
- How to continuously listen to a file location and stream a file if there is one? You can add a EI Task instead of the API to invoke the file read. This task can be scheduled to run at mid-night everyday or once a second — useful! You may even use VFS Transport of WSO2 EI (with transport.vfs.Streaming=true) instead of the task and the file read. I haven’t personally tried this :)
PS: If you can your Mock server on a different machine on the same LAN now you have a way to transfer your files without a pen-drive!!
