Searching all files in a root directory using WSO2 Enterprise Integrator

WSO2 File Connector — Search operation

WSO2 Enterprise Integrator is a fully open-source product that can be used to integrate/ connect any service, data source, application with one another.

WSO2 Enterprise Integrator comes with a set of connectors for popular business applications. So that those applications can be connected to WSO2 Enterprise Integrator and use that applications’ operations easily.

Those enterprise-ready connectors are stored in the WSO2 Connector store. WSO2 connector store provides these connectors free of charge and all the connectors, connector documentation, and connector source code is publicly available.

WSO2 File connector is a connector designed to connect to various file systems and perform various file operations such as search files, move files, read files and so on. This file connector uses apache commons VFS(Virtual File System) transport for processing.

I want to implement a use case to get a list of all JSON files remaining under a root folder and its child and grandchild directories for some processing purposes. And this can be easily done by using the WSO2 File connector using its search operation.

File Directory

At this point, I hope you have installed WSO2 Integration studio, to develop your message flows and you have downloaded a WSO2 Enterprise Integrator binary and you have started it in the integration profile.

If you haven’t please refer following for the instructions on how to do those,

• Download and install WSO2 Integration studio

https://docs.wso2.com/display/EI650/Installing+WSO2+Integration+Studio

• Download WSO2 Enterprise Integrator

https://docs.wso2.com/display/EI650/Installing+the+Product

• Start WSO2 Enterprise Integrator in Integrator profile

https://docs.wso2.com/display/EI650/Running+the+Product?src=sidebar

Now open your Integration studio, and create an ESB solution project to create your message flow. Add a REST API to that project by right-clicking on that ESB solution project and click on New > REST API.

The next thing to do is Downloading and Installing the file connector. Follow the below steps to install it at the integration studio and enterprise integrator.

• Installing a connector to WSO2 Enterprise Integrator and enabling it

https://docs.wso2.com/display/EI650/Working+with+Connectors+via+the+Management+Console

• Installing a connector to Integration studio

https://docs.wso2.com/display/EI650/Working+with+Connectors+via+WSO2+ESB+Tooling

Now when you open your REST API in designer view, you will see a new set of operations in the pallette under ‘File connector operations’

Drag and drop a search operation to the in-sequence of the REST API resource. Configure the following properties in your search operation.

• Source — This is the root folder location, here no regular expressions are supported
• File pattern — Specify a regular expression or exact file name to search files. This file pattern does not consider the file location path.
• Recursive search — If you want to search all the subsidiaries under the root folder, set this value to ‘true’, else set it to ‘false’

There are other few properties in this operation other than above. Above are the only properties we need for our use case.

Then add a property mediator before the search mediator with the following content.

<property action=”remove” name=”NO_ENTITY_BODY” scope=”axis2"/>

This property is useful to return search operation output as a response from REST API.

Here,

Next, add a respond mediator to the message flow to return the response of the search operation.

Now your message flow will look like following,

Now build and deploy this artifact. For instructions

Now create a sample directory structure as in image 1, and execute your API, this will return all the file locations as an output.

Next steps

• Let's think you need to filter out certain files a specific file folder or specific folder levels to the output.

We can achieve this by adding an iterate mediator and a filter mediator after the search mediator and specify a regex to filtering

This article explains a part of a use case that I developed, If you want to view the full use case source code you can view it from the below link.

https://bitbucket.org/andradikla/ei_fileconnectordemo/src/master/

Thank You