METL: Defining Custom File Names
METL is a web based integration platform used to perform ETL operations. A common task for any ETL is writing to a file. This can be a daily log, a report, or a data file to be consumed by another application. Depending on your requirements, you have a variety of options for choosing a filename.
The most common method I see for writing files is to use a static filename but sometimes we need more control. This can be as simple as adding a date or a sequence ID to the filename. With the use of the script component, anything is possible.
The simplest way to configure the text file writer is to provide a simple string for a file name.
There are times when it would be helpful to maintain a version of the file sent. In this case, the easiest solution is to include the date in the file name. We can do this by using the built in METL Flow parameter “_date”. METL will substitute any parameter tokens found in a file name to determine the updated flow name. The following example will create a file with the name, “log-2018–12–24.dat”.
There are situations that may require a more complicated filename or maybe you need a date format other than YYYY-mm-dd. I see this most often when we needed to replace an existing integration and the downstream system expects a specific date/timestamp pattern. In this case, we can use the power of the script component to generate any filename.
The following steps demonstrate how to create a filename with a specific date format.
- Add a new Groovy Script Component and update the onHandleMessage script. In our example we want to create a file with the date and time. Example: MY_FILE_ID_20181224_224813.dat
Lines 1–4, are obvious. Line 5 replaces the standard onHandleMethod function, forwardMessage(). forwardMessageWithParameters(…) allows us to append additional parameters to the incoming message and forward it on. We only want to append the generated filename before forwarding the incoming message.
2. Wire up the new groovy script in front of the text file writer.
3. Configure the text file writer to use the generated filename. Make sure the option, ‘Get File Name From Message’ is checked. Also update the setting, ‘Param Name for Message Based File Name’ to be set to “filename.property”.
At this point, you should be able to run your job and generate your updated file.
If you haven’t had a chance to try out METL you can download the application from JumpMind. If you are interested in running the latest, want to contribute to the community, or would like to submit an enhancement you can go directly to the source on GitHub.
