Schedule MuleSoft Application with Third Party Scheduler Tidal — A Pattern

Introduction:

MuleSoft has support for scheduling MuleSoft applications with an out-of-the-box scheduler component. However, there are organizations who prefer to schedule all enterprise applications through an enterprise level scheduler application like Tidal, instead of an application native scheduler component. This article describes an integration pattern that can be followed to schedule a MuleSoft application using a third party scheduler like Tidal.

Tidal Software is an enterprise workload automation solution that orchestrates a complex sequence of jobs across multiple applications, systems, and IT environments. This also helps centrally manage the scheduled jobs across the enterprise. Tidal provides a status monitoring screen that creates a single pane view of job statuses running on multiple applications. Failed jobs can be re-processed by submitting the jobs centrally from the Tidal console, a preferable alternative to logging into each enterprise application and separately re-processing. For example, some jobs are implemented with MuleSoft flow and some jobs are implemented in Oracle Apps. Using Tidal, all of these jobs can be centrally managed. Also, the sequence of jobs is important to maintain; for example, if jobs implemented on Oracle Apps are executed successfully, then jobs implemented on MuleSoft are supposed to be invoked. Using Tidal, this type of complex job execution sequence across disparate enterprise applications can be maintained in an automated way.

A centralized scheduler can also manage a central time zone for jobs implemented through different applications and from organizations across the globe.

We will discuss the integration pattern to schedule MuleSoft applications using Tidal in two major steps:

1. Pattern to invoke MuleSoft application from Tidal
2. Pattern to update MuleSoft job status through asynchronous callback on the Tidal job status portal

There are some business use cases where scheduling of MuleSoft applications is required, including the use of MuleSoft applications to sync data between applications, data replications between applications, and batch jobs or ETL/ELT implementations.

Pattern to Invoke MuleSoft Application from Tidal:

Configuring a REST Web Service to schedule through Tidal is automatically supported by Tidal. MuleSoft has rich support for exposing REST API so it is recommended to expose the MuleSoft applications as REST over HTTPs, which are supposed to be scheduled through Tidal. Also, it is recommended that Tidal pass some custom HTTP headers values exposed by the MuleSoft API for observability and traceability of transactions.

For example:

• x-client-name — to trace that transaction initiated by Tidal.
• x-correlation-id — uniquely identify the transaction.
• x-job-id — Job ID that will be sent on callback to update job status.

A sample screenshot of Tidal setup (here Job ID is also used for transaction correlation).

MuleSoft applications that are supposed to be scheduled with Tidal need to be exposed as a REST API. The API should also support some custom HTTP headers to grab job-related information as mentioned above. A well constructed RAML for API specification is strongly recommended. The GET method is preferred for Tidal to invoke the MuleSoft API (remember that the API specification is for Tidal to invoke the API). Upon invocation, the MuleSoft flow will start the data sync-up or batch processing, including reading or extracting data from an application. Then, the data is synced to target applications. Additional message processing will be completed using MuleSoft out-of-the box components.

MuleSoft updates job statuses on Tidal so that the operation team can properly monitor the jobs from Tidal. In the next section, we will discuss how to callback on the Tidal status REST API to update the job status from the MuleSoft flow.

Pattern to Update MuleSoft Job Status through asynchronous callback on Tidal Job Status Portal:

Batch processes and data sync-ups are often long-running processes so it is not advisable to send the final job status synchronously to the Tidal portal at the end. For example, if the application is deployed on MuleSoft CloudHub and input to Tidal through the Dedicated Load Balancer, then Tidal will send a timeout after 300 seconds, by default. However, the data sync-up job is still successfully executed on CloudHub runtime. This will create false alarms for the job monitoring and support team and unnecessary effort will be invested to track the false error further.

To avoid this, it is recommended to send the job status as a callback to the Tidal status portal. Tidal provides an out-of-the-box REST API for the job status from the application that implemented the job. This REST API supports basic authentication to grab the status of the job from authenticated applications. Some of the status code that Tidal supports to capture the job status is as follows:

The sequence of status updates through Tidal callback URL (Tidal REST API URL) on MuleSoft flow will be as follows:

1. Send a synchronous and immediate response 200 Ok to close the thread for direct connection between Tidal and MuleSoft gracefully.
2. Use an asynchronous scope in the MuleSoft flow and implement the actual data processing flow. MuleSoft should not wait until the end to send back the 200 ok response to Tidal.
3. In the MuleSoft asynchronous scope, send an asynchronous callback response with status 107 (externally defined) before starting the actual data sync-up process (or the core processing). This will indicate to Tidal that the job execution request has been submitted.
4. Once the job and either the core data sync-up or data processing are complete, MuleSoft updates the status from the flow through an asynchronous call back with 101 (completed normally).
5. If the MuleSoft application stopped because of an error, then from the error flow (from on-error propagation or on-error continue scope) send status code 103 (completed abnormally) through asynchronous call back.

A sample request to update the status on Tidal using the Tidal REST API is as follows:

— — — — -

<tes:CmdLine.tescmd xmlns:tes=”http://www.tidalsoftware/client/tesservlet”>

<cmd>jobset -s 101 -i 1111 </cmd> /*set status code 101 for the jobid extract from incoming request*/

</tes:CmdLine.tescmd>

</entry>

— — — — —

The overall integration pattern can be depicted as follows:

A sample MuleSoft flow, with an HTTP listener is depicted below (note: it is recommended to use an API with proper API specification for better management). This flow is scheduled through Tidal to sync customer data between CRM applications and asynchronously update the job status to the Tidal status port.

Conclusion:

In data integration or ETL/ELT implementation, it is very common to schedule a MuleSoft application to start during a particular time frame (i.e., daily, weekly, monthly, etc.). Data organizations also commonly need to execute series of sequential, scheduled jobs. For example, different executions need to be controlled based on the status of the parent and child jobs. This type of complex job scheduling can be best implemented with third party scheduler applications like Tidal. Since MuleSoft is gaining popularity in data organizations, it is becoming a growing requirement to use patterns to schedule MuleSoft applications with Tidal (or similar enterprise scheduler applications).

I hope this article provides an idea of how a pattern for scheduling MuleSoft applications with Tidal can be established.