Batch Monitoring and Data Processing: Challenges, Automation, and Outcome

IT Operations is the heart and soul of any data-driven organization. The processes that support the data injection into Data Lake are vital, as this data is used in corporate decision making. Accurate data is essential to business growth and to build new business strategies.

Batch Job Monitoring
The biggest challenge when processing ETL jobs is achieving error-free data and end-to-end integration between applications. The introduction of a system that can run jobs automatically, does not need to be monitored at every step, and manage its own dependencies, would help reduce time constraints and error-prone manual intervention. The overall benefits of an automated system are increased, such as processing efficiencies and fewer mistakes.

Manual Activity
In addition to ETL jobs running smoothly, the manual activity also includes meeting Service-level agreements, job failures, root cause detection, and manual file processing.

Challenges

• In large organizations, where real-time data must be available around the clock, jobs are scheduled to run across the company because of which human support is required 24×7.
• The turnaround time for recovering from job failures is prolonged due to the human dependencies and collaboration of multiple teams.
• Monitoring multiple jobs simultaneously requires substantially more resources, and the results may still not be accurate. Example: identifying long-running jobs manually amongst 100+ jobs running parallels might cause service-level agreement miss.
• When engaged in mundane work like monitoring the jobs day and night, the human intellect is not utilized to the fullest. The repetitive work also degrades the quality of work and people’s interest in their jobs.
• Most of the organizations spend 30% of their revenue on operations as they continuously require more manpower for merely keeping the lights on.

Source: McKinsey’s Insurance 360º benchmarking

By implementing the following approaches, the challenges can be resolved to a certain extent:

Best Coding Practices
The code can be optimized to meet the service-level agreement. This solution can solve 20% of the issues highlighted above. However, this does not solve the problem caused due to unknown environmental factors. Human intervention becomes essential to troubleshoot this scenario.

Scheduling ETL Jobs
Jobs need to be well-planned and scheduled according to data and time dependency to avoid job contention. By scheduling the jobs in a planned way, we can ensure the timeliness and accuracy of data, but we cannot foresee the failures and delays. We need intelligence to predict the runtime of a job on a particular day. This prediction would turn the process pro-active rather than reactive.

Solution for all challenges: Robotic Process Automation (RPA)
The Robotic Process Automation solution involves building an artificial intelligence soft robot for automating repetitive work done by humans.

A few ways in which RPA can deal with batch monitoring and data processing challenges:

• 24X7 monitoring of the jobs can be achieved with Soft Robots
• The certainty of a job to fail can be predicted
• Trained models can use the run history of jobs to predict job failures and delays in the schedule
• The concerned teams can be notified in time for the next actions to be taken
• Reduced involvement of multiple teams on repetitive job failures, leading to the reduction of the turnaround time
• Increased cost-effectiveness as the human effort is replaced by Soft Robots

A big shout out to Lakshminarayana Sade and Nisanth Korrapati for sharing industry insights and best practices for optimizing batch monitoring and data processing.

Author: Laveena Ramchandani
Laveena Ramchandani working as a Certified Associate Consultant at Saama Analytics with more than 6 years of experience in ETL, Data Warehousing and RPA

Originally Published on saamaanalytics.com