CRON Scheduler with .Net Core
Schedulers are applications that perform some tasks periodically. One of the terms that most developers are familiar with regarding schedulers is — CRON jobs. It is a very useful application as it can run in the background.
I was recently trying to build a scheduler in ASP.Net core using Quartz.Net which is an open-source job scheduling system. When building a scheduler with .Net it is a common practice to configure the scheduler in the program.cs file. We specify the schedule of a task once. When we need to change the schedule we normally update the schedule and restart the application. However, there are scenarios when we want to store the job schedules in the database and update them from a different application — for example via an admin backend application. In this case, we would not want to manually restart the application every time someone changes the schedule in the database. With Quartz, if we want to update schedules without restarting the application, the scenario becomes a little tricky.
In this article, I will discuss the approaches that I followed to build a scheduler and add the functionality to update the schedules without stopping the scheduler. You can find the complete code on GitHub. Please click the link here to view the source code. I am assuming that everyone reading this article is familiar with .Net Core and schedulers in general.
An Overview of Quartz.Net
First I would like to give a brief overview of the different features of Quartz that we will use and the role that it will play in our application. If you would like to read in details more about these features, I highly recommend reading the documentation of Quartz.Net.
In Quartz, each task that we perform is called a Job. Each job can either run just once during a specific date and time, or it can run periodically at specified intervals. A scheduling system can have multiple jobs that get executed at different times and at different rates. Each job has its name and a group name. Multiple jobs can belong to the same group. To uniquely identify a job, we use a Job key which is a combination of the job’s name and its group name.
Next, we specify a schedule for each job — we specify when and how often the job will get executed. We can also specify a start and an end time. The schedule is generally written as a CRON expression. In Quartz, we call this schedule — trigger. Like a job, a trigger has a name and a group name. We uniquely identify a trigger using a trigger key which is a combination of its name and the name of the group that it belongs to.
A scheduler is the manager of the scheduling system. We assign multiple jobs to the scheduler and attach a schedule to each job. Finally, we start the scheduler. The scheduler decides when a job will get executed as per the schedule and executes it. It keeps a record of each job — when it was last executed, what its schedule is, etc. This is known as Job detail. Each job gets executed as long as the scheduler of the job is running. We can have multiple schedulers in a scheduling system. But for our case, one scheduler will suffice.
A scheduler fetches job from a Job Factory. When the scheduler decides that it is time for a job to execute, it fetches an instance of the job from the factory. The factory creates an instance on-demand and returns it to the scheduler. When the job completes, the factory disposes the instance of the job.
We usually configure the scheduler when the system starts — in the program.cs file. But we know that the code in program.cs executes just once. If we specify the job and triggers in this file, how do we update it when someone changes the schedule of a job in the database?
The solution that I could think of was to have a separate job that runs periodically. When it gets executed, it will fetch the schedule of other jobs from database. It will decide which jobs need to be rescheduled and it will update schedules of those jobs accordingly. Let’s call this job Zookeeper and the jobs that it will manage — Workers.
An Overview of the Source Code
The main classes that we will use in this application are as follows:
- Program.cs: This file is the entrypoint of our application. It is responsible for configuring different services, the scheduler, and starting the scheduler.
- AppSettings.cs: Normally we would store the schedules of our job in the database. But to keep this application simple, I will store the schedules in appsettings.json file. When our application is running, we will update the appsettings.json file to change the schedule and see if the job gets rescheduled properly. This file contains the JSON key of each schedule and fetches the schedule from the JSON file on demand.
- ZookeeperJobFactory.cs: This is our job factory that creates an instance of each job and returns it to the scheduler before a job gets executed.
- ZookeeperJob.cs: This file is the zookeeper. When executed, this object goes through each worker job and fetches its schedule from AppSettings. It verifies whether the job needs to be rescheduled. If it does, it updates the trigger of the job replacing the old schedule with the new one. It then passes the job and its new trigger to the scheduler.
- WorkerJob.cs: This is the generic worker class. All worker jobs extend this class. It contains all information about the worker — Its job key, trigger key, key of the appsettings.json file that contains the job’s cron expression, and the method to initially configure the worker job and attach it to the scheduler. In this way, if we create a new worker job, all we need to do is create the job class, update the appsettings.json to store the cron expression, and add the worker’s class name to the list in ZookeeperJobFactory class.
- InvoiceNotificationJob.cs: This is a sample worker job. It just prints the date and time when the job gets executed. But we can create our worker jobs and add the code to specify the task that the worker will perform.
- BackOrderNotificationJob.cs: This is also a sample worker job like InvoiceNotification.
- SchedulerConfiguration.cs: In this file, we add the code to configure the scheduler for the first time.
When configuring the jobs, we will add dependency injection so that the jobs are singleton classes. When creating the instance, we will fetch an instance from the ServiceProvider so that we get the same instance of the job every time and do not create a new instance of the class every time.
Explanation of the Source Code
Filename: Program.cs
public static IHostBuilder CreateHostBuilder(string[] args)
{
return Host.CreateDefaultBuilder(args)
.UseWindowsService()
.ConfigureServices(async services => {
services.AddOptions();
ServiceRegistration.AddServices(services);
await SchedulerConfiguration.Configure(services);
});
}We configure the application in this method. We create a Host for the application and configure our services and scheduler. We call the method
ServiceRegistration.AddServices(services);to configure our services and the method
await SchedulerConfiguration.Configure(services);to configure and start our scheduler.
Filename: ServiceRegistration.cs
public static void AddServices(this IServiceCollection services)
{
services.AddSingleton<ZookeeperJob>();
services.AddSingleton<InvoiceNotificationJob>();
services.AddSingleton<BackOrderNotificationJob>();
}In this method, we configure our ServiceProvider and specify that our jobs should be created as singleton instances. When we need to get an instance of the job, we make a call to the ServiceProvider so that it returnsthe same instance every time.
Filename: SchedulerConfiguration.cs
public static async Task Configure(IServiceCollection services)
{
try
{
var serviceProvider = services.BuildServiceProvider();
var factory = new StdSchedulerFactory();
var scheduler = await factory.GetScheduler();
scheduler.JobFactory = new ZookeeperJobFactory(serviceProvider, scheduler);
await ConfigureWorkers(scheduler);
await ConfigureZookeeper(scheduler);
// Start Scheduler
Console.WriteLine("Starting Scheduler");
await scheduler.Start();
await Task.Delay(TimeSpan.FromSeconds(1));
} catch(Exception e)
{
Console.WriteLine("Error configuring scheduler. " + e.Message);
}
}This is the method that configures our scheduler when the application starts. First, we create an instance of ‘StdSchedulerFactory’. This is the standard scheduler factory provided by Quartz. It returns an instance of the scheduler. Next, we assign a JobFactory to the scheduler so that whenever the scheduler decides to execute a job, it fetches an instance of the job from the JobFactory. We finally configure the Worker jobs and the Zookeeper job. Then we start the scheduler. We pass the same scheduler to both type of jobs so that all jobs reside within the same scheduler.
static async Task<IScheduler> ConfigureZookeeper(IScheduler zookeeperScheduler)
{
// Configure Scheduler for Zookeeper
var zookeeperTriggerTime = AppSettings.GetValue(AppSettings.ZookeeperTriggerTime);
Console.WriteLine($"Configuring Zookeeper Scheduler with Trigger time: {zookeeperTriggerTime}");
var zookeeperJob = JobBuilder.Create<ZookeeperJob>()
.WithIdentity(ZookeeperJob.JobName, ZookeeperJob.JobGroup)
.Build();
var zookeeperTrigger = TriggerBuilder.Create()
.WithIdentity(ZookeeperJob.TriggerName, ZookeeperJob.TriggerGroup)
.WithCronSchedule(zookeeperTriggerTime)
.Build();
await zookeeperScheduler.ScheduleJob(zookeeperJob, zookeeperTrigger);
return zookeeperScheduler;
}The above method configures the Zookeeper. It first retrieves the CRON expression for the Zookeeper from AppSettings. Next, it creates a job for the Zookeeper and specifies a job name and group name that we specified in ZookeeperJob.cs file. Finally, it creates a trigger for the Zookeeper job and passes the job along with the trigger to the scheduler.
static async Task<IScheduler> ConfigureWorkers(IScheduler scheduler)
{
foreach(var worker in ZookeeperJobFactory.Workers)
{
scheduler = await worker.ConfigureScheduler(scheduler);
}
return scheduler;
}The above method loops through each of the worker jobs that are available and specified in the ZookeeperJobFactory file. It then configures the scheduler to add a job and trigger for the worker.
Filename: WorkerJob.cs
public async Task<IScheduler> ConfigureScheduler(IScheduler scheduler)
{
// Configure Scheduler for BackOrder
var triggerTime = AppSettings.GetValue(GetAppSettingsTriggerKey);
Console.WriteLine($"Configuring {GetAppSettingsTriggerKey} Scheduler with Trigger time: {triggerTime}");
var job = JobBuilder.Create(_jobType)
.WithIdentity(GetJobName, GetJobGroup)
.Build();
var trigger = TriggerBuilder.Create()
.WithIdentity(GetTriggerName, GetTriggerGroup)
.WithCronSchedule(triggerTime)
.Build();
await scheduler.ScheduleJob(job, trigger);
return scheduler;
}Above method contains the code to configure a worer. It fetches the CRON expression for the worker from AppSettings. It creates a job and a trigger for the worker similar to Zookeeper. Finally, it passes the job and trigger instance to the scheduler.
Whenever we create a new worker job, we extend the worker class and pass the required parameters to the parent so that the system can configure those workers during startup and reschedule the jobs when necessary. An example is as follows:
Filename: InvoiceNotificationJob.cs
public class InvoiceNotificationJob : WorkerJob
{
public static string JobName = "InvoiceNotificationJob";
public static string JobGroup = "InvoiceNotificationJobGroup";
public static string TriggerName = "InvoiceNotificationTrigger";
public static string TriggerGroup = "InvoiceNotificationTriggerGroup";
public InvoiceNotificationJob() : base(
JobName, JobGroup, TriggerName, TriggerGroup,
AppSettings.InvoiceNotificationTriggerTime, typeof(InvoiceNotificationJob)
)
{
}
public override IJob GetJob(IServiceProvider serviceProvider)
{
return serviceProvider.GetService<InvoiceNotificationJob>();
}
}When the worker gets executed, the scheduler calls ‘Execute’ method of the worker. So, all our code related to each worker will start from this method.
As discussed earlier, ZookeeperJobFactory is the class that returns an instance of the job that is about to be executed.
Filename: ZookeeperJobFactory.cs
public class ZookeeperJobFactory : IJobFactory
{
public static readonly List<WorkerJob> Workers = new List<WorkerJob>()
{
new InvoiceNotificationJob(),
new BackOrderNotificationJob()
};
readonly IServiceProvider _serviceProvider;
readonly IScheduler _workerScheduler;
public ZookeeperJobFactory(IServiceProvider serviceProvider, IScheduler workerScheduler)
{
Console.WriteLine("Creating new Zookeeper Job Factory");
_serviceProvider = serviceProvider;
_workerScheduler = workerScheduler;
}
// ...
}When creating an instance of this JobFactory, we pass the ServiceProvider and the Scheduler object. It uses the ServiceProvider to fetch an instance of the Job and it passes the Scheduler object to the Zookeeper Job so that it can reschedule worker jobs.
It also contains a list of all Workers available in the application. After we create a WorkerJob, we should add an instance of the job to this list so that our application knows about the Job.
Filename: ZookeeperJobFactory.cs
public class ZookeeperJobFactory : IJobFactory
{
// ...
public IJob NewJob(TriggerFiredBundle bundle, IScheduler scheduler)
{
//Console.WriteLine($"Creating new job for {bundle.JobDetail.Key.Name}");
IJob job = null;
if(bundle.JobDetail.Key.Name == ZookeeperJob.JobName)
{
var zookeeperJob = _serviceProvider.GetService<ZookeeperJob>();
zookeeperJob.AddWorkers(_workerScheduler, Workers);
job = zookeeperJob;
} else
{
foreach(var worker in Workers)
{
if(bundle.JobDetail.Key.Name == worker.GetJobName)
{
job = worker.GetJob(_serviceProvider);
}
}
}
return job;
}
// ...
}The above method is executed to fetch an instance of the job before the scheduler runs a job. Here, the ‘bundle’ object contains all information about the job that the scheduler intends to execute. We fetch the name of the job from this object and return the appropriate job.
If the scheduler asks for the Zookeeper object to execute the job of Zookeeper, we fetch an instance of the object from ServiceProvider. Next, we pass the scheduler and list of workers available to the zookeeper instance so that it can reschedule required jobs.
On the other hand, if the scheduler asks for a worker object, we loop through each worker and return an instance of the worker to the scheduler.
Filename: InvoiceNotificationJob.cs
public class InvoiceNotificationJob : WorkerJob
{
// ...
public override IJob GetJob(IServiceProvider serviceProvider)
{
return serviceProvider.GetService<InvoiceNotificationJob>();
}
}The above method takes a ServiceProvider and returns an instance of itself by fetching it from the provider. We already have an instance of the job in our worker list and we could have returned that. But I implemented this method to show that we can use the serviceprovider as well.
Rescheduling Jobs
In this section we will discuss the main part of the application — rescheduling jobs without needing to restart the application.
Filename: ZookeeperJob.cs
public class ZookeeperJob : IJob
{
// ...
async Task rescheduleJob()
{
//Console.WriteLine("Rescheduling worker jobs");
if (_workerScheduler != null)
{
foreach(var worker in _workers)
{
var job = await _workerScheduler.GetJobDetail(worker.GetJobKey);
//Console.WriteLine($"--Rescheduling worker {job.Key}");
var trigger = await _workerScheduler.GetTrigger(worker.GetTriggerKey);
if(trigger != null)
{
var cronTrigger = (ICronTrigger)trigger;
var newExpression = AppSettings.GetValue(worker.GetAppSettingsTriggerKey);
if (cronTrigger.CronExpressionString != newExpression)
{
Console.WriteLine($"--Rescheduling job for trigger {trigger.Key.Name}");
var newTrigger = getNewTrigger(trigger.Key.Name, trigger.Key.Group, newExpression);
await _workerScheduler.RescheduleJob(trigger.Key, newTrigger);
}
}
}
}
else Console.WriteLine("Could not reschedule worker as the scheduler is null");
}
public Task Execute(IJobExecutionContext context)
{
try
{
// ...
var task = Task.Run(async () => await rescheduleJob());
task.Wait();
if (_workerScheduler != null && !_workerScheduler.IsStarted)
{
Console.WriteLine("Starting worker scheduler as it has not been started yet");
_workerScheduler.Start();
}
return Task.CompletedTask;
} catch(Exception e)
{
Console.WriteLine("Received an error when executing Zookeeper job. " + e.Message);
return Task.FromException(e);
}
}
// ...
}Whenever the Zookeeper Job is executed, the ‘Execute’ method is invoked. Since this method is synchronous, we create a task to execute our asynchronous method ‘rescheduleJob’ that reschedules our job.
In ‘rescheduleJob’ method, we loop through each worker and retrieve their job key to fetch the corresponding job detail from the scheduler. Next, we fetch the trigger of the corresponding job using the trigger key of the worker.
var job = await _workerScheduler.GetJobDetail(worker.GetJobKey);
//Console.WriteLine($"--Rescheduling worker {job.Key}");
var trigger = await _workerScheduler.GetTrigger(worker.GetTriggerKey);Next, we fetch the current CRON expression of the worker from the trigger and then the CRON expression of the worker available in AppSettings by passing the appsettings.json key for the worker stored inside the worker object.
var cronTrigger = (ICronTrigger)trigger;
var newExpression = AppSettings.GetValue(worker.GetAppSettingsTriggerKey);Next, we compare both the expressions and if they do not match, we create a new trigger object with the same trigger key and the new CRON expression. We then pass the job detail and the new trigger to the scheduler. The scheduler replaces the old trigger with the new one as they both have the same key.
if (cronTrigger.CronExpressionString != newExpression)
{
Console.WriteLine($"--Rescheduling job for trigger {trigger.Key.Name}");
var newTrigger = getNewTrigger(trigger.Key.Name, trigger.Key.Group, newExpression);
await _workerScheduler.RescheduleJob(trigger.Key, newTrigger);
}That’s it! Our worker jobs get rescheduled without needing an application restart. In this application, I fetch the CRON expressions from AppSettings but you can replace the code to fetch the data from a database instead. You can then create an admin panel application to manage the CRON expressions of worker jobs. You will then get a scheduling system that dynamically reschedules jobs.
Running the application
If you would like to run the application, you can clone the source code. It uses .Net Core 3.1. Run it using your Visual Studio. While the application is running you can see the console output from each worker job. It specifies the datetime when the worker is executed.
You can see output of each worker job in the console. As per the CRON expression, BackorderNotification is being executed every 3 seconds and InvoiceNotification is being executed every 5 seconds.
If you would like to update the CRON expression while the application is running, navigate to the base application folder and then to CronService/bin/Debug/netcoreapp3.1
Open the appsettings.json file highlighted above and edit the CRON expression of a worker job. Save the file. You will see that next time, the worker job gets executed according to the updated CRON expression.
Conclusion
In this tutorial we had a brief discussion about how scheduling system and more specifically Quartz.Net works. We then went through a demo application to see how we can build a scheduling system that has the ability to reschedule jobs on the fly without needing a restart. I hope you enjoyed reading this article and find it helpful. I would love to hear your feedback and your ideas of how you would implement this feature in a different way. Bye!
