Optimizing Backend Systems: How I Employed a CRON Job in a Nest.js Microservice
Streamlining API Usage: Automating a call through a CRON, potentially saving millions of API calls to the monolith at scale.
Table of Contents:
Background
If you’re only interested in the API call automation/CRON job, you can skip ahead to the implementation.
Let’s start by discussing why I developed a microservice running a CRON job that calls an API at a set interval.
The backend described here was designed for a truck management application catering to businesses. Each business had multiple drivers, and it was necessary to track their locations. Consequently, our backend facilitated the process by allowing the truck drivers (clients) to update their locations at regular intervals. This enabled businesses to monitor the locations of their trucks, as shown in the article cover.
In this article, the term “clients” refers specifically to truck drivers. However, it should be noted that our application may have other types of clients, including businesses and administrators.
Figure 1 showcases the initial state of the basic backend during the early stages of development, characterized by the absence of user activity. Notably, the depicted backend exhibits a straightforward system that lacks notable elements or complexities.
Problem
As illustrated in Figure 2, an anticipated increase in the number of clients highlights the need for our backend to efficiently handle a projected load of approximately 10,000 Daily Active Users.
As the client base expands significantly, the same backend becomes subject to multiple queries. This challenge can be effectively addressed through the use of horizontal scaling and a load balancer, as illustrated in Figure 3.
However, what I inadvertently overlooked is the mounting intricacy of the Location Updating API.
Consider Figure 4 which illustrates an array of locations consisting of 1440 data points for each client. This count corresponds to the maximum number of locations within a 5-day timeframe. The limitation stems from the constraint that locations can only be spaced 5 minutes apart, resulting in 12 locations per hour and 288 locations per day. Thus, accumulating 288 locations daily leads to a total of 1440 locations over 5 days. My goal was to eliminate outdated locations whenever a new location was added within these 5 days. However, it’s important to note that this requirement may be subject to change in future revisions.
When it comes to adding a new location while preserving the order, there are two options to consider. The first involves removing the element at index 0 using the JavaScript array.shift() function, followed by adding the new location at the end of the array using array.push() as shown in Figure 5. This approach entails a time complexity of O(n) + O(1). Alternatively, one can opt for another method where the last element is removed using array.pop() and the new location is added at the beginning using array.unshift(). Interestingly, this alternative approach also results in the same time complexity of O(1) + O(n).
As development progressed, it became increasingly impractical to handle multiple requests every 5 minutes due to the growing demand for extensive calculations such as time-related calculations, data and coordinates verifications before pushing the location data into the database.
Concerns arose regarding the reliability of clients as they could make excessive API calls. For instance, they might make 5 API calls within 5 minutes, despite only requiring a single call within that timeframe. Additionally, clients faced limitations when in network dead zones, rendering them unable to make API calls for extended periods. Furthermore, as the user base grew, the system’s data volume expanded exponentially, resulting in a corresponding increase in API calls from each client. With just 1000 clients, if each client made one API call per minute, it would amount to 1000 API calls per minute and 1.4 million API calls per day, highlighting the magnitude of the issue.
Solution
To address the aforementioned issues, I developed two distinct microservices that effectively separate the responsibilities of location verification and database integration. This architectural enhancement is illustrated in Figure 6.
In the implementation, I will primarily focus on the automation microservice, which is responsible for automating the API calls to the monolith for pushing location data into the database.
Within this architecture, I developed a separate microservice responsible for automating API calls to the monolith and efficiently updating location data for each truck. By scheduling a CRON job every five minutes, the microservice performs a bulk update by sending an object containing arrays of locations in the request body. This approach optimizes efficiency and scalability by reducing API calls to a single instance every five minutes. Simultaneously, it minimizes database queries while ensuring an even distribution of locations while eliminating extra reads on the main database.
Moreover, this microservice also functions as a listener for new API calls and provides analytical insights as a response.
Implementation:
To automate an API call in your Nest.js microservice, follow these implementation steps:
Creating a Nest.js project.
To install the Nest.js CLI package globally, execute the command
npm i -g @nestjs/cliThen, create the project by running:
nest new project-nameAlternatively, you can create the project in the current directory with:
nest new .To create a microservice, install the official Nest.js Microservices package by running:
npm i --save @nestjs/microservicesFor scheduling jobs or CRON jobs, install the official Nest.js scheduling package by executing the following commands:
npm install --save @nestjs/schedule
npm install --save-dev @types/cronTo make API calls include the HTTP package by installing it with:
npm i --save @nestjs/axios axiosFigure 7 shows what our file structure should look like.
Paste the following code in the main.ts:
import { NestFactory } from '@nestjs/core'
import { AppModule } from './app.module'
import { Transport } from '@nestjs/microservices'
async function bootstrap() {
const app = await NestFactory.create(AppModule)
app.connectMicroservice({
transport: Transport.TCP,
options: {
port: 8081
}
})
await app.startAllMicroservices()
await app.listen(8080)
}
bootstrap()In main.ts we are simply creating a microservice and a HTTP server that is listening to API calls on port 8080 while our microservice runs on port 8081.
Or if you don’t want to set up an HTTP server you can use the code in Figure 8.
In our app.module.ts we would paste the following code:
import { Module } from '@nestjs/common'
import { LocationService } from './location/location.service'
import { LocationModule } from './location/location.module'
import { SchedulerModule } from './scheduler/scheduler.module'
@Module({
imports: [SchedulerModule, LocationModule],
providers: []
})
export class AppModule {}Optional: Setting up another module
Figure 9 shows the code in the location.module.ts. You can use the location module as a guide to create and use your own modules.
Figure 10 shows what the location controller looks like. It handles incoming Get requests and handles location_updated events.
Figure 11 shows the location service which performs various operations. For instance, you have the freedom to create dynamic cron jobs based on specific events, enabling versatile customization possibilities.
Scheduling and Automating the API
The main meat of the article
It is advisable to follow best practices by creating a dedicated scheduler module. This approach allows Nest.js to generate scheduled jobs without any unnecessary duplication of jobs. Our separate scheduler module is shown in Figure 7.
Now to set up our scheduler module we need to paste the following code in scheduler.module.ts.
import { Module } from '@nestjs/common'
import { ScheduleModule } from '@nestjs/schedule'
import { SchedulerService } from './scheduler.service'
import { HttpModule } from '@nestjs/axios'
import { LocationService } from 'src/location/location.service'
@Module({
imports: [ScheduleModule.forRoot(), HttpModule],
providers: [SchedulerService, LocationService]
})
export class SchedulerModule {}In this module, we import the HTTP module, enabling API calls to our external monolith, as well as the scheduler module responsible for scheduling the API. If necessary, you can substitute the “LocationService” in the providers with your own service to perform additional operations tailored to your specific requirements.
Now finally paste the following code into your scheduler.service.ts. Make sure to replace the link and apiData in this.httpService.patch. Also, replace the location service with your service or remove it if you don’t have your own service.
import { HttpService } from '@nestjs/axios'
import { Injectable } from '@nestjs/common'
import { Cron, CronExpression } from '@nestjs/schedule'
import { catchError, firstValueFrom } from 'rxjs'
import { externalLinks } from 'src/links'
import { LocationService } from 'src/location/location.service'
@Injectable()
export class SchedulerService {
constructor(
// Replace the LocationService with your own service
// private readonly locationService: LocationService,
private readonly httpService: HttpService
) {}
// This Job is called every 5th minute.
@Cron(CronExpression.EVERY_5_MINUTES)
async locationBulkUpdateJob() {
// Replace the line below with any function you want to execute
// const apiData = this.locationService.fetchData()
const apiData = {}
// Replace the link
// all http methods can be used here like .post, .put, ect
// httpService.get || httpService.delete ect
const response = await firstValueFrom(
this.httpService.patch(externalLinks.backendProdLink, apiData).pipe(
catchError((error: unknown) => {
throw error
})
)
)
// Replace the line below with any function you want to execute
// this.locationService.clearCache(response)
console.log(apiData)
}
}The scheduler service utilizes various packages to facilitate its functionality. Here’s a brief explanation of each package:
- rxjs: RxJS is a library that enables reactive programming using observables. It provides powerful operators and utilities for handling asynchronous operations, managing streams of data, and handling errors.
- @nestjs/schedule: This package is used to schedule Cron Jobs within Nest.js. It includes CronExpression, which offers a range of predefined cron expressions. For example, the expression “*/5 * * * *” represents running the job every five minutes.
- @nestjs/axios: This package allows for the utilization of the Axios HTTP client within Nest.js. Axios simplifies making HTTP requests and handling responses conveniently and efficiently.
The provided code demonstrates an example usage of these packages within the SchedulerService module. The locationBulkUpdateJob() method is decorated with @Cron, specifying a CronExpression to run the job every 5 seconds. Inside the method, the LocationService is used to fetch data, and then an HTTP PATCH request is made using the HttpService from @nestjs/axios. The response is handled, and the LocationService clears the cache. Lastly, the fetched data is logged to the console.
Summary
- This article explores automating API calls in a Nest.js microservice.
- It discusses the challenges posed by a growing client base and a complex Location Updating API.
- The solution involves implementing a separate microservice for automating API calls to the monolith.
- By scheduling a CRON job every five minutes, the microservice performs bulk updates, reducing the overall number of API calls.
- This approach improves scalability, and performance, and addresses issues like excessive API calls and data volume growth.
- The article provides step-by-step instructions for implementing automation in a Nest.js microservice.
That’s All folks!
Thank you for taking the time to read this article. If you found it helpful, don’t hesitate to show your appreciation by clapping. Feel free to leave any questions or comments, and I will be glad to assist you. Stay tuned for future articles, including deployment instructions for this microservice project.
