How to design task management system?

Source

We use a Task management system (TMS) across different domains, and it has several variations. Here, we are designing a TMS with the following features.

• Priority: Tasks have different importance, so we need to treat each task differently.
• Shared: There is a task pool, and the workers pick up them based on their priority.
• Variety: Each task requires different skills or steps to complete.
• Time bounded: Each task has an expected completion time, and we need to make sure we complete maximum tasks on time. If the tasks are approaching their deadline, then task priority will be increased automatically.
• Workers: can be humans or machines, and they complete the task according to steps and priority. We need to maintain their status as they may be unavailable all the time.
• Comments: We can add comments to the task as we work on them.

Main components

Tasks Service

It is an API service that provides different endpoints to do the following actions:

• CRUD operations on tasks
• CRUD operation on task comments
• CRUD operations on workers
• Reports APIs for Task metrics
  - Open tasks
  - Tasks completed on time
  - Tasks not completed on time
  - Tasks that are nearing the deadline etc

Task Queue

We need to handle various different tasks so it is better organize them as separate queue for each type of task.

• Configuration: Task queues holds configuration about tasks like fields, steps, and workflows.
• Permissions: We can assign workers to these queues so that they can only access the tasks from the assigned queues.

Implementation: We can use a Database table to implement a task queue for better flexibility. We can change priority easily and also customize workflows for each type of task, but this also brings the following challenges:

• We need to use locking to assign the unique user to the task, which might sometimes be hard to get right.
• We need to use distributed or NoSQL databases to scale.

Task Manager

This schedules the tasks by continuously polling the task queues and assigns them to the workers based on their priority, skills, and availability. It also updates the task priority if it is approaching the deadline.

Implementation: When a new task arrives, we can immediately check if we can assign it to any existing workers. Also, when a worker is requesting a new task to work on, we can assign it immediately by checking the current queues. Finally, for resigning tasks, we can use cron-based systems like AWS Event Bridge, Java spring scheduler, etc., to monitor and assign the task to a new worker periodically.

Authentication Service

We can use role-based access control (RBAC) to authenticate the users to their tasks and other resources of the system. We can maintain the users and their associated roles in the database. We should store user passwords in third-party services like AWS Cognito, Firebase, or Microsoft AD for security and extensibility.

Task Management System Architecture

Workflows

User Creation

• We call User created API with necessary Roles required for the user and then API calls the third party identity provider(IdP) to create login and password after that we create an entry corresponding to the user in the database.

User Authentication

• The user enters login credentials on the portal. Then the portal authenticates the user against the identity provider(IdP) and then passes the JWT token to the back-end authentication service via API. The service validates the token and provides access to other APIs. We can store the token in the web browser local storage to avoid re-login.
• Two token strategy: Token API generates a new API-token based on the token provided by IdP, and all other APIs only accept the API-token. This design scales better as only token API needs to be changed to support new IdPs in the future. Also, we can embed necessary application context in API-token to reduce DB calls required for each API call.

Task Creation

• We need to select the type of the task, priority, and expected time of completion.
• Back-end API validates the task based on the type and pushes it to the corresponding task queue.

Fetch the next task

• When a worker fetches the next task to work, the task manager polls all the assigned task queues for the next high-priority task and transfers them to the worker.

Working on the task

• The worker follows predefined steps to complete the task, and once each step is completed, they add comments so that other workers can continue on them.

Reassigning the task

• When a worker has gone offline without completing the task and is nearing the deadline, the task manager automatically reassigns it to the new worker.
• The worker can manually reassign the task to some other worker if they need help on the task.

This blog doesn’t cover complete design but can give an idea about how you can approach a Task Management system. Let me know if you have any questions or suggestions. Thanks for spending your time on my blog.