Progress Tracker in Slack

In our workflows, we use Slack WebHooks to send notifications about job failures and other useful information. However, none of these messages are dynamic. Technically we could send multiple Slack messages to update the status of a task, but those messages would flood the channels, deterring people from reading it.

How to create a progress tracker similar to this across the network?

We had an Airflow task that we wanted to be able to estimate how long it would take to complete and set out to visualise a task’s progress in Slack, similar to what we can find in many operating systems.

Static vs Dynamic Slack Messages

To create this dynamic slack message, we needed:

1. Time elapsed measurement
2. Suitable metric for measuring completion rate
3. A method to send dynamic messages to slack

1. Was easy- we could store a timestamp at the beginning of the task, and compare the current timestamp to get the elapsed time.
2. Depends on the task. In our case, we were trying to update a large number of records, and we can measure the number that has been sent based on the number of elements in my payload. So we used the number of records sent compared to the number of expected records to be updated as our metric for completion rate.
3. We found that the Slack WebClient had the ability to override existing slack messages based on their channel ID and timestamp. This meant that in theory, if we stored the timestamp of the first slack message, we could use it to create a progress tracker, by overriding the message over time.

We define a ProgressTracker class and set up its Slack message to look something like this:

Line 1: [environment] Some Title
Line 2: [emoji indicating complete or failure] (progress bar) (completion percentage)
Line 3: Estimated time remaining
Line 4: How long it has been running for

Line 1 is simply passed in as a variable, and Line 4 is based on the time elapsed measured.

Line 2 requires a progress bar, which can be created using slack formatting:

█ can be replaced with chr(9608) if a decimal representation is preferred

The number of █ characters increases based on the percentage complete, creating the bar.

How granular it is depends on the percent_per_chunk variable. In my case, every 4% of completion is represented by a █ — so a maximum of 25 chars for the bar, which comfortably fits into a slack window for most desktop resolutions.

Line 3 requires some math:

We derive this formula based on the fact that when completion is 0, the remaining time is infinite. When completion is 1, the remaining time is 0. For all completion values in between, we assume a linear relationship. For example, when completion is 0.33, the remaining time should be 2x that of the time elapsed.

Sending the message to Slack is done through the WebClient, with the first message’s timestamp being saved in our ProgressTracker.

Sends the first frame of the Progress Bar to Slack and saves the timestamp

Subsequent frames edit the same message in Slack

We can then instantiate a ProgressTracker within Airflow tasks for progress tracking:

Dynamic Slack messages based on completion metric

On top of the general benefits of squeezing more useful information into our screen space, we can use ProgressTracker for live tracking of any metric that an Airflow task can access. In the context of Business Analytics at Foodpanda, this could be workflow-related such as how many records have been added/updated/migrated, or a tool to monitor vendor or rider data in real time.