Building Scheduled Report through WhatsApp using Python

In the spirit of exterminating coronavirus disease efforts across the globes, I want to share how we can get insights in seconds about what happen to surround us in Indonesia, about the updates of this pandemic, and how dangerous it is.

data pipeline

SPOILER ALERT !!!

Some of you maybe already understood or you had done this before, even with more powerful features, like example chatbot, two ways messaging service, or any else.

output preview

☝️ that’s what I felt after I built this through some scripting in python. I felt wow, this is so amazing, not so difficult but quite challenging and after you understood, you can relate to a lot of things.

In this session, I choose to relate with very warm COVID news, how this virus can change millions of people’s lives to become new normal.

LIFE is gonna change, embrace it or leave it

Nevertheless, we should keep going and it would be so meaningful if you can take advantage even after this new normal, like what I do now, I have more spare time to read or learn various sources and try to implement with my creativity. Maybe previously I don’t have time to do this because you’re too busy with office life (read: waste your time in commuting, chit chat). In short, THANK GOD FOR THIS WFH DAYS!

HOW TO CREATE and DEPLOY a SCHEDULED REPORT through WHATSAPP

1. Use Twilio API to access the backend of WhatsApp API which enable you to send messages programmatically through WhatsApp
2. Create a python script to access Twilio API using specific credentials and also create API requests to the content
3. Use cron scheduler (Advance Python Scheduler) to schedule sending a message through WhatsApp
4. Build and deploy an app in Heroku, (free) cloud platform

WHAT IS TWILIO ???

Twilio is a communication API built on top of some famous chat and messenger API like WhatsApp, Telegram, text message, and so on.

To use (free version) Twilio, you must sign up and login first to their website: https://www.twilio.com/try-twilio

I already have an account so I will directly login to the site.

1. You will see your dashboard, you can go to the right section, click on 3 dots button and go to Programmable SMS
2. Click WhatsApp (Beta)
3. You will see how to connect specific WhatsApp number with server WhatsApp number unless to do so, your WhatsApp number can’t be detected in server-side

THIS IS THE ONLY WAY !
in this case, the “server whatsapp” is +1 415 523 8886 , you need to message join spoken-dog to that number from any whatsapp, then your number will be in the list which server can reach.

4. If you had sent the message, wait for a couple of seconds, it will show Message Received! (this means your WhatsApp number has been recorded in server)

5. Then you can proceed to step Send one way WhatsApp message , which you test your previous established connection between server WhatsApp number and yours, this can be done by choosing 1 of 3 opts there, then click Make Request , your number will receive the content of the BODY stated in there

6. ALRIGHT, Twilio part is DONE

CODING THE PYTHON SCRIPT …

This is the main heart of this job, by this script, it should be able to access and download the content from public websites automatically, then access Twilio API to send messages to any number.

In my case, you will need an IDE stack to track your changes and the code itself, I use VScode (it’s free, easy, well explained and integrated with git also terminal itself, I recommend you this tools if you just start to code)

VS Code preview

1. Create script.py
2. Import all needed library

from twilio.rest import Client
import pandas as pd
import requests
from datetime import datetime

2. Connect to Twilio API, you can do this by copying the Request code, choose Python

account_sid = <your_ACCOUNT_SID> #replace this with yours
auth_token = <your_AUTH_TOKEN> #replace this with yours
client = Client(account_sid, auth_token) 
 
def send_message(receiver, message):
    message = client.messages.create( 
                                from_='whatsapp:+14155238886',  
                                body=message,      
                                to=f'whatsapp:{receiver}' 
                            )
    return message

3. I will present about Top 5 Province with Highest Growth COVID Updates, grab the data from URL (all I need is JSON formatted output)

url = "https://services5.arcgis.com/VS6HdKS0VfIhv8Ct/arcgis/rest/services/COVID19_Indonesia_per_Provinsi/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json"
data_json = requests.get(url).json()
df = []
for row in range(len(data_json["features"])):
    df.append(data_json["features"][row]["attributes"])
df = pd.DataFrame(df)
df = df.drop(['FID','Kode_Provi'], axis=1)
data = df.sort_values(['Kasus_Meni'], ascending=False)[:5]

dataframe output

4. Converting from cleaned dataframe to tidy message

province = data["Provinsi"].tolist()
current_timestamp = str(datetime.now())
messages = f"THIS is UPDATED REPORT up to {current_timestamp}"
for prov in province:
    each_row = data[data["Provinsi"] == prov]
message_partition = f"""
[{prov}]
    Kasus Positif = {str(each_row['Kasus_Posi'].tolist()[0])}
    Meningkat sebanyak {str(each_row['Kasus_Meni'].tolist()[0])}
    Total pasien sembuh sebanyak {str(each_row['Kasus_Semb'].tolist()[0])}
    """
messages = messages + message_partition

preview

BUILDING THE SCHEDULER

I will use Advance Python Scheduler on this session, it simply cron job inside python package, you can read and learn through this URL

1. Create scheduler.py
2. Import needed libraries and script.py

from apscheduler.schedulers.blocking import BlockingScheduler
from script import *

3. Create the scheduler and function you want to run

scheduler = BlockingScheduler()
def the_funct():
    for numb in receiver_list:
        print(f"sending message to {numb}")
        send_message(numb, messages)

4. Add scheduler job and start it (be mindful that you want to run this in the cloud platform, which the time is mostly UTC, so please set your cron time job to -7Hrs before your expected time)

scheduler.add_job(the_funct, 'cron', hour=10)
scheduler.start()

AUTOMATE BY DEPLOYING IT

After all, you have done previously, the job can only run in your local machine, which will need your laptop to turn on every time, it’s not effective at all. So, this time, we will automate the scheduler and its function by deploying it in an online server, like Heroku.

Heroku is one of the famous cloud platforms that you can deploy any apps you built-in there, there’s a free and paid version as well, but I will just use free version so far for this tiny use.

1. You will need to create account and login through Heroku, just straight googling it
2. Create requirements.txt contains all your used packages and its related version

twilio==6.41.0
pandas==0.25.3
requests==2.22.0
apscheduler==3.6.3

3. Create Procfile (NO EXTENSION !!!), scheduler.py is my script name for the scheduler, you can change it to your own

clock: python scheduler.py

4. Just go through the step Heroku CLIto deploy in Heroku using CLI (copy and paste commands in your terminal)

5. After you are done pushing repo to Heroku git, Heroku will automatically compile your code and build the app

6. VOILA, it’s done, your app is already built-in Heroku and just wait for your WhatsApp message in specific time you set previously

CONCLUSION

I’m happy with this quick and hacky solution although I know some of the other solutions that maybe can do similar things (could be from Airflow, composer, cloud function, cloud scheduler in GCP environment), but all those kinds of stuff are quite complicated and less worthed I think if we use for this simple task (sometimes you just need to do a simple task instead of very complicated pipelines right?).

There are still a lot of room for improvements, I just have an idea how to create a chatbot, two ways messaging, so if users want to know COVID monitoring from a specific region, it will automatically reply with the updated report for that region, it will need further data cleansing and upgraded skills for sure. So, every time users send a message, the server is computing and send back the answer.

moreover YOUR TINY PART of KNOWLEDGES is WORTHED to SHARE, so LETS SHARE to the WORLD and CREATE IMPACTS

For full codes, you can view in this Github (https://github.com/karlchris/covidmonitoring)

REFERENCES:

• TWILIO (https://www.twilio.com/console)
• Government COVID Data (https://bnpb-inacovid19.hub.arcgis.com/datasets/data-harian-kasus-per-provinsi-covid-19-indonesia)
• Heroku platform (https://dashboard.heroku.com/)
• AP Scheduler (https://medium.com/better-programming/introduction-to-apscheduler-86337f3bb4a6)