Customise your own Twitter notification with Telegram Bot

Linus Ng
Linus Ng
May 27, 2020 · 5 min read

Telegram has a lot of potential as a messaging app. The Bot APIs can act as assistants to perform many functions in chats and channels. At the same time, Twitter provides easy-to-use APIs for developers to publish, manage, and analyse tweets. With the flexibility the APIs offer, the potential for Twitter and Telegram in data science is endless.
During the coronavirus, I wanted to keep myself up to date with the daily coronavirus figures. In the UK, the figures are released on a daily basis but at a different time throughout the day.

1. The official daily tweet on coronavirus in UK

What I think would be nice, is to get a notification when the figures are released, together with the data that I am interested in.

2. The automatic telegram notification

In this tutorial, I will walk through on how to create your own Twitter news alert. This will include
1. Scrape tweets using Tweepy

2. Extract numeric data from tweets

3. Create a telegram bot, and have the bot sending message to channel/chat using Python

4. Create a crontab job that enables the script recurringly

1. Scrape tweets using tweepy

  1. Login to twitter developer section.
  2. Click ‘Apps’, then ‘Create an app’.
  3. Complete the application
  4. You can see the app that you just created, and the consumer API key and consumer secret key.
  5. At ‘Create my access token’, you can get the access token and the access token secret.

You will need the script below to mine the tweets, save the auth keys as a dictionary.

import tweepy
import datetime
auth = {'consumer_key': 'XXXXXX',
'consumer_secret':'XXXXXX',
'access_token_key':'XXXXXX',
'access_token_secret': 'XXXXXX'
}
class TweetMiner(object):result_limit = 20
data = []
api = False
def __init__(self, keys_dict=auth, api=api, result_limit = 20):

self.twitter_keys = keys_dict

auth = tweepy.OAuthHandler(keys_dict['consumer_key'], keys_dict['consumer_secret'])
auth.set_access_token(keys_dict['access_token_key'], keys_dict['access_token_secret'])

self.api = tweepy.API(auth)
self.twitter_keys = keys_dict

self.result_limit = result_limit
def mine_user_tweets(self, user="linusnhh",
mine_rewteets=False,
max_pages=5):
data = []
last_tweet_id = False
page = 1

while page <= max_pages:
if last_tweet_id:
statuses = self.api.user_timeline(screen_name=user,
count=self.result_limit,
max_id=last_tweet_id - 1,
tweet_mode = 'extended',
include_retweets=True
)
else:
statuses = self.api.user_timeline(screen_name=user,
count=self.result_limit,
tweet_mode = 'extended',
include_retweets=True)

for item in statuses:
mined = {
'tweet_id': item.id,
'name': item.user.name,
'screen_name': item.user.screen_name,
'retweet_count': item.retweet_count,
'text': item.full_text,
'mined_at': datetime.datetime.now(),
'created_at': item.created_at,
'favourite_count': item.favorite_count,
'hashtags': item.entities['hashtags'],
'status_count': item.user.statuses_count,
'location': item.place,
'source_device': item.source
}

try:
mined['retweet_text'] = item.retweeted_status.full_text
except:
mined['retweet_text'] = 'None'
try:
mined['quote_text'] = item.quoted_status.full_text
mined['quote_screen_name'] = status.quoted_status.user.screen_name
except:
mined['quote_text'] = 'None'
mined['quote_screen_name'] = 'None'

last_tweet_id = item.id
data.append(mined)

page += 1

return data

With the above TweetMiner function, we can start collecting tweets with just two lines of codes.

Tweet = TweetMiner(result_limit = 10)
uk_tweets = Tweet.mine_user_tweets(user=’DHSCgovuk’, max_pages=1)

2. Extract numeric data from tweets

tweets_df = pd.DataFrame(uk_tweets)

The dataframe will contain all the tweets that we scrape, but we are only interested in the tweets that includes the daily figures. So we need to do some filtering. I chose the word ‘died’ as a unique string to help collect all the tweets.

tweets_df = tweets_df[tweets_df['text'].str.contains("died")]

To better visualise data, I have created a date column and changed the dataframe to numeric.

tweets_df['date'] = tweets_df.created_at.dt.strftime('%Y-%m-%d')
tweets_df = tweets_df[['screen_name', 'date','text']].reset_index(drop=True)
tweets_df = tweets_df.apply(pd.to_numeric, errors='ignore')

From the tweet, there is a lot of figures we can extract. Let’s try to get the death rate as an example. The way that I calculate the death rate is (cumulative death) / (the number of positive cases).

You can find a lot of regex pattern online. I used the following pattern to match all the number within the tweet to extract the figures as two new columns, ‘death total’ and ‘positive total’.

number_regex = '\d+(?:,\d+)*'
tweets_df['death_total'] = tweets_df.text.str.findall(number_regex).str[8]
tweets_df['positive_total'] = tweets_df.text.str.findall(number_regex).str[5]

We can now create a column called ‘death_rate’ by putting the figures into the formula. We can use the similar ways to extract other figures. But for now, let’s add data transparency by extracting the url with another regex.

url_regex = '(?:(?:https?|ftp):\/\/)?[\w/\-?=%.]+\.[\w/\-?=%.]+'
tweets_df['url'] = tweets_df.text.str.findall(url_regex).apply(''.join)
Your dataframe shouold look like this.

To draft up the telegram message, we need to extract the numbers from the dataframe.

death_total = format(tweets_df.death_total.iloc[0], ',')
death_rate = round(tweets_df.death_rate.iloc[0]*100, 1)
url = tweets_df['url'].iloc[0]

With the variables, we can start putting them into a template.

msg = 'On {0}, the death toll has reached {1}, the death rate thus far is {2}. {3}'.format(latest_date, \
death_total, death_rate, url)

3. Send telegram message via Python

  1. On Telegram, search for ‘BotFather’ and send a ‘/start’ message.
  2. To create new bot, Send another ‘/newbot’ message, then follow the instructions to configure a name and a username. Your user name will be the bot_chatID.
  3. send ‘/token’ to get the token that access to HTTP API and then save it as bot_token.

What we are doing here is to use bot to send message to a target channel. So the bot_chatID should be the channel name you create, starting with ‘@’. Remember to make sure you bot is the admin of the channel. Otherwise, the bot allowed to publish content to the channel.

import requests def telegram_bot_sendtext(bot_token, bot_chatID, bot_message):
send_text = 'https://api.telegram.org/bot' + bot_token + '/sendMessage?chat_id=' + bot_chatID + '&parse_mode=Markdown&text=' + bot_message
response = requests.get(send_text)

Run the function, and voila! You will see the message is being sent to telegram!

4. Running the script recurringly

00 21 * * * /path/to/script/.sh

For example, the above crontab is set to automatically run the script every day at 9pm.

Alternatively, you can also run the script every five minutes and send out a message once the daily figures are released. That would need another tutorial. But you can check out how I do it on my github.

That’s it — now you have written a python script that scrapes tweets, analyses data, and automatically sent message to a telegram channel.

The Startup

Get smarter at building your thing. Join The Startup’s +788K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Linus Ng

Written by

Linus Ng

Data science | NLP | News Automation Currently: Bloomberg LP

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +788K followers.

Linus Ng

Written by

Linus Ng

Data science | NLP | News Automation Currently: Bloomberg LP

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +788K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store