Automated Medium JavaScript article a day Telegram bot
Introduction:
What is the purpose of creating it?
I was not reading enough articles and personally I have difficulty when it comes to decision making. I am always flooded with many similar tier choices and when there is no clear winner it becomes hard to choose. I know Medium already sends a daily digest every day but first of all it is by email which I get a lot of and also there are few articles to choose from. Probably I should start using it, but the problem I was trying to solve is that I was not reading enough about JavaScript to expand my knowledge and there are so many articles online as well.
So I have decided to build something useful that will help me tackle that problem, also make a cool project to get some practice and discover new frameworks and libraries! I made a Telegram automated bot that can send a SINGLE article to you directly if you subscribe to it via a message.
What is Telegram and why Telegram bot?
Telegram is a cloud-based messenger which can be used for file sharing or having groups up to 200,000 members (good luck with filling up 200,000 members in a single group). But it is easy to use as it has apps on desktop and mobile phones and is a powerful file sharing option up to 2GB each.
I have chosen to use Telegram bot for this project as it is easy to set up a bot as there is Telegram Bot API provided by Telegram developers themselves which makes the setup significantly easier.
First problem I had to solve is how to pull the latest Medium article in the JavaScript topic section. I have never done any website web scraping for particular things on the page. I was suggested to web scrape using Puppeteer which is a Node Library which provides a high-level API control Chrome or Chromium over DevTools Protocol. I was a bit confused in the start as when I started exploring Puppeteer it could open a website link and extract a screenshot or PDF from a website and save it in the project directory. But unfortunately those were not very useful as screenshot is a picture and PDF is not a perfect version of the webpage as it is impossible to read in VSCode to extract/scrape anything from it.
So Puppeteer was out of the equation.
Next step I was playing with Postman GET requests of topic/javascript page of Medium and seeing if there is anything to catch/filter there and extract like a particular CSS className or anything I could grab from the result to get what a single article I wanted. So I continued and stumbled on StackOverflow thread which says that Medium API is write-only and is not intended to retrieve posts. That person recommended to simply use RSS (Really Simple Syndication) feed but the example that person gave was for feed on individual Medium profile feed and not topic page full of articles. Then I found a saving grace link on Medium help section that shows if you put feed after https://medium.com/feed/topic/javascript it will return not a webpage but an XML file which is a significant progress and a great leap forward towards the finishing line. This technique of adding ‘feed’ is similar to adding ‘json’ at the end of a Reddit link and receiving a json object back.
Then all I had to do is make an ‘axios’ GET request to RSS feed link of topic/javascript on Medium and parse it, I used ‘xml2js’ for that and I have a json object.
const xml2js = require("xml2js");const axios = require("axios");async function pullMedium() {const { data: xmlData } = await axios.get("https://medium.com/feed/topic/javascript");return xml2js.parseStringPromise(xmlData);}
So this function will be exported and used later in the project, this function just serves to GET using ‘axios’ the json object of topic/javascript page.
Explain the process of creating telegram bot
Making Telegram bot is super easy and there are multiple tutorials online, all you have to do is search for ‘BotFather’ telegram bot in the search bar in Telegram application which will help you to create one or use the link I attached to ‘BotFather’ (I have done the in-app search method myself).
Then you type ‘/newbot’ to create a new bot and prompt you for a Telegram bot name which must end in ‘bot’ and then you receive a newly generated token to access the HTTP API for a bot. Tadaaam you have a newly created bot and subscribe to it.
Explanation of the integration part of pulling article and integrating it on the bot
Now I have a function that pulls an article from the medium and I need to somehow display it in my Telegram bot. That is where Telegraf is coming into the play which is a Bot Framework for Node.js. It is super easy to use as you need to install it with ‘yarn’ or ‘npm’ and import it into the file and then create a new Telegraf instance with the bot token you have received when you created a bot with ‘BotFather’.
const { Telegraf } = require("telegraf");const bot = new Telegraf(BOT_TOKEN_HERE);
Then you can do various commands, for example: it hears if you type ‘hi’ to it and bot will reply ‘Hello Medium!’
bot.hears('hi',(ctx) => ctx.reply('Hello Medium!'));There are many more commands you can use which you can check out on Telegraf documentation for bot ( https://telegraf.js.org/#/?id=bot ). Now you can communicate and set up the bot!
Each user who is subscribed to a bot has a unique chat id which will help the bot send a message to that user. To obtain chat id you need to search the bot in the search bar in the Telegram application as it was done with ‘BotFather’ and press ‘START’ to obtain a context from which you can extract the chat id. Only after pressing ‘START’ to a bot can you start interacting with a bot. I wrote a command with Telegraf that bot listens to when you write ‘id’ to the bot it will reply with a context chat id.
bot.hears("id", (ctx) => {return ctx.reply(ctx.chat.id);});
Explanation of the automation part
Now at this stage we have an XML json object, a way to send messages via Telegraf to the user subscribed to the bot, all we need now is the automation part. That is where Chron Node.js package comes into play, it is a tool that allows you to execute something on a schedule, which is a perfect tool to use for our use. Now all you have to do is install and import it and set it up to the time schedule you need, in my case every day so the syntax for ‘cron.schedule(0 9 * * *)’ which would send it at 9:00 am every day of the month and week.
It uses unique syntax as each asterisk has a meaning behind it.
For testing purposes I have set it up to send me an article every minute from my local machine running as a server for code to be executed.
Hierarchy in my code
Cron schedule function will wrap function that pulls and parses XML medium page into json object and then the function that pulls article will wrap Telegraf sendMessage function which is used to send message to the user.
cron.schedule("* * * * *", function () {pullMedium().then(function (result) {articleLink = `[TODAY'S ARTICLE ](${result.rss.channel[0].item[0].link[0]})`;(async () => {const users = await User.findAll({ attributes: ["chatId"] });users.forEach((user) =>bot.telegram.sendMessage(user.dataValues.chatId, articleLink, {parse_mode: "markdown", disable_web_page_preview: false }));})();}).catch((err) => console.error(err)); });
Finishing line
I needed to set up a database which in my case was Postgres and interacting with Postgres I used Sequelize which is a promise-based Node.js ORM for Postgres. Then I needed to set up a remote server with a virtual machine on it. I needed a database to store chat ids (as a reminder when you ‘START’ the bot the context is created which has a unique chat id) and pull it from there in the form of an array and loop through the array of chat ids and send a message via Telegraf to each user who is subscribed.
const users = await User.findAll({ attributes: ["chatId"] });users.forEach((user) =>bot.telegram.sendMessage(user.dataValues.chatId, articleLink, {parse_mode: "markdown", disable_web_page_preview: false })
# COOL THING ABOUT TELEGRAM
Some articles have ‘ϟ INSTANT VIEW’ option which is fascinating as it allows users to view articles from around the Web with ZERO loading time! But unfortunately it is not available for desktop application but on the phone it is there! As Instant View pages are extremely lightweight and are cache on the Telegram servers! Can read more on https://instantview.telegram.org/ .
# BACK ON TRACK
Then needed to add some finishing touches like /subscribe and /unsubscribe commands which are also easy to add via ‘botfather’. So to create ‘/subscribe’ and ‘/unsubscribe’ you would write ‘/setcommands’ to the bot and it will ask you in response ‘choose a bot to change the list of commands’ and you select the bot you want to do it for
And after selecting your bot it will ask you to input list of commands which in my case was:
‘subscribe — to subscribe
unsubscribe — to unsubscribe’
Small thing is when you set commands be sure to write them in one message like I did otherwise it just saves the one you set the latest instead, there should be a better way for sure as bot can have multiple commands but it worked for me this way!
And then you have those commands added to your Telegram bot and you can test them out by writing ‘/’ in your bot input text field
After that you need to write commands that bot will listen to and then execute a callback which in my case ‘/subscribe’ would create the instance in the database and catch if it already exists and ‘/unsubscribe’ would destroy/delete the instance with that context chatId.
bot.hears("/subscribe", async (ctx) => {try {const userCreated = await User.create({ chatId: ctx.message.chat.id });return ctx.reply("successfully subscribed! Enjoy the knowledge");} catch (error) {return ctx.reply("you are already subscribed!");
} });bot.hears("/unsubscribe", async (ctx) => {const deletedUser = await User.destroy({where: { chatId: ctx.message.chat.id }, });return ctx.reply("successfully unsubscribed, see you again!"); });
Now that you can subscribe and unsubscribe which will make appropriate changes in the database.
Also needed a remote server that has a virtual machine with Linux on it and all dependencies installed required to run the code. Remote Server have to be running the code constantly as running it all the time on a local machine is inefficient and energy consuming to have a computer running all the time in non-sleep mode.
On the 2nd week of using the remote server the server was maliciously used, but I am not surprised because it was super cheap (2$ per month) and it is a story for another article! So be sure to protect your remote server to prevent unauthorized access and do not ignore network security as a lot of people start taking measures when they attack already happened and not before to prevent that from ever happening!
Also a small tip if you will be testing the code on local server be sure to stop the node process on the remote server as you will get bad output as there will be duplicate messages (delayed promises) and unresponsiveness to commands, just don’t do it trust me! Spent quite a bit of time debugging that issue.
Conclusion/Thesis:
Results of the project
This project was interesting to make as I was diving something that I have never explored before and discovered some new libraries and technologies. Telegram has an easy documentation on how to set up a bot and there are multiple other resources to guide you. Telegram Bot API is set up very nice as well to abstract the user from diving into how to actually create a bot and instead Telegram chose the friendly UI approach with a ‘Admin’ bot like bot that you can write commands related to bot. If you want to see commands on the bot just write ‘/help’ to BotFather! As bot is sending me an article the project is a success!
What could have been done better?
First of all I could work on myself so I could make decisions better and just use daily digest email from Medium, but in all seriousness I think for what I wanted I have done everything. I could have more commands to the bot like subscribe and receive different articles from different topics, so it’s not strictly limited to ‘topic/javascript’.
I could have also added ‘/next’ article command as some articles will not fit your skill set for example ‘5 tips to boost your Angular skills’ when you have never touched Angular or have no interest in using it or this could be the sign that you need Angular might be the next thing you need to learn and love! (doubt)
Also I just extract the first article from web scraped XML file and parsed into json object in the ‘Latest’ section of the page which is not even sorted correctly as there are articles posted later than the first one below.
Could also have a smarter and better way of pulling better articles like the first one in the ‘Popular in JavaScript’ section. There are a lot of directions this project can be improved or tweaked, but I am happy with the result and my little adventure!
Resources/References:
- https://help.medium.com/hc/en-us/articles/214874118-Using-RSS-feeds-of-profiles-and-publications
- https://stackoverflow.com/questions/36097527/how-to-retrieve-medium-stories-for-a-user-from-the-api
- https://telegraf.js.org/#/
- https://core.telegram.org/bots/api
- https://scotch.io/tutorials/nodejs-cron-jobs-by-examples
6. https://www.google.com/ (For research and troubleshooting purposes)
Check out the GitHub repo for project code if interested https://github.com/rharkov/daily_javascript_mediumArticle_bot
If you try to run the code it won’t work as there is ‘secrets.js’ file in .gitignore that has my BOT_TOKEN which is required when setting up connection to the bot via Telegraf.