Perpetual Report Generation with Github Actions
One thing I love is laziness. I can work really hard for something else to do my tasks. That something is usually a machine. In this case, I will write about how to prepare an automated daily report and publish it using Github Actions, R, RMarkdown, Python, Docker and Telegram. Our report will be about intraday electricity market prices. Data is taken from EPIAS Transparency Platform.
Why? — U do dis
Good question. Mostly curiosity. Also, these practices are quite useful for real life problems. Reports, dashboards and newsletters are everywhere.
Usually there is a real person behind them. In the worst case they download the data, do analysis in Excel, copy paste findings to a Word document and send via email. This is the second reason: Scalability. If an algorithm can do it, let it do it.
Finally I get to promote startups and technical non-profit organizations! There is a huge area if you want to use it (for free of course).
What? — Outputs
- Outcome is a daily report served at https://berkorbay.github.io/intraday/ and updated each day automatically.
- Codebase can be found at https://github.com/berkorbay/intraday so you can prepare similar settings for your own use (MIT License)
- You can subscribe to the Telegram channel at https://t.me/seffaflik
How? — Process
- I used Python to gather raw data from the platform and sending Telegram messages. I feel more comfortable with Python when dealing with brief commands at Terminal (Bash, zsh etc.). Nothing special here: pandas and requests most of the time.
- I used R and RMarkdown to process the data and churning the resulting report in HTML format. HTML format is better than PDF at this point because of the transient nature of daily reports and ease of accessiblity (i.e. just click on the link and you get the report).
- Personally, the best R project environment is the package environment, so that we can turn it into an proper package later. I use golem to initiate package environments. I mainly used tidyverse packages (dplyr, lubridate, readr), echarts4r and reactable to prepare daily report.
- I used Github Pages to present the report and Github Actions to orchestrate the whole process in a scheduled way (cron job). In order to keep package installation to a minimum, I used a Docker image as the starting container.
- Finally I used a Telegram bot and a public Telegram channel to notify people about the prepared report.
Where? — to stop
Some final remarks.
- Currently report generation takes about 2–3 mins each day and it is not human time. It is a huge time saver. I thank Github for this (ps. I hope what I do is not against Github Actions policy) opportunity.
- There are a few improvements left (e.g. automated scheduling of startup promotion) and then I think this project is complete. Crafty eyes might recognize “inefficiencies” (i.e. mess), but it serves its purpose and that’s enough for now.
- I hope reports like Intraday report proliferate in the future especially for the energy community. Energy markets are data intensive and its players smart and technical people.
Who? — to contact
- Send me a message on Linkedin
