Bus Evide — Ultimate KSRTC Bus Timings App — Sreekant Shenoy
Doesn’t this look amazing? Yeah! I know.
I work on my app equally looking into the material looks of any app or websites I build. I like it clean!
This app is called ‘Bus Evide’, which translates to “Where is my Bus?”. This app is a solution to getting timely Kerala KSRTC bus data, Bus schedules, routes & all the bus details.
This was a fun project for me, which took around 5 days to complete. Every day I worked on this app for 14 hours, just to keep myself engaged in the vacation time and learn as much as possible.
This app idea and everything is completely built from scratch. Officially I have no bus timings data to begin working on this. I had to figure out a way.
Technologies Used:
- NodeJS (Backend Server) — Express, Cors, EJS, Javascript
- Postgres (PSQL) (Database)
- Java, XML, Volley, Material View 2.0 (Android app)
- Python (Web scraping) — BeautifulSoup, Requests, Postgres
This divided this journey into chapters to let you understand my process of working with these type of works.
P.S. If you love reading my articles, do bookmark my website for more! ?
Chapter 1: Finding the KSRTC Bus Timings Data
Of course, getting the data is the main thing of our ultimate goal. Once you get the bus timings data, you are ready to begin doing anything you like app or website!
To get the data, I had to do research works on the availability of the datasets, whether any public/private API’s are provided officially by the KSRTC website for developers.
To my bad luck, there wasn’t any! That’s the sad part when you try to improve & and help the govt. oriented services. I hadn’t stopped there. ?
I looked across the web again and came across ‘ Aanavandi ‘ website. YES! A dedicated website/blog for KSRTC buses timings run by the general KSRTC bus enthusiasts. They had the entire bus data, route & schedules after requesting the govt. continuously for many time.
I appreciate the Aanavandi team for their continual efforts put into for helping the users using their website. Their effort could not be justified with words, because their team took more than 6 months to enter the bus timings data and to come up with a website & app along with that, Hatsoff!’
Chapter 2: WebScraping & Building the Database
I am awesome at web scraping, I love doing it.
This is that time I got really excited and wrote a code just of python with the beautiful bs4 module.
I had to then arrange for a database that could hold a huge amount of data and be able to process quicker. I started liking Postgres especially that its an enterprise database and works best from the command line.
I found a page on the website that would help me fixate the web scrapping entirely. Opened up my Sublime Text and wrote the code for the same.
You might feel it would have taken 10’s and 100’s of hours to process this. I would say, it took me just 2 hours. When you write an easy web scraping code, it works by the looping which is asynchronous. That’s dead slow & not useful for taking tons of data off the web.
NEVER DO THAT!
Introduce to the concept of multiprocessing or multi-threading (both have different internal working), this process could be completed way before time. Read more on using multiprocess in python here.
Multiprocessing works like promises in javascript synchronously. In easy words, It fetches the web-page/information in parallel, and stores it in memory. It processes the next half of the work such as inserting this to the database only when it finds free time between other requests or at the end of the process.
Processing this data, I got a huge collection of 40,157 KSRTC bus routes that run across Kerala. Actually, there are more buses, but anyways. This was massive with a total distinct count of 2,181 bus stops which were most common.
My main table that had 40,157 rows was the list of all bus details, but not its route. The routes of the buses were on a different table.
One of the most challenges of this phase was how I was going to filter the bus available across a route from this location to that location. It took almost 2 hours to figure that out.
I could either connect the two tables using the unique routeID, making it both an ultra-complex SQL statement and wastage of an extra column in all of 40,157 tables (You know.. just like a primary key). That’s not my way!
I ran another SQL statement that inserted the routes of each of the bus in its column separated by comma (shown in the red box). Now, this column could be used to check if the bus runs from one location to another.
To list the buses running from ‘Ernakulam’ to ‘Thrissur’, I could do this.
SELECT * FROM bus_table WHERE bus_route LIKE “%Ernakulam%” and bus_route LIKE “%Thrissur%”;
Chapter 3: Setting up the API using NodeJS
One of the easiest yet the backbone of this project is setting up the backend server.
I could have chosen Django, but I loved the simplicity of NodeJS more especially when dealing with JSON & API’s.
Little later did I know later that Javascript had just blown my mind as usual.
In JavaScript, its all about simple-user friendly codes and handing the synchronous and asynchronous methods.
That always excites me!
Promises in Javascript was mainly the key that helped perform the async loops and database searching before the response was given by the server.
Chapter 4: Developing the app & publishing to PlayStore
To make the app, All I required was a good UI at the beginning and for that, I take inspirations from dribbble and UpLabs website which have an awesome collection of designers works.
After that, I sketched the following screens. It’s beautiful! wow!
The app is live on play store and yes, It’s time for you to go explore the app & leave a rating!
NOTE: My original plan was to make this app for the Aanavandi team. I did ask their team and wait for 2 weeks for a positive reply before publishing app on the PlayStore. They did reply and rejected my offer. 🙁
I hope you learned something new from this project case study. I wish to come up with more amazing ones soon.
DO Learn, Share & Apply!
Adios.
Originally published at https://sreekant.me on July 14, 2019.
