Open Source / Web Scraping

VITask- An API server for VTOP

The VITask API server acts as a custom-built API for our client applications on Android, iOS, Windows, and even a WebApp.

Welcome to our first blog post! We are Team VITask, the team behind the API service VITask!

VITask aims to simplify the process of building clients for VTOP.

The Necessity

Students studying in the Vellore Institute of Technology are already aware of the fact that the official website of the Institute does not only have an awful UI but also lacks good clients for other platforms such as Android/iOS. Since there is no public API it is extremely hard for the students to come up with their cross-platform clients to solve the problem. Nevertheless, for years students with proficiency in coding have been trying to build their clients and some of them have even succeeded in doing so. But due to the recent migration from FFCS to VTOP, most of these clients have stopped working. Moreover, the Institute lacks a centralized platform where one can check his pending assignments as well as calculate his attendance. The lack of Moodle integration has been a major limitation of the applications in the past.

The Initiation phase

In 2018 when I was still in my first year I wanted to build a service where people can check their timetable (or make a custom one). This service which I called Timeo was planned as a centralized service for Timetable, FAQs(related to FFCS), and Faculty Ratings/Feedback. One key feature of this service was the ability to create your custom timetable beforehand to avoid difficulty during course registration. I talked with my friend Vandit regarding this project and he was one of the earliest supporters of the idea. There was however one major roadblock in this project. To build such a service we needed the list of courses beforehand. Vandit arranged a meeting with the faculty responsible for managing this but she immediately rejected our idea. Disappointed with the result I realized that we are on our own to build anything.

Timeo was scrapped but then in July 2019, I came up with the idea to build VITask. The idea behind the service was simple- It would provide a custom-built API to develop cross-platform applications related to FFCS. I gathered a small team consisting of my friends Abhishek, Apurv, Hari, Harsh, Mayank, Prince, Raghav, Swapnil, Tushar, and Vandit and began writing the application in PHP. cURL was used to send the request to fetch the data and then Simple HTML DOM parser was used to parse the content. The APIs returned JSON of the structured parsed data and thus enabled the easy creation of Android/iOS apps. We completed the service with an auto-captcha solver and a beautiful web UI. The Android apps were under development when we got the news that FFCS is being replaced with VTOP.

The New Start

It was devastating news for the VITask team that FFCS was being replaced by VTOP. Because this meant all our contributions till now went to drain. Even the captcha solver which worked for ASP could no longer be used for the new reCaptcha. For months the project was abandoned because the VITask team simply lost its motivation. Then after a few unsuccessful attempts with selenium, I was finally able to automate the VTOP login system and extract the content with BeautifulSoup. I decided to proceed with Flask instead of PHP due to support for Selenium and BeautifulSoup. Firebase Realtime DB was finalized to store the data because of its easy integration.

The Development Phase

While working for the website team for Vibrance in my 2nd year of University I came in contact with some new talented developers. One of them was Yash who had experience with ReactNative and immediately joined to help the VITask team. Another person who joined the team was Apoorv who was a senior in 3rd year. He showed his interest in the idea of VITask providing open APIs to build cross-platform applications and the team which consisted of 11 people rapidly expanded to around 20.

One of the many fun aspects of a group project is the influx of new ideas from the internal team itself. Initially, VITask had a loading time of around 50–60 seconds for fetching and setting up everything. It also occupied a lot of RAM in the VPS since each user login created a separate selenium instance. The captcha also needed to be manually solved to authenticate. Apoorv streamlined the process by using the Pillow library to automatically solve the captcha. He then went one step ahead by optimizing the fetching-process to use Python Requests instead of Selenium. This meant that the login which initially took around 50–60 seconds could now be done within 10 (and without any captcha).

Future Prospects

VITask has been rapidly evolving. The project contributors are themselves taking responsibility for upgrading it. It is slowly growing into a community-driven project but it is still far from any ideal Open Source project. The idea to integrate everything related to VIT on a single platform (from VTOP to Moodle) and making it available to everyone is transforming from a figment of my imagination into an actual application. With the upcoming launch of VITask API Server and our different cross-platform clients for Android, iOS, macOS, Windows, and Linux I believe that VITask can leave a long-lasting impact among the young developers in our University.