Rate-Limited API and Fiddler: Real Life Case on Limited Official API
There’s this a case with third party application which does not making any sense. Our company operated in Education sector and implement digital based learning using digital books. Every time when students complete their payment related to book access, that related students should have an access directly to that book(s). However, what happen on the field is our staff needs to manually update that particular third-party apps with the student data, like which books he/she requested, does she/he already have an account to access the third-party apps, etc. The process, of course, is not only time-consuming, but also prone to complaints and such.
After hours of surfing on the apps documentation, it seems like they have an API related to the previous time-consuming activities. We can automatically update the systems with the particular student’s data every time a payment has been made, so i set up our programs so it can automatically update based on data progression on our database. Things seems to be working well, until we reach our peak moment when students payment activities increased significantly. That API can’t handle our request that much. I looked up the documentation again and it states that the api can only handle one api request per second, when our systems request three api request in one payment activity per second. To make things worse, there’s a rate limit of 4000 account request per day. I don’t know why I miss this note in the first place, but what makes me confused is why my company’s top level never negotiate this kind of thing when they know we have more than 300,000 students and it could increase every semester.
I don’t know what when behind the curtain, but it seems like there’s no movement regarding the improvement at the moment. They event revert to manual system for no reason other than “to speed things up”. Curious about that “to speed things up” words, I take a look at the apps for the first time. Weird enough, I am not granted any access to it even when I created the automatic solution before. By apps, I mean a desktop apps used to update our book catalogue, student data, library, etc. To put it simply, it is the control panel of our digital books ecosystem. By using this desktop app, our staff can update user and it related books on bulk using excel based files, a feature which is not offered on the API itself and also there’s no rate limit.
Even with this, it still not efficient because our staff will be occupied with finding data, creating files and uploading it directly from their computer. To really speed things up, I decided to test whether i can intercept the apps request to find where the excel is transferred to and i can replicate it within our programs. I stumble upon Fiddler Classic, an app that helps on intercepting request from desktop or web application.
It’s pretty simple and straightforward i just need to open the fiddler, run the desktop application, upload an excel, find any host/url with the application name and inspect the request. Luckily, the particular activity and the URL name is similar, and I can directly inspect the body and its content. After inspecting the URL, it seems like most of the data is static, by static I mean it doesn’t require any activity like login, update session and such. I can easily replicate this activity in our programs to solve the rate limit problem and the redundant activities. I decided to create a cron based script which run every 10 minutes (to mimic our previous API based system, just in case this way is illegal) and fetch any data that needs to be updated to the books system, create an excel file and uploading it to the previous inspected URL. After all this kind of struggle, it's kind of weird that actually this process can be exposed through their API service, because it almost the same as how I interact with before.
