Shiksha Social Signup
Revamping Registrations
The BIG Rewrite
Author : Anirudh Dey
A user’s journey on a website generally begins with registration. They signup and then access various contents on the website. Recently, we at Shiksha had the opportunity to make this experience even smoother for the users.
The idea to revamp registration seemed not only beneficial from a technical standpoint but from product perspective as well. All teams across Shiksha were already in the process of shifting our existing tech-stack to Microservices and React PWA, it seemed the correct moment to do our bit as well.
Despite the obvious pros, the initial question was :
“Is a rewrite from scratch required for such a core functionality?”
Robert L. Glass’s quote from Facts and Fallacies Of Software Engineering paved the way for us.
“Modification of reused code is particularly error-prone. If more than 20 to 25 percent of a component is to be revised, it is more efficient and effective to rewrite it from scratch.”
The introduction of Social Signup (Google and Facebook) effectively meant that we had to make large additions to the existing codebase. And so, we finally decided to migrate from PHP to Java Microservices for backend and to React PWA for frontend.
Defining Benchmarks
Surpassing set standards in terms of performance, scalability and availability were on our mind from day 0. Zero downtime and smooth transition from old to new tech stack with minimum exceptions was a challenge in itself. It was an exciting opportunity for our team to scale the new registration and response service to at least 10X of Shiksha’s peak statistics where X was the peak registration number in Shiksha ever. But we aimed at much more.
Discover. Design. Develop. Deploy.
The norm for the next few days, if not weeks, were endless discussions, to understand existing flows, to not miss out on anything, for optimizations, designing flowcharts followed by swift discards for better alternatives. The aim behind this was to design in such a way as to not have any major bottlenecks and pain points towards the end.
The project was in strong adherence to the basics of Software Development Life Cycles from the very first phase of planning and requirements and the project was executed with active participation and efforts from all stakeholders which ranged from Business to Product and from Developers to QA.
Overcoming the hurdles
We came across some issues and concerns that we tackled one by one.
Talking about backend, our major concerns were:
Concern 1: Limit real-time database inserts.
Solution: Messaging queues. Kafka, in our case. Data that was trivial in nature or pertaining to just tracking could easily be inserted asynchronously. Inserts that feed user data to other data sources such as Solr and ElasticSearch could be fed via queues too. In our main registration API, we were able to decrease real-time database inserts by a good 60%.
Concern 2: Serve data from frequently APIs quickly
Solution: Cache all the things. Well, not all but a lot using Redis. Data and fields in our registration and response forms broadly consist of Shiksha’s entities. These could certainly be cached. Redis and caching of certain APIs with predefined TTLs provided us with better performance and decreased response time. This decrease in some cases was more than 50ms and allowed us to handle more load.
Concern 3: Incorporate Concurrent calls.
Solution: Reactive Programming, especially Mono and Flux came to our rescue in order to make concurrent calls where the sequence was not of importance to fetch data from various data sources.
Concern 4: Ensuring extensibility and maintainability
Solution: The code was written and designed in such a way so that later on, existing code could be incorporated or modified in various modules across Shiksha. Design Patterns and best industry practices were used. Many registration APIs were used with little or no modifications across Shiksha Assistant, Shiksha Android App and Shiksha StudyAbroad.
In terms of front-end, major opportunities were:
Challenge 1: Make new registration/response form available on NON-PWA pages on Shiksha Website.
Solution: We did this by opening the registration form on a separate URL and then embed it into the non-PWA pages using iframe.
Challenge 2: Shiksha has more than 5 different types of forms. Before revamp,all were totally separate. An exciting task for us was to make it common. A singular change should reflect across all forms and on all platforms (Mobile Site, Desktop, App).
Solution: This was taken care of by careful designing of architecture along with singular components that can be used for multiple purposes. Post revamp, the new form was platform and type independent with codebase centralized at a single point.
Challenge 3: Loading of form JavaScript and associated entities should be on demand
Solution: This was achieved using ‘loadable’ which ensured that code splitting was done for the registration/response form components and a separate webpack chunk was created for it. It helped us to ensure that the form should in no way affect the size of other pages and chunks simultaneously being served since we have hundreds of different CTAs on Shiksha. Webpack-bundle analyzer was a great tool that we used to figure out any bottlenecks.
More information regarding this can be found at Check-In at Shiksha.
Performance and Load Testing
To ensure that we met the performance targets set for our team, we created a custom suite on Apache JMeter for testing purposes. A variety of tests were performed which analyzed not only standalone APIs but also successive API calls which were done in a bid to replicate user behaviour who might submit one layer of our form after the other.
The performance testing of the API 1 followed by successive API 2 gives the following concurrency vs latency graph.
Since the latency stagnated over a period of time even when increasing concurrency is an indication of the fact that API is working correctly. If there would have been a y=x projection in the graph, that would have been an area for concern for us as it would have implied that on high traffic days, response time would start increasing as well. But since it started tending towards zero or negative slope, it gave us the confidence that no red flags would be raised in case of high traffic or load on our website.
The End Result
The question is, did we accomplish in our goals? Indeed, we did. Despite the challenges faced, the performance metrics and stats are a testament to that fact.
- The performance benchmarks that were set for ourselves, we have successfully achieved. Rather, overachieved. The majority of our APIs respond within 70 ms on a daily average basis. Service has consistently performed well on high traffic days as well with a sudden increase in throughput.
- In terms of form load, the new PWA form loads within 15 ms on average which is a marked difference from the old form whereas the getRegistrationForm API performs under 10 ms on average.
- Faster APIs combined with the client-side rendering of the forms mean better performance for users traversing our website.
- Enriching experience in terms of learning for all developers involved due to the paradigm shift in Shiksha’s tech stack.
