Building a new platform for innovation: A technical perspective
By Jiquan Ngiam
What’s up with this old-to-new platform switch? What’s different about Coursera’s new platform? In this blog post, Director of Engineering Jiquan Ngiam shares and reflects on how our technical platform has evolved over the last few years.
When Coursera launched, we worked on a simple LAMP stack that kept us fast, scrappy, and stable. Our early platform accommodated rapid development of the many features we needed to support a diversity of courses — everything from quizzes, to peer review assignments, to forums, to authoring tools. By prioritizing agility, we were able to get hundreds of courses up and running in a matter of months.
A year of this fast-paced growth, however, left us with a tech stack that had grown into a monolith. We had added features incrementally without thinking holistically about the complete architecture, resulting in different pieces being too tightly coupled. For example, we supported a feature that connected peer reviews to discussions so that assignment submissions and reviews could be cross-posted; but we had failed to make a clean separation between the forum and assignment functionality, which made the overall system fragile and difficult to operate.
The technical structure of courses also made course reruns very challenging. Setting up a repeat session of a course required extensive manual labor, and as a result, few of our partners had the resources to continue offering courses beyond their first session. We only had around a few dozen courses available for enrollment at any time, even though our catalog was much bigger. This meant that learners had limited access to courses, and many learners who were interested in a specific course never had the opportunity to enroll.
Furthermore, we did not build the course experience to be an optimal mobile learning experience. We had to create workarounds for our mobile clients to make API calls, and contort our quiz experiences to work on mobile.
In sum, our ability to continue innovating and improving the learner experience was extremely limited. We realized that if we wanted to iterate quickly, we had to decouple and rethink our architecture. Fortunately, we came to this realization quickly. In 2013, we started moving systematically in the direction of a service-oriented architecture.
We began by building independent services that our monolith could call out to and leverage: notifications, eventing, authentication, and others. Building these initial services helped us develop the expertise and tooling (including Naptime, Courier, and Rapidash) that we needed to build services; we also built out development tools.
In 2014, we were ready to step up our investment into the new platform. We formed a new team, FLEX (Future of Learning EXperience) to focus on the new platform development and transition. We built services to serve our courses, including separate services for assessments and scalable programming assignment grading. Features on the new platform were also developed with mobile functionality in mind.
One of our key goals in developing the new platform upgrade was to be able to offer courses continually. Today, the new platform hosts more than 1,000 active courses, each offering a new session about once every two weeks. And in the last 2 years, we have worked hard to ensure that the new platform provides an outstanding experience for the instructors who teach on Coursera, as well. We now have an advanced instructor authoring experience, including a course versioning scheme that allows instructors to make, and test, major updates to courses without disrupting the learner experience.
We have also seen our engineering velocity accelerate: in 2015, after further decoupling our frontend and backend stacks, we started making over 30 releases daily. We now use the latest technology in our stack, including Cassandra, React, Scala, Swift, and more, and we run dozens of A/B tests every week as we work to optimize the learner experience.
As the new platform has improved, the shortcomings of the old have become clearer. Not only does the old platform not support many of the new features we have developed for the new platform, but it is also very costly to maintain. Bugs and outages occur frequently on the old platform, and keeping it up and running requires developer time and resources that would be better spent improving the overall learner experience. Furthermore, when we develop new features for the new platform, we need to consider how those features will interact with the infrastructure that both platforms share. We are frequently forced to spend significant time ensuring that changes to the new platform do not break features of the old.
The new platform is now developed to a point where all of our existing course content can be migrated to it. We are therefore taking the final step of moving all courses to the new platform, and formally retiring the old. We are working closely with our partners and instructors to make this transition as smooth as possible for learners, specifically with regards to content availability and course records. With a very few exceptions, courses that are currently hosted on the old platform will move to the new one in the coming months. Learners who have completed courses on the old platform will retain access to their Course Certificates and final grades in their profiles.
We still have work to do — in the coming year, we will be focused on further improving site performance, guiding our learners better, and helping learners achieve bigger personal and career goals. And we are looking for more smart, motivated people to join our team! If working on a technically challenging problem that has real impact to people excites you, we hope you will apply for one of our open positions.
Originally published at building.coursera.org on June 21, 2016.