Breaking Up Our JavaScript Monolith with Rapidos
By Evan Yeung
The frontend infrastructure team at Coursera works hard to make developers as productive as possible. This includes keeping up with the latest and greatest tools and libraries, but also means making builds and deploys as fast and painless as possible.
As our codebase has grown to harbor over 40 single page apps in one monolith, we found an opportunity to improve our developer experience by splitting up our applications.
Why Rapidos?
A couple years back, we built a system called Rapidash to decouple our frontend and backend deploy processes. However, as our codebase grew, new problems surfaced that Rapidash just wasn’t built for:
- Build and deploy times became longer.
And longer. And longer. From about 3 minutes when Rapidash was created to over 15 minutes today. Every time we added a new single page app, the compile time went up. - Bugs block deploys.
A single bug introduced into master grinds continuous integration to a halt until a fix is deployed or the commit is reverted. This prevents everyone from deploying changes, even ones not associated with the app that broke. - It’s easy to break someone else’s code.
Using reusable components means you don’t have to rewrite code. It also means someone else may have started using your component in their app. Now, when you update your component, their app could break because of it.
Rapidos is born
A solution to these issues was already being used in Coursera’s backend: microservices!
Well, not quite microservices, but this was the inspiration. Our backend developers were enjoying the safety of working on their own services that they could deploy and rollback independently of other services. We took this model and broke apart our monolith, allowing developers to build and deploy individual apps. Now,
- We can parallelize builds while keeping the amount of code in each build small. As we grow our codebase and add more apps, the build time should remain approximately constant as each new app is built independently.
- Since Rapidos is capable of deploying apps individually, if one app breaks, the rest of the site can still be deployed.
- Rapidos will tell you which apps changed with your commit. If someone used your code in their app, you’ll be notified to go and test that app (or get them to test it)!
Aside from improving the developer experience, Rapidos also advances our engineering culture of ownership and accountability. It enables each product team to wholly own their applications’ delivery cadence, monitoring, availability, and performance.
Building Rapidos
The Rapidos service itself is a build pipeline for coordinating and running tasks. It’s just another microservice in our Scala backend.
When a developer commits new code, Rapidos is notified of the update and begins a transpiling job on Jenkins. Once complete, apps changed by the commit are bundled and optimized before being uploaded to Amazon S3, ready to be served through Amazon Cloudfront. Unit and integration tests are run, and if everything looks good, the developer is notified via Slack to go and deploy their change! From Slack, it only takes two clicks to select their commit and deploy.
One of the ideas we focused on with Rapidos was improving developer experience. Beyond the obvious idea of making it as easy as possible to get code changes into production, this also meant making Rapidos itself transparent and easy to debug for other developers. By organizing jobs into modular tasks and keeping as much of the frontend build code in the frontend repository as possible, Rapidos was designed to be easy to use and understand.
Over the next couple months, we’ll be moving all of our frontend code to be deployed with Rapidos. Our codebase, and frontend engineering in general, is getting more complex as we build more products, grow our engineering team, and launch new features. With that in mind, it becomes more and more important to focus on developer productivity. Rapidos is one way we’re helping our developers focus on writing code and getting stuff done.
Hi! I’m Evan Yeung, an intern on the frontend infrastructure team at Coursera. Over the past 4 months, I’ve had an amazing experience building Rapidos, meeting new people, and learning at Coursera. I’d like to thank my mentor, Jon Wong, along with rest of the frontend infrastructure team for a great fall term!
Originally published at building.coursera.org on December 15, 2016.
