Traveloka Web Framework-ed
This is a story of our soon-to-be-deprecated JS framework used in Traveloka mobile web. No, this is not a sad story, but rather a journey full of insight and learning along the way.
The story began at early 2015, when we decided to write our own server-side rendered, single page application JavaScript framework.
Why?
You might be wondering why we decided to build our own JS framework.
Remember, that was early 2015, where Angular was still gaining traction, jQuery was popular as ever, and React? Well, let’s just say people still couldn’t get over HTML-in-JS.
Frontend engineering was still relatively quiet, there’s no JavaScript fatigue, everyone was happy with their tech choices at that time.
We’re using jQuery bundled inside OOP-heavy application structure via Google Closure Compiler. Our HTML app shell was generated by a monolith Java application served in EC2. We created components using something similar to Angular directive (don’t ask me why we’re not using Angular). Our mobile web worked perfectly as single page application.
That was, until we had to consider SEO.
Search Engine Optimization
Whether you like it or not, Google is a gateway for many people coming to the web. When they’re looking for something unknown, the first thing that comes to their mind is Google.
Due to rising popularity of mobile devices and single page application built especially for mobile web, in mid 2014 Google introduced the ability to crawl your site like any normal person. Google can now open your SPA and render the content in client side. This means our SPA should be perfectly fine in the eye of the crawler, right?
If something seems too good to be true, it probably is
Our research showed otherwise. Using app shell method means there’s almost no content at all to be displayed at initial render. Googlebot needs to download and parse a big chunk of JS before seeing anything, and as you might know with most SPA (ours included), it can take a long time to wait. Crawler, doesn’t like to wait.
The best solution we could think of was to make our page content visible in initial render without any JavaScript needed. With that in mind, we’re brainstorming on how we should tackle this problem.
Server Side Rendering (SSR)
We know that SPA provides better UX for the user for the subsequent navigation, but SSR is beneficial for crawlers and (arguably) users as well.
We’d never done anything like this before, resource around this subject was scarce, and the framework offering at that time didn’t give us what we really wanted. They’re either client-side only SPA, or classic progressive enhancement with only a sprinkle of JS.
Then, we did something that any curious engineer would do, we opened a lot of sites and analyzed how their sites worked. We wanted to know if there’s a site that at least had a similar requirement as us. Finally, we stumbled upon Twitter.com, (the old twitter, not the new PWA Twitter Lite built using RNW 😉) and we smiled.
Hybrid Single Page Application
Much to our surprise, Twitter had already implemented exactly what we wanted (we found out later on that Google and Facebook had done similar thing as well). Closer inspection using Chrome DevTools network panel showed us the way they implemented this:
One key concept with this approach is almost everything is rendered on server-side, including when doing client-side navigation. With server-side rendered HTML on every navigation, the included JS payload is used mostly to add interactivity on top of existing HTML.
Remember our Java-jQuery stack? Turns out our stack was a great match with this approach. We already rendered HTML (albeit only app shell) in Java, and there’s already client-side navigation with hash history. What we needed was the way to fetch JSON payload when doing client side navigation. That’s when we started building our own JS framework.
We call it Blocks.
Blocks Framework
Yeah I know, it’s cheesy, lazy, and unoriginal. By now you must be tired hearing about lego, block, brick. Modularization blah blah blah. The truth is, the name came from how we structured our pages and components.
A common approach when doing client side navigation is to re-render entire page with the new set of components. But, we did things differently, or at least different than most people.
We provided a set of placeholders that component could render into. Each component would have a specific placeholder (for example the most common placeholders were header, content and footer). A placeholder could contain multiple components, but a component could only have one placeholder).
This way, when doing client-side navigation, instead of fetching one big blob of server-rendered HTML inside JSON payload, the API could return a list of smaller HTML with their specific placeholders, then the bootstrap script included in initial load would render new components inside their respective placeholders.
Using placeholders meant we could reduce HTML overfetching by comparing current rendered HTML with the necessary HTML we needed to reconstruct the new page. We could also compute necessary DOM operation to replace the page by reusing as much rendered component as possible. Yes, this is similar to how most Virtual DOM libraries work, only our version was much less powerful.
JSON API?
Another important part for this framework to work properly, was to provide JSON API that returned partial HTML and list of necessary JS and CSS. We did this by utilizing the same placeholder concept explained earlier.
Each page was implemented as a Java class that extended this abstract class:
To create a page we extended this class and provided a base template to be passed to super(). In the class constructor we could register our components and JS file necessary to render this page. When request came, we already knew what components existed in this page, and we could generate full HTML string by executing renderHtml method.
We did similar thing in renderJson method. Because the list of components was known ahead of time, we could create specific response type that lists the components needed to be rendered in specific placeholder if client-side navigation occurred.
How did we know the difference between initial render and client-side navigation? We put a special query string in our URL. The controller could determine which response that would be sent based on the query string alone. This could be as simple as this:
Initial load
GET https://m.traveloka.com/en-id/faq
Content-Type: text/htmlClient side navigation
GET https://m.traveloka.com/en-id/hotel?type=json
Content-Type: application/json
Error Handling
The key to good hybrid SPA is how we handle error gracefully. With a lot of moving parts when navigating between pages, one error in the middle of navigation process can break the entire page and frustrate users. To do this, we employed a simple error handling approach:
Whenever an error happened in the middle on navigation, we simply refreshed the page.
These kind of errors happened after we changed the URL, so refreshing the page solved all possible problems in client-side navigation and provided good UX for our users. Our pages worked normally as classic HTML page.
Deprecation
Fast forward 3 years, and almost no page uses this framework anymore. The remnant can still be seen in various places, but it’s no longer the same framework that we built before. What we use today is the usual stack of react application you can find on the web: react-router, react-loadable, next.js and friends.
If this framework solves a lot of our use case, why did we deprecate this?
One of the reason is partly because of the migration effort from Java monolith to Node.js microservices (for web servers) in late 2015.
To accommodate the migration, we created different libraries to mimic the behavior of our previous Java-based API, and in turns made other engineers confused between which part is the framework, and which part is the actual function of that specific library.
The way we ported our framework from Java to Node.js was quite complicated and full of hacks, just for the sake of making the framework worked in the new environment. This created a huge technical debt that slowed us down especially when we tried to add any improvement or optimization on top of this framework. Instead of having clean code in the new environment, we polluted it from the start by writing a hack.
Another reason is the difficulty of integrating JS code inside google closure compiler with modern JS ecosystem around npm. We had even built the transpiler from GCC-ES5 to Flowtype-ES6 but it was never picked up. With React on the rise, we’re not sure we wanted to keep our jQuery spaghetti in the future.
Conclusion
Providing good baseline framework that performs well in our users’ device while maintaining good DX for our engineers is hard. It takes full time effort. It’s even harder that we had to build this with limited knowledge that we had (we’re less than 1 year in the company when we wrote the framework).
But writing this framework gave us a lot of knowledge along the way. We learned the intricacies between mobile browsers especially regarding client-side navigation (Blackberry and Windows Mobile took the crown for this). We learned how the web worked, we could learn from other people indirectly by scourging their sites with DevTools. We also learned that we’d been doing route-based code-splitting even before it was cool!
Ultimately, we learned that even though the idea were novel, our implementation was not. We failed to integrate the framework with React and its ecosystem properly.
So yeah, we didn’t build a JS framework for fun. If you heard that we liked to build framework ourselves, it’s because nothing is available in the open source ecosystem that works for us yet. Fun is only one of possible side effects that happen while you’re building a framework yourself. Fear is the other side effect 😆.
At this time of writing, we’ve been actively developing yet another React-based UI framework solving some issues that comes with universal-rendered react application (SSR + client side hydration).
The goal of this new framework is to provide great DX for our engineers while at the same time set good performance baseline out of the box for any universal-rendered React applications. A win-win situation for both our engineers and users.
We can’t wait to share our findings after we finish experimenting with this new framework.
The JavaScript ecosystem has grown a long way since we wrote the framework back in 2015, and a lot of people are experimenting with great things that we can learn today. This is partly why we’re optimistic about what we’re currently building. With so much tools available for us, now is really a great time to build on the web.
