Route between SSR and CSR service for Website using HAProxy

Creating separate microservices and routed differently at Mitra Bukalapak

The t̶r̶o̶l̶l̶e̶y̶ render problem

If you are working related to deployment and infrastructure, chances are you have already heard of a thing named HAProxy, a marvelous piece of software that exists as a solution for routing and load balancing. Here, in Bukalapak (especially at Mitra Bukalapak), HAProxy is actively used to solve problems of implementing SSR (Server-Side Rendering) mechanism under existing CSR (Client-Side Rendering) service, without need to rewrite our frontend stack at the core. If you ask me as a Software Engineer, HAProxy is a lifesaver for our development. It saves our development time, is easily configured and separates our service into smaller, different contexts, which means easily maintained (instead of one giant service with multiple contexts).

The Background

Mitra Bukalapak Web (which resides at mitra.bukalapak.com), was a web-based application that was mostly rendered and processed client-side (which is popularly called Progressive Web Apps). It used to be hosted by a microservice, managed by Kubernetes inside the whole cloud architecture. Initially, we had an architecture that looked like this:

Yeah, this is oversimplified lol

By that architecture, the SSL (https) was already managed by our Bukalapak nginx in the outer layer, so software developers could focus on our microservices (which are orchestrated by Kubernetes). Basically, the flow was:

• users accessed mitra.bukalapak.com,
• request forwarded into pwa-service ,
• and then it serves the HTML, JS and CSS needed to the client.

Since pwa-service is client-side, it sends a bunch of JS bundles and lets the browser render the rest. Things are good and work as intended. Until…

The Problem

In pwa-service, we had a landing and catalog page, so non-logged-in users (not registered) may see what our products look like. Captions and text being added for explanations of the product and how it works. But then we realized that non-logged-in users are “organic users” that mostly come from search engines. This is where we realized, we pwa-service cannot provide rich search results due to the nature of the client-side. We were unable to rewrite our pwa-serviceto be SSR-compatible since the codebase was too much oriented to the client-side.

The (almost) Solution

We had assessed the problem, and the first attempt was to create another microservice that was built with SSR in mind and serve pages that needed to be pre-rendered. We picked some pages from pwa-service to be moved to our new service (called ssr-service) and listed the possible routes. We also put common code into libraries, so we didn’t have to rewrite most of the things. Therefore, the proposed idea was:

• Route the candidate, SSR-compatible pages to be served by ssr-service, and
• Route the logged in pages to still be served by pwa-service.

Neat’o! …

… or not. We faced a problem.

Basically, our nginx is in a shared domain, across the whole Bukalapak. For every single route that added to the config, we must assess the impact on other routes in Bukalapak and dry run the config (which takes quite a long time due to the size), and we didn’t expect one thing:

There are pages with the route that can be served by either pwa-service or ssr-service, depending on certain conditions.

At the time, we were incapable of implementing such logic in our nginx. Until…

The HAProxy Solution

Our colleagues suggested another software called HAProxy. It’s lightweight, standalone, performant, and surprisingly easy to integrate into our system. We removed all routes we had been added from nginx, and added a config to let all traffic from mitra.bukalapak.com go through our HAProxy. We used HAProxy version 2.1 under alpine docker.

So, in terms of HAProxy, we can translate this architecture into two backends (service/destination) and one frontend (entry). The base structure of HAProxy configuration (haproxy.cfg) looks like this:

For example, the pwa-service resides at 172.16.1.1:3000 and ssr-service at 172.16.1.1:3001. Both services should have /status an endpoint that returns HTTP 200 (as described in httplog and http-check config). In addition, the configuration under defaults describes whether a backend is up or down.

Then, we want /artikel, /tentang-mitra and /aplikasi-mitra to be rendered in SSR. It means those routes need to be registered and mapped into the backend ssrservice. To achieve this, we need ACL (Access Control Lists) and use_backend (plus default_backend for default):

Basically, acl is like a boolean match, which in the programming language, it looks like:

bool tentang_mitra_page = path_beg(“/tentang-mitra”);

For extra notes, we have determined to avoid using path_reg (or any regexp function) unless there’s no other way to parse the route since regex execution is quite expensive to process (and bad regex is even more) and it may consume higher CPU usage and drag down the HAProxy speed (thus latency increase).

Now, to solve a route that might be served by either of the backends (with conditions), we can just add more ACLs and chain them at use_backend. For example, you can detect a logged-in state by checking if a cookie user_session exists or not.

Since the cookie is embedded in headers, HAProxy can read cookie just fine by using hdr or hdr_sub (which means header or header substring).

In our case, /grosir should be available for both login and non-login users, but each is served by a different backend.

This way, non-login users will have their /grosir served by ssrservice and logged in users will be served by pwaservice.

In addition, by using hdr/hdr_sub, we can set a page to be protected when accessed from certain clients by its user-agent or other headers. In this case, we want to have /promo only been accessible from certain apps. It would look like this:

Deployment

For deployment, since our HAProxy is orchestrated by Kubernetes, we only need to configure a Dockerfile using haproxy:alpine the template and copy the haproxy.cfg into it. Expose only necessary ports that we explicitly add in frontend bind parts (port 80, for example). Because of very small footprints, we only need to allocate about 64MB memory and ~0.125x CPU, and then, we replicate it to 3 pods. This way, mitra.bukalapak.com will stay up (zero downtime), even though one of the instances goes faulty, crashing, or even just rolling out config updates.

Results & Conclusion

And there we go! It works neatly in production and it has almost zero latency overhead whatsoever. Also, we saved development time from approximately 3 months with the risks of hybrid SSR — PWA being faulty (due to code rework) into less than 4 weeks (two sprints) with minimal risks and easier rollback if something went wrong. We have successfully implemented a system where traffic is routed to between Server-Side Rendering service for non-login users and Client-Side Rendering server for logged in users without the need to rewrite our current service into a complex system with hybrid capabilities. Although in the end we have to write a new service (for ssr-service), and we got helped by numerous production-ready boilerplates and framework so it takes very little time to do so.

It’s pretty fun for us tinkering HAProxy due to its wide capabilities and easy syntax. We have shared the knowledge of this HAProxy with other colleagues internally and people seem to be excited and understood easily. In addition, we are able to add some improvements in terms of reliability and monitoring.

Additional Improvements

Logging

One of the basic requirements of service going up is logging, which HAProxy already supports out of the box. By doing this line,

log stdout format raw local0 info

any activities will be logged to stdout and then, we can trace the log inside Kubernetes pod logs. But logs are a bunch of text, and most of the time, we want to have a summarized, general idea of what’s going on at HAProxy. If you (or your company) are using Prometheus as the data source, HAProxy has a very good module to automatically log things into Prometheus format:

HAProxy Exposes a Prometheus Metrics Endpoint — HAProxy Technologies

But in our case, we are using another log processor that uses JSON format as a data source (like Kibana). By default, HAProxy logs the activity using their own format, but it’s easily configured how the log is written. Therefore, the idea is to reformat the log into JSON using this simple line:

Now, every log will be formatted as JSON with these labels:

Things that are prefixed by “%” are placeholders of values HAProxy provides. List and details of what value HAProxy provides are written here: HAProxy Enterprise Documentation version 2.2r1 (1.0.0–233.105) — Configuration Manual

Now you can easily query those data (using your preferred data discovery) like how many requests elapsed more than 3 seconds (duration > 3000) or count of requests that goes into the backend pwaserver. Marvelous!

Stats

HAProxy also has a minimal status page that displays an overview of incoming and outgoing traffic using “stats”. We use this to inspect HAProxy healthiness at the time (usually when our on-call engineer inspects the reliability of our services). The values are quite raw but nice enough to indicate what went wrong or well. The config is as simple as this:

Then, from the internal network, we browse to /haproxy URL to see the page.

In addition, since these stats will return HTTP 200, we used it as a healthiness check for our HAProxy in Kubernetes.

Dynamic Resolve (DNS)

During our adoption of cloud architecture, our services are now configured with a hostname, instead of direct IP and URL (because of service discovery). HAProxy also supports hostname, so all we have to do during adoption was just change the IP to:

In this example, ssr.mitra.internal was resolved to 172.16.1.1

It works, but we didn’t realize how the HAProxy resolves the hostname. It turns out that, under the hood, HAProxy resolves the domain using the active DNS used on the network at the startup, and then uses the IP. In most cases, it was not supposed to be a problem. But we are talking about cloud architecture, that sometimes even the internal IP changed midway, and HAProxy didn’t realize the “ssr.mitra.internal” was changed into another IP (due region switching or migration), and then suddenly failed.

Fortunately, HAProxy also has a solution to this kind of behavior. By declaring the DNS server in the HAProxy configuration, and tell the HAProxy to actively refresh the hostname when being accessed:

and then add the resolver to the backend:

Now the backend will attempt to check the resolver internal DNS to resolve ssr.mitra.internal before continuing to forward the request.

For more details, the HAProxy team wrote a very nice article about DNS as service discovery at their blog: DNS for Service Discovery in HAProxy — HAProxy Technologies

That’s all from me. I hope my experience tinkering with HAProxy might give you insights and additional perspectives on how you configure HAProxy.

Thanks for reading!