Building a scalable back-office web application
Our journey designing and implementing a new back-office application to solve speed, independence and other issues
Introduction
We are Subito, an Italian second-hand marketplace that’s part of Adevinta. Seen from the outside, the marketplace application may look simple, but things are much more complex behind the scenes. We have an engineering team of (roughly) 60–70 people to run the application, divided into many autonomous teams based on specific business needs.
Originally, Subito was not structured this way, we changed, and with us our product, migrating from a monolithic architecture to a microservices approach. The product has evolved, both according to business needs and to changes in our structure and processes. However, the back-office application that was born with the product has not evolved in the same way as our company. It’s still structured as a monolith, a single application that handles all the administration tasks, such as content and user moderation, among many others.
We need to have these functions in our application — but we also need developers to be able to implement new features as quickly as possible. The evolution of this application delivers reduced return on investment because it takes too much time and effort to create new functionalities. Over time, monolithic application design has caused developer productivity to drop.
Last year we realised that it was time to take our back-office application to a whole new level, because adding features had become too slow, cumbersome, error prone and was not aligned with the current company domain structure and technical event architecture. Our old admin panel had few tests and there was no clear separation of concerns. Because of this, a developer was never sure that by modifying some code he was not breaking another team’s functionality, and that is the worst feeling for a programmer.
So what follows is our journey to create a new back-office application to solve the current problems.
Requirements
We sat around a (virtual) table and gathered all the requirements for our new application, including:
- Security: the tool exposes administration tasks, so it must be secure, and prevent any unauthorised user from accessing it. Plus, different users should have permissions that allow them to perform only authorised operations, so we need a RBAC (role based access control) system
- Independence: different teams must be autonomous and independent when adding/editing features. We don’t want to fall again in the trap of team entanglement, so a team must be free to deploy its own features whenever required without the risk of breaking others
- Extensibility: obviously, teams must be free to add their own features, so it should be easy to add a widget (more on what’s a widget later) and to manage it
- Speed / ease of use: because it is a back-office tool, you want to spend as little time as possible working on the app to free up more time to invest in product features (thereby adding value to the product). Making new contributions must be as easy as possible and common functionality must be in place — we don’t want teams to be concerned with authentication, access-control, etc.
What we did
With these requirements in mind, we decided to create a web application divided into multiple parts:
- Backend: the backend is a Node.js application (using Express) that takes care of authentication, authorisation (RBAC), and then can proxy requests to other microservices, built and maintained by the teams based on their needs. It also serves the frontend.
- Frontend: composed of two parts:
1. A client-side rendered React application, built with a micro-frontend architecture, that enables teams to work independently (more on this soon)
2. The micro-frontends (that we call “widgets”) that every team can build and integrate into the main React app
You can think of the React application as a skeleton that supports some common features like error tracing, logging, first-level routing, etc. This ‘skeleton’ then loads and renders widgets based on the route.
The widgets are micro-frontends, nothing more than React components that teams build for their needs. This allows us to have a widget to moderate content, another to moderate users, etc.
This was our high-level vision, read on to learn how we went about building it.
Security
Security is a top-level priority at Subito, so we implemented multiple layers of defence to further reduce risks.
Authentication
The most important thing is that every route of the application is protected by an authentication middleware. We have a corporate Okta instance, so we used it to provide SAML authentication. Only users registered in our Okta instance can login to the application. We also use Okta to manage group memberships and permissions.
To further protect the application it is only accessible through a company VPN.
Access Control
At this point we have a web application accessible only by certain users, but we don’t want every user to be able to perform every action — a content moderator should not be able to delete a user account for example, nor should they be able to access certain sensitive user information.
We know that we have certain “types of users” that are going to use our admin panel, so we categorised them into groups in Okta, using this platform to manage group access control checks. This means that in the future, we just need to add a new Okta group if we need a new ‘type’ of user.
When a user logs into the app, Okta tells us which groups they belong to, so we created a micro-service that takes care of the access-control checks. Every request made from the frontend is taken over by the Express instance, before proxying the request to the right service, it asks the access-control service if the requesting user can do the required action.
The access-control service has a set of policies that describe which type of user can perform which action, and based on this the service provides an “allowed” or “denied” response. If the Express instance receives a “denied” it returns a “403 — Forbidden” response, otherwise it proxies the request to the correct service and sends back the response. From that point, every micro-service is responsible for its own business logic, allowing different teams to own their features.
Independence and extensibility
Considering the current problem with the old administration panel, we realised that this aspect is as important as the security one. If we don’t get this right, in a couple of years we are going to have the same issue: no one will want to add new features because it may break others, conflicts arise when multiple teams work on the project, deployments become harder to coordinate etc.
This challenge is what the microservices approach was born to solve, and we were already adopting it for backend services. The backend Express instance provides common functionality, then delegates the business logic to different services.
With the backend issues resolved, attention turns to the frontend where building a monolithic React app creates the same problems — we cannot deploy independently, we risk cohesion etc. Fortunately, Webpack 5 was released as we were beginning this phase of the project and the new module federation plugin caught our attention.
The micro-frontends architecture already existed, but it was a bit cumbersome to implement, sharing dependencies was hard and there was no standard in place.
The module federation turned the tables a bit, as it established a way to easily architect micro-frontends, with shared dependencies for projects using Webpack (and soon the plugin was born also for rollup if you are interested).
We thought that this was a perfect fit for our needs, so we wanted to test it out and see if it was solving our issues without being too painful to implement.
First we tested to see if it was possible to achieve with server-side rendering, but it was not easily possible (at the time). There were some hacks to make it work, but considering that for a back-office panel we don’t need SEO or incredible performance, we didn’t look too far into this option to avoid unnecessary complexity. Instead we opted for a client-side rendered React application.
You can find a lot of examples of what can be achieved with micro-frontends in this github repository.
So what we built, as a platform team, is a skeleton React application which is served by the Node.js instance and handles several common features. Based on the route, the application renders the right widget, so teams (almost) never contribute to this application, they just implement widgets to be embedded containing all the business logic they need.
But wait, what is a widget? A widget is the implementation of a micro-frontend. This sounds complex, but in reality it is just a React component. Thanks to the module federation plugin, the component can be “embedded” into the skeleton application at runtime, while still sharing the common dependencies (like React, React Router, React-dom and many others) to avoid bloating the client. When built, the widget produces some static files, so you only need to host it somewhere (an S3 bucket for example) to be fetched at runtime.
These widgets are implemented in a separate repository (a monorepo), each one considered a standalone project. Technically widgets can have different dependencies (you could create a widget in Vue or Svelte for example) and they are deployed independently. Each widget has its own test-suite and the monorepo provides a Storybook instance where you can develop your component and use it as documentation before embedding it into the real application.
Here’s a couple of simplify graphics to explain how the application is structured. The black part is the browser itself, green parts are components provided by the skeleton application and the red ones are the micro-frontends, the only part of the app that is developed by the product teams.
Speed and ease to contribute
If you think about it and re-read all the above, you may realise that the entire system is not trivial. To meet the different needs of the architecture, the application has grown a lot and the more complex the application, the more difficult it is to make simple and fast contributions to it. The team behind the admin panel are engineers in the platform team and we are responsible for multiple common features. Our focus on the developer experience helped us to try to make the contribution process as smooth as possible (obviously it is still improvable, we’ll never reach perfection!).
On the skeleton application side, we set up all the common things the application needs so other teams don’t have to worry about them. This includes:
- Authentication
- Role based access control listed above
- Client-side error tracing with Sentry (every route that renders a widget has its own Error boundary with sentry integrated).
- A common logger usable by all widgets
- A set of helpers that widgets can use to solve some common issues that everyone uses, e.g. show/hide a UI part based on the logged user, because the API to delete an account may not be accessible by most users, but to provide a much better experience we want to hide parts of the UI to prevent confusion
Another important thing that we’ve done to increase the speed at which we can implement features is by adding an already built design system (component library). We have our own, but it was built to create the user interface of our marketplace, which is very different from an administration panel.
Instead of losing time implementing all the components that we would need here, (and most importantly to avoid polluting our design system), we decided to use Material UI; it’s well known, the documentation is great, it has (almost) every component you might need, it fits perfectly and you get a consistent UI throughout your entire application.
The part of “developing the widget” was good, but building new widgets was creating a bit of friction, because we had to do a lot of small tasks. In the skeleton application we needed to:
- Create a file that was the entry-point to load the new widget. Basically this meant copy/pasting a sample that provided all the good features explained above
- Add an entry point into the routing configuration to tell the app to display the new widget
- Add a few lines of code to create a link on the sidebar / homepage (if required)
- Add a line of code in the HTML file
- Change a couple of other details e.g. adding some extra lines in the Webpack configuration
In the widget repository we needed to:
- Copy/paste a sample widget that we’d built previously (containingTypeScript, Jest, React, Storybook, Material-UI and other important pre-configured libraries)
- Replace the widget name in some files
- Insert a port that is not already used by another widget (used to serve the component locally)
Some of these configurations are needed by the module federation plugin (to implement a micro-frontend architecture), some for other needs, but developers had to add them all. Even if there is a simple Readme with instructions, it’s still easy to skip something or to get something wrong, especially for a new contributor who doesn’t know the project architecture.
At this point, the contribution process was still a bit too resource-intensive. Early contributors needed additional assistance to get started or missed something in the setup, so we were not 100% happy with the as-it-was state.
Our original goal was to enable developers to do the minimum amount of work to set up a widget and integrate in the skeleton app. Freeing up most of their time to writing the actual widget with the logic they need.
To get closer to this objective, first we created a couple of scripts.
For example, instead of copy/pasting the widget and replacing the same collection of files, we refactored the widget as a template (using Mustache) and added an NPM script to scaffold a new widget that was using the template. The script then auto-filled the values using inputs supplied by the dev. This worked great, but it required developers to edit all the other files.
This is where Codemod libraries like jscodeshift come into play. This is a library made by Facebook to manipulate JavaScript files. It reads the JavaScript/TypeScript source code, breaks it down into an AST (abstract syntax tree) and gives you the possibility to step in, make some transformations and write changes to a new file.
It can look a bit scary at first, but if you try it, it’s simple and powerful. In Codemod we saw the potential to automate all the “edit config (and other) files” steps, reducing cognitive load on the developers who are just wanting to create a new widget.
After some trial and error, we refactored the two NPM scripts created previously (one to scaffold a new widget, the other one to integrate it into the skeleton application).
Now all the editing tasks are automated:
- When you run the script you get some questions in the prompt — widget name and port (with already a free one suggested)
- Which route path should show the widget
- If you want the widget displayed in the sidebar and/or in the homepage, etc.
The script then uses a couple of libraries to change every single file including HTML, TypeScript and YAML files. The “how to create a widget” section in our Readme now states, “Run the NPM create-widget and follow the instructions.”
We are very happy with the result and received nice feedback from a couple of devs that have used this new feature.
Conclusions
Like every application, we still have a lot of improvements that we want to add. For example, we have Sentry set up to catch all the errors in one project (think of it as a “bucket” of errors). It would be better to have a separate project for each widget, but this is not currently a strong need, so we kept the existing configuration.
Like every architectural choice, micro-frontend architecture comes with some trade-offs. It enables team independence and some new scenarios, interesting mostly for big companies or particular projects, but the added complexity may not deliver enough value in most cases. Always double-check your requirements and your engineering team structure for this kind of decision.
We are happy with the current “how to contribute” section of the project. The latest contributors have said it was easy and they were able to create new widgets in a short amount of time without any prior knowledge.
Working in the platform team, always focusing on developer experience has been useful for completing the last mile of this project, and we think it will pay off in the long run. Making it as simple as possible to contribute new features is invaluable because contribution is one major aspect of making the entire project successful.
