Rebuild of idealo’s start page

Authors: Philipp Thomas and Ingo Stöcker (software developers in team Tyrell Corp of PA Product Exploration at idealo).

Motivation

Over the last few years, idealo grew a lot, and slicing the organization and its subdomains became one of the most important challenges to ensure fast product development.

idealo — founded in 2000 — is still operating a monolith application (called IPC — idealo price comparison) which still delivers some of the core business cases.

After the extraction of a few page types (e.g., product page, wish list page), it was time for a green-field rebuild of the start page.

The main goal of the project was a technical migration to speed up product development. This gives us the opportunity to create a user-centric design easier and even faster.

Among others these were the requirements:

• Security
• Page Speed
• SEO (Search Engine Optimization)
• Tracking (business + user-related KPIs)
• Modern tech stack
• Cloud-readiness

High-level architecture

In the early research phase, we decided to have a dedicated front- and back-end application.

Due to the different tech stacks and a clear separation of concerns, it was a straight-forward-decision. Furthermore, you can scale the application according to their requirements which makes the whole setup more sustainable.

Additionally, we enjoy faster build times in each application and could select one of the modern front-end tech stacks.

We did roadshows in each involving and affected team to collect their feedback and get a commitment to start with the implementation.

Here you can see our architectural approach. We will go into detail in the next two chapters.

Frontend

Back in 2017 idealo decided to build new services as Self-contained Systems (SCS). This approach fits the per-page-sliced idealo frontend where teams have e2e responsibility for a complete page.

But everything comes with a price. The frontend teams must handle the integration of their features depending on the tech stack of the surrounding page.

Since an easy contribution from other teams was one of the core requirements of the project, we investigated micro frontends. But it turned out that the flexibility of micro frontends comes with strong coupling and higher operational costs. And big companies like Zalando, who invested a lot into micro frontends in the past, are moving back to a more centralized frontend application.

That’s why we decided on a “hybrid” based approach (teams must provide React components by NPM packages) which offers the opportunity for easy integration of new components. This composition happens on build time to avoid breaking updates of components on runtime.

Spring Boot is still the foundation of many idealo services and frontends. From our experience this approach comes with some drawbacks, especially the integration of modern frontend technologies is hard.

In the last years, idealo frontend teams adopted the JavaScript UI Library React a lot and the expertise in the company is growing. Due to its Client-Side-Rendering (CSR) approach, we had to find a solution to render the start page also on the server (SSR).

So, we gave Next.js a try. Why? It provides SSR and CSR for each integrated (React) component out-of-the-box. Additionally, it comes with a promising static site generation (SSG) feature.

To onboard new components from other teams, we created collaboration documentation. Those teams must develop their components in a separate git repository and provide their components as a NPM package.

To integrate the components, we demand test and mock data in the start page repository for a better local development experience and to run integration tests in our CI/CD pipeline. Additionally, the teams must provide smoke tests and/or contract tests to ensure the API.

Since about 2017 idealo is establishing a Shared Design System. In the early stages, there were plain CSS components — caused by the diverse tech stacks of the idealo frontend teams. With the high adoption of React in the company we started to build shareable React UI components based on Storybook — called “iDS-react”.

In the start page project, we even had the chance to contribute. We introduced the so-called “Product Tile” and “Carousel” components to “iDS-react”.

Backend

For the backend the requirements were simple. Relevant data must be aggregated and stored asynchronously to render the start page as fast as possible.

Apache Kafka streams provide the raw data of all the sub-domains. Our backend — the so-called data provider — consumes the necessary Kafka topics and stores them in a database . To improve the response times of the start page, the data-provider pre-aggregates view models on each change of the raw data.

The data-provider is built upon Spring WebFlux with a full fledge reactive stack. For the database, we chose MongoDB as we had no requirements for complex entity schemas. Due to its capability of non-blocking I/O, it’s the perfect match for the reactive web service. With this non-blocking stack, the data-provider can handle tons of requests at the same time with fewer resources needed compared to conventional web servers.

For the architectural structure of the backend, we chose The Onion Architecture. In a nutshell, this is a very lightweight approach to decouple the domain code from the outer, more infrastructural layers like web or database interactions. We applied this structure as a pattern to the code project, which makes it much easier to find different parts like for example 3rd Party API clients (outer onion layer) or start page-related code (inner/core domain layer). Another advantage of this clear structure is that you can replace individual parts of the system without a big refactoring.

Conclusion

With this migration project, we were able to improve the start page for all stakeholders and their requirements including us as a platform team and, which is most important, for our users.

Due to the modular approach, other teams can contribute with full e2e responsibility for their features. We also reduced complexity by getting rid of a lot of legacy code living in the old monolith. As an example: We simplified the handling of the diverse number of bots, the page is crawled by.

From a DevOps point of view, the comfort also increased. The services are state-of-the-art and cloud (agnostic)-ready. Even if it is running on self-hosted Kubernetes for now, we can lift it into the public cloud with ease in the future. Our trunk-based development approach deploys every commit to production. This leads to lower time to market and a more confident developer experience with e2e responsibility.

And finally, we did this project for the idealo users. With the mentioned technical foundation, we can react to the users’ needs faster. Also, the bounce rate decreased. Our customers also profit from the SEO and page speed requirements we had. Looking at the Google Core Web Vitals, we recognize, that the performance improved very well. Especially the Largest Contentful Paint and the Cumulative Layout Shift got much better.

Those improvements have already led to a better UX. Compared with the ability of faster and easier feature onboarding, the start page is now well prepared for the future, so that we can help our customers even better with their buy decisions.

Do you love agile product development? Have a look at our vacancies.