The story of Frontend at Feedzai
Since the first release, in 2011, our product had a Frontend component.
At the time, the Frontend consisted of 4 jQuery plugins that connected to the server and allowed to receive data in realtime from data processing streams. These plugins were called jPulse and allowed not only to receive the data but to plot it in line charts, column charts, pie charts and tables.
Below you can see how we would initialize a plot connected to a data stream and the corresponding chart that was rendered.
var dataseries = {
stream : 'transactions',
field : 'duration'
};
$P.plot( '#mydiv',[dataseries] );
At that time, we didn’t have any kind of Frontend tests, we didn’t use any package manager, and we had about a dozen files.
As time passed, we had to implement several requirements and, at some point, we forked the chart library that we were using (flot) to which we gave the name Flotzai. And that became a huge nightmare. Flotzai’s main file had around 4000 lines, lots of global variables, variables whose name had one or two letters and no tests! And there were other files that started to grow considerably (more than 1K lines). So, touching this code became a huge pain.
Shortly after, we started what was, at the time, my master internship and thesis: Pulse Views. The generic name was on purpose, it was given so that we could make this web application whatever we wanted. It started as a dashboard building tool (to avoid having to program them by coding) and, over time, it would become something entirely different.
Pulse Views started being developed only using jQuery but, after a short time, we decided to migrate to Backbone.js. Backbone.js allowed us to structure the application in a much better way, by allowing to easily create reasonably reusable components that were reasonably self-contained and extensible. In theory, this was possible with the jQuery API but using Backbone.js it was much easier and straightforward.
Before the first release of Pulse Views, in 2013, we were able to migrate the project to Require.js (a module system), which gave us another big improvement in terms of maintainability and organization. And we were able to have some unit tests, a linter (jslint, although we weren’t able to clean all the linter errors) and some end-to-end selenium tests, courtesy of our QA team.
At the time, using libraries such as Backbone.js, Require.js, and having unit tests was revolutionary for Frontend at Feedzai.
In 2014, we implemented internationalization (to Portuguese). The application already had around 2000 HTML templates and it was necessary to manually change each one of them and translate more than 3000 UI strings!
Besides internationalization, we also implemented localization (render dates and numbers according to the local format). The biggest challenge here was to integrate Google’s Closure library into our project. At that time, we didn’t use NPM and package management worked based of references in the global scope. In order to avoid those global references and to integrate with Require.js, we had a tool that would wrap libraries with compatibility code. But oftentimes, we had to adapt this wrapper for a specific library. It was far from being an ideal solution. Fortunately, we now have tools like NPM (although we weren’t able to get rid of the wrapping tool completely yet).
By the end of 2015, we started introducing React in Pulse Views. The goal was to be able to implement complex components more easily and to not be stuck in a deprecated technology.
In the first release where we used React, we only used it in a progress component that was quite complex. It was necessary to re-render several parts of the UI, and this was something in which React is much better than Backbone.js (where you either re-render everything in a View or you need to implement the imperative logic to change the HTML elements).
It was also in this year that we launched our first version of our Data Science product, a part of Pulse Views dedicated to the creation of machine learning models. We used this opportunity to continue our adoption of React.js, and we implemented new components using it as much as possible.
This was the year in which we integrated NPM in the project for the first time. This initial integration was only for tooling reasons: running the linter and some CSS tools.
Around this time, and as the Frontend team started to grow and started to be split into several products, we felt the need of having a place to get together and think about our common problems. To address this, we started doing internal meetups with the several Frontend engineers(that were spread by several teams).
In 2016, we had almost 200 React components and 878 JS modules in Pulse Views! It was in that year that we started our second Frontend project: Case Manager. It uses a modern React and Redux stack, which we choose because we were very happy with using React in Pulse Views and because Redux was the most mature state management library at the time.
The first version of the project had already 171 JS modules. At the same time, we created the UI Kit, our internal library of UI components, initially with 20 components.
In 2017, we upgraded our build stack from RequireJS to Webpack and Babel. RequireJS was implemented at a time where there were no standard for JS modules and it no longer made sense since there was a standard module format.
This was a huge change, that required changing all the +1000 modules from the AMD format to the native one. Fortunately, we didn’t had to do it manually! We used a script that converted the Javascript code automatically and we just had to deal with a few edge cases manually.
This change also removed an indentation level from all the code, so we had to do a commit that touched all the lines, which is something to avoid to keep a clean git history, but in this case it had to be done.
With this change, we were able to use new ES6 features (that, until that time, we couldn’t use because we didn’t have a transpiler), we no longer needed to load hundreds of files in development (RequireJS load files individually) and we were able to reduce the production compilation time (using Parallel Webpack).
There were downsides as well: the application became harder to debug because source maps didn’t work well in some cases and even less with an application as large as ours.
We also used this opportunity to move as many dependencies as possible to NPM. We went from 44 local dependencies (i.e., without package management) to 19 NPM dependencies and 26 local ones.
This migration was a huge headache. Besides having to migrate a large number of modules, we hit several difficult problems and we had to do some compromises. One of those was to keep RequireJS in the bundle, given that it was used to allow dynamic imports and that we hadn’t time to implement that in a better way. It was also necessary to implement a custom Webpack loader to be able to load soy templates and to fork an existing one to do the Require.js integration.
At the time, we had 227 React components in Pulse Views.
At the end of 2017, we also migrated from the Backbone.js Router to the React Router. The main reasons for this migration were:
- Improve the developer experience. Our old routing architecture was based in implicit behaviors and in a hierarchical structure. This made finding the file corresponding to a UI URL difficult, more so for new hires.
- Migrate the application shell to React, since we wanted to migrate Pulse Views to React.
Given that all the pages were in Backbone.js, we had to implement some wrappers in React to the different types of pages but, in the end, the solution was quite solid. Currently we have all the routes organised in a single folder which acts as an index of the code.
There were some things that we weren’t able to get to, namely:
- We had to keep the Backbone.js Router because it was still necessary in some cases that we hadn’t time to migrate.
- We didn’t had time to migrate the top level views to React.
This year, we started a new project: Genome.
This is a data visualization project that renders thousands of transactions (e.g., the transactions in an online merchant) and their links. The objective of this application is to detect fraud patterns.
This is a React/Redux application that, due to data visualization requirements, also uses other technologies like D3 and Canvas.
Due to the challenge of rendering so much information on the screen at the same time, we had to a lot of performance testing and optimization, and this will be the theme of a future post on this blog.
After all these years of development, the main lessons that we learned were:
- It’s very important to attack technical debt early on.
- We should strive to remove unused features.
- Have unit tests and linter passing from day zero.
- Invest in big and necessary changes that improve maintainability and developer experience.
And, in the future, we would like to:
- Be a React shop (and migrate Pulse Views entirely to React).
- Reuse more UI components between projects.
- Improve the efficiency of our Frontend engineers.
To wrap up, we will present some statistics of the evolution of the code of the projects.
