Report Generation, but different
Have you ever been tasked with adding a report into your product?
Most software developers find themselves coping with similar challenges during their career. So did our team. Our Java legacy-code product had an existing solution based on Jasper reports engine. Jasper uses “server-side” templates, data sources and configurations, which become a rendered visual report.
Our report required a dynamic structure (data-driven expandable sections, different linguistic alignments, iFrames from external sources such as Google Maps and more) which made the pre-defined template concept challenging. On top of not satisfying these new requirements, it turned out the existing template-based solution was hard to work with and slowed down development of new reports which made our managers open to investing some effort designing a better solution.
So we went on a journey to find an easier solution to our problem.
What is a report?
Real-time web-based dashboards usually give the best visibility of the information produced by a service. Yet, a report as a signed snapshot of a certain moment still has its virtue. In fact, customers demand this kind of time-freezing perspective of the constantly changing data in your application. We tried to find a way to gain the most from the web technologies and yet package it as a report.
Chrome, but different
We looked for a way to freeze an HTML document and package it as a report. Chrome is the most common web browser in the world (and in our team :). We all have it installed on our local development machines, and we knew it is possible to ‘export’ a rich HTML document as PDF using the browser. From that point, we quickly ended up playing with Headless Chrome. A headless browser is a web browser without a graphical user interface and is mostly used for WEB-UI testing purposes. We found it pretty handy to use Headless Chrome as a back-end process to render PDF files based on our web application pages.
The chosen solution
We decided to use an asynchronous interaction using a queue as the service’s input and output APIs. Our service will manage the report generation process using an RDBMS, consume report requests from the input queue, and trigger the report’s web page loading using Headless Chrome. Then we’ll use Chrome’s API to produce a static PDF from the final form of the report’s Document Object Model. Finally, the service will collect this PDF, upload it to a shared file system, and publish the result to the output queue.
The architecture
Below is a sketch blueprint of our architecture that serves the needs of the report we had to build. The main flow goes like this:
- A client application submits a report-request-message to an input queue
- Report Generator Service (AKA RG) manages the request’s state in an RDBMS
- RG activates a local Headless Chrome with the relevant path which hosts the report (a React document)
- The React document fetches data from other microservices via RG Proxy Service
- The generated PDF is uploaded to a remote files storage (e.g. AWS S3)
- RG Service submits a result response with a PDF link, to an output queue
- The client application consumes these messages and downloads the PDF files
Challenges
If you’ve been following along, you’re probably thinking, this is too good to be true.
But there were lots of challenges on the way. We had to build a docker image with an executable, Headless Chrome. We had to find the best set of parameters to make Chrome produce the best PDF and optimize Chrome to work with less memory and disk space to reduce file sizes. We had to find ways to make the data fetching feasible and secure, tune the cluster to scale to handle high volume of traffic, and had to send the finished report to clients securely.
Let’s discuss some of these challenges:
Challenge #1 — Slightly different Docker image
Our company has a cool platform for dockerized microservices construction. To allow us to execute the Headless Chrome system process from our Java code, we had to build a Docker image with a pre-installed Chrome embedded in it. We’ve placed an installation package on a shared location in our organization network, and we used it to add a Docker layer as follows:
By this installation the Chrome execution path becomes “/opt/google/chrome”, and we could use java.lang.ProcessBuilder to control the execution of the headless Chrome.
Challenge #2 — Getting the timing just right
The most challenging issue we encountered was the timing of rendering. Since our reports allow asynchronous data fetching from external sources, it is a real challenge to identify the exact moment when the web page “loading” has ended.
Even if we settle for a constant value — i.e. introducing some constant delay of N milliseconds before allowing Chrome to take a snapshot of the DOM — it wasn’t trivial to find the right way to make Chrome adhere to that value. After some Googling and many dry runs, we’ve found the parameter — virtual-time-budget which, as we understood it, roughly represented what we need. This helped resolve the issue, although it seems that there is some sort of algorithm to shorten this time when possible, otherwise we could not explain PDF rendering speeds we experience (faster than the virtual-time-budget).
Challenge #3 — Error Handling
As mentioned above, we changed the traditional way of preparing data for visual reports. In engines like Jasper, data fetching usually occurs in the preprocessing phase, and the rendering phase gets the data ready for presentation.
Because we chose a web-based approach, the data fetching happens as part of the rendering phase, but that means that failures in fetching the data must be addressed or else we’d simply produce partial or empty reports. Therefore, we added a mechanism (illustrated below) that allows the client layer (in this case — the React doc) to report log events and even throw an exception (b) that will halt the report generation process on both server & client side. In case the data fetching completed successfully, Chrome will have the up-to-date DOM (a) ready for PDF rendering.
Challenge #4 — Publishing the result
When a report generation has completed, and a PDF file is ready, we upload it to a remote shared file system. From that point we had to support two key use cases: downloading the report file, and sending it by email. The first use case was trivial, and we used AWS S3 SDK to solve it. For the second use case, we used an existing proprietary transactional email microservice. In order to send our report’s PDF file as an attachment, we had to make some changes in the microservice. We “taught” this microservice to import the file from the shared file system and then add it as a multipart content of a mime message:
Challenge #5 — Taming Chrome
Google Chrome is sometimes associated with resource hogging. Building a stable service that will support our needs and would be able to handle diverse rates of calls required some additional tuning.
We have used a messaging queue as our entry point and configured it to allow consuming of a single message per service’s instance at any given moment. Each node’s Headless Chrome is now performing a single operation at a time. To accommodate scale, we scale-out our cluster by adding/removing nodes based on the queue size.
Our service’s cluster is ready for the ride.
If you wanna snapshot — snapshot, don’t talk
Finally, let’s take a look at some of the results of our new mechanism. The structure of the report below is recursive in the sense that it fetches 2 sub-structures of the same type, so you can see “duplicate” sections on the 1st and 3rd pages with relevant data per each instance of the sub-structure. We also draw maps by dynamic data fetched from other services. This snapshot is actually three PDF pages of the same report.
Some more colorful examples:
Closing thoughts
Being able to implement a non-trivial task using tools that we use every day in both our professional and personal lives is a refreshing experience. When developers cope with a new technical problem, they occasionally tend to seek new technologies to solve them. Though I can’t guarantee that using Chrome was the easiest way, I can surely say it was interesting and a learning moment.
Thanks for joining our ride.
