Fighting for bytes in the frontend

On tempering greedy drawn widgets and saving iPads from running out of memory

Performance is a very hot topic in the front-end community: as products, applications, and sites grow in complexity, they exchange large amounts of data to support beautiful graphics and online interactions among many users. The pandemic accelerated the digitalization of offline processes. Therefore, we need to upgrade and optimize current solutions to handle increasing loads.

For Miro, 2020 was the year of performance optimization. More users interacting on a board means that they generate more content, and their web browsers receive, process, and display more data. In real time.

Intro

I’m part of the Canvas Widgets team: we develop widgets, so that users can interact with boards.

Some time ago, we received product feedback from users. One thing was extremely clear: when a board contained many widgets drawn with the pen tool on an iPad device, the interface elements would start to blink, then they would disappear, and it wouldn’t be possible to use the board any longer.

Let’s have a look at the statistics:

• 130 tickets logged between January and October 2020; about 13 tickets per month.
• 35% of all iPad-related remarks contain negative feedback that describes this problem.
• 89% of all widgets created on iPads in a month are drawn with the pen tool.
• Widgets drawn with the pen tool are the second most used widgets on iPad devices.

It didn’t take long to form a hypothesis: some iPad models didn’t have enough memory.

We started our research. The first thing that came to mind was level of detail abstraction systems.

LoD (Level of Detail) defines caching at the rendering level. A sequence of images replaces the original content of the widget at specific zoom levels. Each image has a size that is a reduced copy of the previous image by a factor of 2.

LoDs are stored in the memory. They are turned on at certain points in time: as a rule, when the actual size of the widget on the screen is reduced relative to its specified size. Each drawn widget has one image for iPad devices, and two images for desktop and web.

It looks like we found a potential cause. However, turning off LoDs would cause FPS to drop when working on the board, and we don’t want that.

First solution: vector LoDs

When a Bézier curve is drawn, an image is created, and then it is displayed instead of the widget, based on a set of rules.

What if instead of an image, and instead of drawing a Bézier curve, we only connected the points of the curve? The curve wouldn’t be displayed as a smooth line, but it wouldn’t be noticeable to users.

We named this solution “vector LoDs”, we built a prototype, and then measured the impact:

To test vector LoDs, we drew 4000 curves of the same type on a board. This is what we observed:

1. Vector LoDs used ~256 MB of memory.
2. Raster LoDs (images) used ~316 MB of memory.

In total, we reduced memory consumption by ~20% — a good start, but not there yet.

We realized that this wouldn’t significantly help users, so we kept searching for better optimization points.

Second solution: curve merge

When you release the mouse button while using the pen tool, the action creates a new widget. Each widget has data such as points, styles, scale, sizes, and position.

If you use the pen tool to write text, in most cases the style of the widgets rarely changes. Therefore, it’s not optimal to create a new widget each time the mouse button is released. If you look at the screenshot below, why should the dots be separate widgets, instead of being part of the characters they belong to?

It should be possible to create a new widget based on criteria such as the amount of time since creating the previous one, or distance on the board from the last created curve. This approach reduces the amount of metadata for each curve.

We named this solution curve merge, we built a prototype, and then we measured the impact:

To test curve merge, we drew 1000 squares with the same number of points on a board. We drew the sides of the square as separate curves, which produced a widget for each side.

1. The version with separate widgets used ~160 MB of memory.
2. Combining widgets used ~105 MB of memory.

In total, we reduced memory consumption by ~35%.

By combining curve merge with LoD optimization, we cut memory consumption by ~50% — Now, this is something, isn’t it?

So far we prototyped possible solutions to test our hypothesis. And if LoD optimization was transparent — it was easy to assess, execute, and push the changes to release — curve merge wasn’t as straightforward. For example, we immediately had questions about the UX:

• Do we need to move segments separately? If so, how?
• How can we define and set a widget creation timeout?
• How should we delete segments: separately, or all at once?
• How should the widget be shown to board collaborators: immediately after creation, or segment by segment every time the mouse button is released?

These UX questions spawned a number of technical questions:

• How should history record the action: as 1 aggregating step, or should it record each segment as a separate step?
• How should we store data: as a one-dimensional array with separator, as a two-dimensional array, or differently?

At the time we didn’t have resources to take up UX development, so we saved the analytics results, the prototypes, and we started to look for a quicker way to solve high memory consumption.

Third solution: changing data structure

Each point of a curve is stored in the memory as an object of the drawing primitive that the PIXI.js framework provides; namely, the PIXI.Point object. In turn, each widget has 2 arrays:

• points: PIXI.Point[] — an array of curve points.
• controlPoints: PIXI.Points[][] — a two-dimensional array of control points to draw Bézier curves.

To test this solution, we drew 4000 curves on a board. Each contained ~100 points. Then, we examined the memory snapshots:

Total memory allocated to the board (heap): 165 MB.

The storage of one point as a PIXI.Point object is allocated as follows:

1. 20 bytes — the weight of the object itself (shallow size).
2. 88 bytes — the weight of objects that are kept current, and that the garbage collector cannot collect (retained size).

Since we store data of the same type, and we can calculate in advance the required amount of memory to store the points of the curve (64-bit floating point numbers), in this case we can use typed arrays.

Typed arrays are array-like objects that enable working with binary data. While regular arrays can grow, shrink, and store any type of data, typed arrays allow storing numbers from int8 to float64, and they allocate the required amount of memory in advance.

You can read about how typed arrays work here. In short, their architecture has 2 main components:

1. buffer — a binary data storage.
2. view — a context that enables interpreting buffer data as a specific type, such as int8, uint8, int16, uint16, and so on.

Time to test the hypothesis that this data storage format can save a few extra MBs of memory, and to teach our code to work with typed arrays.

Each curve widget stores points in the following format:

Each point is 2 float64; therefore, we will store all points as Float64Array. Storing one point takes 16 bytes.

For the sake of working with the points of the curve, we abstract the widget code from storing, creating, and accessing data; then, we create a convenient interface to interact with an array:

This interface is implemented by two classes:

1. Points — an array of the main points of the curve.
2. ControlPoints — a two-dimensional array of control points for rendering (client-generated).

The final size is calculated as follows:

pointsCount * POINT_ELEMENTS_COUNT * Float64Array.BYTES_PER_ELEMENT

• pointsCount — the number of points on the curve.
• POINT_ELEMENTS_COUNT — the number of elements (numbers) to store a point (by default, 2).
• Float64Array.BYTES_PER_ELEMENT — the number of bytes per array element (8 bytes).

The main difference between Points and ControlPoints is that ControlPoints accepts and outputs data in a two-dimensional array format:

// 'in/out':
[[p1, p2] [p3], [p4, p5]]
// 'store as':
[p1.x, p1.y, p2.x, p2.y, p3.x, p3.y, p4.x, p4.y, p5.x, p5.y]

We carried out our tests on the same board as before.

The final snapshot of the heap showed the following results:

Total memory allocated (heap): 130 MB.

As for the amount of memory allocated to storing curve point data as typed arrays:

By changing the way to store points from arrays of PIXI.Point objects to typed arrays, we reduced their memory consumption by a factor of 3. Since data is now stored in typed arrays, the number of arrays has also decreased and, therefore, the memory allocated to them.

In total, we reduced memory consumption by ~20%.

Output

As a result, we implemented two small optimizations:

1. Replaced raster LoDs with vector ones.
2. Changed the way of storing curve data in the memory.

Together, these optimizations reduced memory consumption by ~40%.

A few months after implementing them, we observed a more positive picture:

• 15 tickets logged in 2 months; ~7 tickets per month, which is half the amount we had when we started our investigation.
• The number of alerts that are sent when the memory limit is reached dropped from 21K to 15K per month.

Can we now confidently state that the problem will never occur again? No, but if it does, it will be less of a hindrance to users, and they will be able to continue working. This also creates a springboard to further develop widgets drawn with the pen tool.

Join our team!

Would you like to be an Engineer at Miro? Check out opportunities to join the Engineering team.