ZEvent Place: How we handled 100k+ CCU on a real-time collective canvas

13 min readSep 22, 2022

Each year, Adrien “ZeratoR” Nougaret runs a charity event named Zevent.

The goal of Zevent is to regroup streamers to collect money for charity.

This year, Zevent was back, BUT, with a surprise: Zevent Place.

Zevent Place is a collaborative canvas where everyone who has donated to the charities can draw pixels. It was developed by William Traoré and Alexandre Moghrabi (myself). The project was inspired by the well-known Reddit r/place that was run on the 1st of April this year.

Adrien “ZeratoR” Nougaret, having been a part of the French community during the Pixel war event that took place on the r/place this year, wanted to create a similar project to encourage people to give more tho the charity. There it was born: Zevent Place.

Zevent Place, even if very similar to r/place, was fundamentally different: On r/place, you are able to draw a pixel every 5 minutes, without restrictions; on Zevent Place there are no time restrictions to draw pixels. Instead, you are given 10 credits per 1€ donated to charity, the credits are used to draw pixels on the Canvas.

There is also another difference between r/place and Zevent Place: on Zevent Place, we introduced a new system: The Pixel Upgrade system. This system allows the players to upgrade a pixel instead of coloring it, upgrading the pixel increases the pixel cost to color it.

As an example: if a pixel has a level of 10, the pixel costs 10 credits to color and 11 credits to upgrade, when upgraded, the pixel level is set to 11, and the cost to color it is now 11 instead of 10.

The goal of this system was to provide a way for communities to protect their creations, by rendering the “attack” on their creations costly for the attackers. Using this method we hoped that community creations should be able to survive waves of attacks of other communities, and it worked well during the event!

First step: Designing the place

During the month of May, we were contacted by Adrien “ZeratoR” Nougaret to work on the Zevent Place project.

At first, the project consisted of a simple canvas where you can draw pixels. Very similar to r/place, but with the credits system.

During this first step, we defined the requirements of the project:

It should be able to sustain a massive amount of CCU (we are talking about an event that is reaching more than 400k viewers simultaneously on 60 of the biggest french streamers channels. And they WILL raid the place (raiding meaning to try to draw their own creations, using the collective force of their community), as we will see later on the metrics graphs, this resulted in HUGE spikes of requests when a streamer starts a raid).
The infrastructure should be cost effective: We are working here to make money for charity, not to loose a massive amount of money on a small project.
The board configurations should be dynamically configurable: The color palette can change at any time, the board can be opened/closed if needed, the amount of pixels gained by € should be configurable and the size of the board should be dynamic.
Moderation tools should be available: Moderators are invisible on the grid (cf. their name is not registered on the “last upgrade by” and “colored by” pixel public metadata) they dont loose any credits when placing pixels, and can use a dynamic brush size to color multiple pixels at once.
Updates must be in real time: The users should be able to see the board evolve in real time, without impacting too much the limited bandwidth we had available.
Pixel cost is dynamic (because of the upgrades system) and must not cost more to the user that what he was informed on the UI.

Infrastructure

For the infrastructure part, we made estimations on multiple platforms: AWS, Dedicated servers, Wasabi (for file storage at low cost) etc.

We decided to use dedicated servers from OVH through a Kubernetes Cluster, because of cost implications of the Bandwidth. After our calculations, through AWS, the cost for 3 days have been estimated between 2k and 13k€ (without taxes). So we prefered to rely on fixed pricing offered by OVH dedicated servers, and create our own Kubernetes Cluster using Kubespray.

The complete cluster was using 4 dedicated OVH and 4 VPS (dedicated to ingress controllers) servers through 4 datacenters (for a total computing power of 32 CPU cores (3.4 Ghz each)).

As you can see, we have multiple services in play on this project:

API Cluster: it is the cluster of APIs that we were using, the cluster is connected to Redis through a Kubernetes Service endpoint. It provides a user-facing GraphQL API & WS GraphQL Subscriptions to receive realtime updates.
Diff Worker: The diff worker is the main element in play here: Its role is to retrieve all the pixel color updates pushed on a Redis queue from the API Cluster, aggregate them and merge them to the current board image. It role is also to create either the raw diff data or the diff image (we’ll discuss a little later what are these) and post the aggregated updates through a Redis Topic.
Redis Cluster: a Redis cluster using Sharding, composed of 4 leaders and 2 followers (that are hot-swapped with the leaders if they fail to respond to health-checks).
MinIO Cluster: a MinIO Cluster to provide efficient file storage and serving with low latency.

User actions design

During the development process, we designed the user interactions to be as much racing-condition-proof as we can. We used application locks to provide an efficient way to avoid racing conditions problems and be sure that the user will not spend more that wat was shown on the UI: on each user pixel update/upgrade actions, the current pixel level known by the client was sent.

We decided to use GraphQL through NestJS and Redis to develop the backend of Place.

Initialize the client

When the client connects on Place, it subscribes to the board updates, then it requests the GraphQL API to retrieve the current state for the board, and the last board image.

The flow is fairly simple, Cloudflare is used as a CDN, all CDN misses are redirected directly to our MinIO cluster which will serve the last generated image of the board.

Color a pixel

The pixel coloration process is fairly straight forward: The user chooses a pixel, the cost of the pixel is transmitted through a GraphQL request when the pixel is selected (by clicking on it), then the client sends the “setPixelColor” mutation to the API with the current pixel level.

Here’s a walk-through of the steps involved in a pixel coloration:

The client queries the current pixel data and subscribes to it, this is done each time a user clicks on the board, therefore selecting a pixel, it is used to display metadata of the pixel on the UI: The current level (price), the last person who colored the pixel, and the last person who upgraded the pixel. The client mostly needs the current pixel level, it is the key we use as a soft lock to avoid spending credits that the user didn’t meant to spend.
The API retrieves the pixel metadatas, they are stored in a Redis Hash using x and y position of the pixel in the key.
The data is sent to the client, which will use the current pixel level on the next request.
The client sends its request to color the pixel, including the current level of the pixel that the client is aware of.
The API retrieves the current level of the pixel.
The API compares the retrieved pixel level and the level that the client was aiming for, if they are different, an error is sent to the client, if not, the color request is sent to a queue.
The queue is consumed by the Diff Worker, which will compare and apply all the colors that have been applied on the board. The diff worker will either generate a “Diff Image” or “Raw Diff Data”, this is depending on the size of the pixels to update. If the “Raw Diff Data” is likely to be more than 1Ko, the “Diff Image” is generated, if not, the “Raw Diff Data” is appended to the “boardUpdate” event. The Diff Worker will also generate a complete image of the board each time it is updated. The “Raw Diff Data” is a String format referencing all the pixels changes (only the last change for each pixels), with the following format: x1,y1,color;x2,y2,color;…
The complete image is uploaded to MinIO to provide serving through the CDN.
If the Diff Image exists, it is uploaded to MinIO to provide service through the CDN.
A “boardUpdate” event is sent through the global WebSockets, to send a notification to every listening clients.
Each clients subscribed to the “boardUpdate” event receives the event, then either merges the Diff Image to the current local board image, or uses the Raw Data provided in the event to draw each new pixels using “fillRect” method on the 2D canvas.
If the Diff Image is set, it is retrieved through the CDN.
If the CDN does not have a HIT on the image, it is downloaded through our MinIO cluster.

Upgrade a pixel

Even if the upgrade process looks simpler, it is a little more tricky to manage than the color one, because it deals with highly concurrent data, which can lead to racing conditions. That’s why the pixel is upgraded then un-upgraded if needed.

Walk-through:

Same as the last flow.
Same as the last flow.
Same as the last flow.
The pixel upgrade request is sent to the server with the target level that the client wants for this pixel. It is set to the current level of the pixel + 1.
The pixel level is pre-incremented by 1, this will allow to soft lock the pixel during the upgrade, because of the fact that no event is still published about this pre-increment, the clients will always have the old level stored in their memory, that will force any client that want to update the pixel to send the last level as a target level, which will throw an error on the API.
The API checks that the correct targeted level is reached. If it’s not the case, the pre-increment is cancelled by decrementing the pixel level, then an error is sent to the client.
The user credits are updated depending on the price of the upgrade, if it fails, the same process as earlier occurs: the pixel pre-increment will be cancelled and an error is sent to the client.
The “upgradedBy” field is updated with the current user’s username on the pixel metadatas.
A notification is through the global WebSockets to be sent to all listening clients.
The WebSockets listens to the notification (the “Websocket global notification” is not a real service, it is directly bundled in the APIs, but it is simpler to understand on the graph using this method).
The notification is sent to all the clients that are listening to this pixel metadata updates. This allows to all clients currently listening to this pixel to update their pixel metadata UI info.

Stats

By using these methods, we were able to manage a massive amount of updates (nearly 3,000 pixel color updates per seconds and 1,000 pixel upgrades per seconds during spikes) with a fairly low Bandwidth footprint (with a max bandwidth of 150 Mo/s with the biggest requests spike) while retaining the requirements of the project and maintaining a fairly low CPU profile (the complete project consumed 16 CPU cores at its peak during the event).

Ingress requests per seconds — Backend

As you can see, the requests per seconds highly depends on two factors: the amount of streamers active during the event (for example, at night, there were not much activity) and we can clearly observe the peaks that I was talking earlier, due to the streamers raiding the Zevent Place with their communities.

We can also observe a drop in requests the last day during 40 minutes, this was due to a downtime that occurred due to some unexpected trafic. This downtime is explained in the “A surprise downtime” part of this article.

Ingress requests per seconds — MinIO

At first, Zevent Place used only Diff Images without using the Raw Diff Data field, which resulted to most the updates being provided directly through images. That worked well but we got problems due to a limitation: the native Ratelimit of Cloudflare. We were creating a new diff every 200ms, which means that a new diff image URL is sent to the clients every 200ms, which triggered the Ratelimit system of Cloudflare.

After a hotfix by adding the Raw Diff Data field, there were a lot less diff images generated, which resulted on no more Ratelimit errors.

Open WebSocket Connections

As you can see, we can clearly see the “final rush” moment of the event, which is the moment where all the streamers are gathering together to encourage their audiences to donate more money, it was also a moment with a very massive amount of raids on Zevent Place, which explains the sudden increase on CCU.

CPU Usage — Backend

The CPU usage values are displayed as CPU Time (number of CPU threads used during a 2min period)

Again, we can clearly see here the 40 minutes downtime during the last day of the event

CPU Usage — Ingress controllers (Nginx Kubernetes)

CPU Usage — MinIO

Network I/O pressure

Upload is represented by the yellow line and download by the green one

Redis keys count

Redis was mostly used to store user credits count and pixel metadatas, this graph does not include other cache data (web cache for other Zevent pages, ratelimit data, …), which resulted in 10M+ entries in total.

Redis commands executed per seconds

Sadly we haven’t registered the amount of messages sent per seconds from the WebSocket events. But it should be around 1 message every 200ms for each client connected (1 global message is generated on each diff generation, every 200ms, 5 messages per seconds for each client connected)

Viewers count on the livestreams

A surprise downtime

During the event, we were able to maintain the uptime of Zevent Place to 98.4% for the duration of the event (3 days). Most of the downtime is due to the loss of 3 of our servers during an unexpected very high peak of requests (due to multiple streamers raiding the Zevent Place at the same time with their communities), this resulted in an unexpected 300k+ open connections, and estimated to be at least 250k CCU.

This is clearly shown on the following graph where we clearly see a drop in the amount of requests sent to the backends.

We discovered that the crash resulted from MinIO having too much Disk pressure and consuming 100% of the Disk IO during this period. This was due to the self-healing process of MinIO. We deactivated this process and rebooted some nodes, and everything restarted smoothly.

Lessons learned

Don’t have blind faith in Cloudflare

Before the event, we have runned three load tests during the livestreams of Adrien “ZeratoR” Nougaret, which resulted in +/- 10k CCU users for our tests.

During these load tests, we didn’t had any problems with Cloudflare, the 429 errors (rate limit) that we got during the event where totally unexpected and never seen before, and are not documented in Cloudflare’s documentation.

We were able to fix the problem relatively fast in our case, but this kind of problems could be much trickier to tackle on some other project. Keep in mint that SaaS and PaaS products have to limit their own resources to avoid having too much usage on their clients. So do not forget that if you use these kind of products and you dont have a confirmation that there are no limits (in general, these kind of limitations are configurable for most “entreprise” level offers, but it is generally costly) it is better to be careful.

Don’t forget to also observe your Disk usage

We took for granted that our disks will never fail on I/O usage. That was a massive mistake and we should have prepared alerts for the disk usage metrics, if we were able to get alerted when the disks are getting to their limits, we would have found the problem and fixed it before it results to an outage.

Kubernetes takes a very long time to boot

In reality, during the failure of our disks, the servers have crashed and rebooted fairly quickly. 30 minutes (out of the 40 minutes total of the outage) were due to the restart and reboot (and resync) of the Kubernetes nodes.

Special thanks

First of all, thanks to Adrien “ZeratoR” Nougaret and the Zevent staff for trusting us for this project.

Thanks Reddit for the initial r/place. We have studied their article on the r/place 2017 before starting this project, and we got heavily inspired by their 2022 edition (by retro engineering it during the event) for the Diff images system.

And finally, thanks to all the donors for giving money to the charities that were involved with the Zevent this year, and helping to construct a unique artistic canvas.