Using Google PubSub Messaging to Efficiently Allocate Kubernetes Resources

Pipelines that handle a variety of tasks with different degrees of complexity require developers to evaluate resource allocation and tool usage at scale. Migrating to Google’s PubSub messaging service allowed us to intelligently scale and distribute Kubernetes workers.

Bluecore Communicate offers a variety of ways for retailers to communicate with their customers: transactional emails (i.e. order received), traditional marketing emails (i.e. SALE), and Bluecore’s favorite, personalized emails (i.e. abandon cart). Processing a transactional email is quite simple, whereas a personalized email requires more complex steps. Therefore, the time it takes to render an email is specific to the type of email we are sending.

In the past year, Bluecore migrated more than 90% of our personalized-message processing from Google App Engine (GAE) to Google Kubernetes Engine (GKE). Unlike GAE, GKE does not support task queues; instead, we used Google Cloud PubSub. As noted in a previous blog post, “Given that sending millions of emails in a serial fashion would take a very long time and the task to process an individual email can be considered Embarrassingly parallel, we made use of GAE Task Queues to scale sends. Switching to GKE required us to look for an alternative way to do this.” In this blog, we’ll discuss different ideas of how to use PubSub subscriptions to scale your system and ensure your GKE workers are efficiently allocated to each subscription. Instead of emails, let’s use a robotic bakery example where each baked good takes a different amount of time.

Scaling

Let’s say you were creating a system that manages a robotic bakery, which can bake cookies, cakes, and muffins. If your bakery has 10 bakers, then you want to make sure that each of them is being used to their maximum capacity before booting up another baker. A queue system works well here, so we’ll use Google PubSub to help you scale efficiently. PubSub has a concept of topics (where our baked good orders will be placed) and subscriptions, where our bakers will work on order messages pulled from the topic.

For simplicity's sake, each baker can prepare 100 cookies, 50 muffins, or 10 cakes per hour; and the bakers are hourly employbots. If the requested items for every order were put into only one subscription, bakery-backlog, then the subscription only offers insight into how many items you have to bake and not the type of baked good. Therefore, if you received an order for 400 cookies, the system would wake up all 10 bakers. Obviously, this is highly inefficient since 4 bakers would do the trick. For this reason, we need to improve our Horizontal Pod Autoscaler (HPA) — in this example, we need to know how many bakers are needed to complete the job in a reasonable amount of time. To improve our HPA, we need subscriptions that give insight into the items we need to bake, which we can use to determine how many bakers we’ll need. So in this case, we can simply create cookies-backlog, cake-backlog, and muffins-backlog. Our policy is that each of the backlogs only contains one type of order. Now, when 400 cookie orders show up in cookies-backlog, our system knows to wake up 4 bakers. Using a variety of metrics to scale helps specify how many replicas you need based on the subscription; Kubernetes will choose the maximum number of “bakers’’ across the various metrics.

Example YAML to put in our HPA

Allocating Resources

Unfortunately, the story doesn’t end here. As a bakery, we can assume that we will receive large orders, like wholesalers, and smaller orders, like parents planning a birthday party. For the wholesaler, we may need thousands of cakes, whereas the birthday party only needs one or two cakes. How do we determine which order to process next?

FIFO

To begin, let’s use a First-In, First-Out (FIFO) model. If the wholesaler places their order first, then the remaining customers will wait and vice-versa. One benefit of this approach is that we have to do less context-switching, so our bakers can really focus on the client’s needs. However, it also illustrates our inability to multitask, limiting our potential growth. We won’t spend much time on this because it will deter customers from ordering from us.

Prioritize David over Goliath

If we assume that the wholesaler expects it to take us more time to complete their orders, then we could try prioritizing the smaller orders first. The benefits are clear, the parents don’t have to wait for the wholesaler’s order to be processed before their cakes are baked. Once the parents receive their cakes, the bakery can focus on the larger order. Again, we can leverage our subscription labels, this time we will use the labels to determine which subscription the worker should pull the next task from. Something like cakes-high-volume-backlog, cakes-low-volume-backlog, etc. In this situation, the worker will only process tasks from cakes-high-volume-backlog when the cakes-low-volume-backlog is empty. We could set some rules about which backlog an order should be placed on, for example, all orders containing more than 1K cakes go to cakes-high-volume-backlog.

In the above image, we show one subscription for all cakes — so we have insight into the type of baked good, but not the customer who requested it. All cakes on this conveyor belt receive the same allocation of resources to produce each cake.

However, in this next set of images, we have insight into both the types of baked goods and the type of client requesting it, so that the bakers can focus their attention accordingly.

At first, this may seem reasonable because we have fewer wholesalers than ordinary customers so the majority of our customers are prioritized. However, as a successful bakery, we have thousands of small orders that may continuously delay the baking of the wholesaler. This allows the smaller orders to continuously “cut the line” and will likely infuriate the wholesaler.

Shuffle the Subscriptions

The separation of subscriptions with high-volume/low-volume was the right idea because it gave us the ability to create a policy regarding which task to process next, but we need to iterate on our policy to more evenly distribute our resources. As the baker, it is not for us to make a binary decision about which customer or order type is most important. We want to ensure that all of the backlogs are progressing simultaneously. Therefore, we can use NumPy random choice to shuffle the subscriptions with some probability for each subscription. We pass NumPy the subscriptions and probabilities associated with each subscription, then it returns to us a list of the subscriptions in “random” order. The worker then retrieves the next task from the first subscription in that list that has a task.

Each subscription is assigned a probability and then we pass that to numpy.random.choice to determine which subscription the worker (baker) should choose from next.

Summary

By following the process above, we were able to efficiently allocate our bakers to different tasks and complete the baking in the required amount of time. As you can see, distributing your PubSub messages across multiple topics and subscriptions empowers you to scale your system efficiently. Furthermore, assigning probabilities to each subscription and then shuffling the messages ensures that each of the backlogs always receives some attention. We prefer this over separate deploys because it provides us with the ability to customize our application without the overhead of multiple deploys.

If you’re interested in building systems like these, or in general tackling some of our technical challenges around our email platform, data science, or infrastructure, check out our careers page!