From LBB
Published in

From LBB

Building a quantity booking system

How we solved concurrent transactions on a distributed system

The problem

Imagine you’re selling 100 tickets to a rock concert, but somehow 110 people buy those tickets. Sounds like a nightmare handling those 10 irate customers, right?

Yep, that will happen.
Photo by Icons8 team on Unsplash

That’s what the problem was. We have items which sell like hot cakes, and we didn’t want to end up on the other end of the phone call above. Without a thread-safe method to handle concurrent transactions in a distributed system, it was possible to go ahead and buy more items than what were available in stock. I’ll explain how we built a system to have better stock management.

The original architecture

The original flow of the application was like this: A user goes to the mobile app, adds items to the cart, and goes to the checkout flow. Once the user adds in her details, and selects the payment method, and clicks on make payment, she’s redirected to the payment gateway. Here, if we receive a successful response from the gateway, we fulfill the order.

The problem arises if two users enter the gateway at the same time, buying the same item, which has a single quantity left in stock. After the gateway response comes to us, we have two successful transactions ready to be converted into orders. Now we can do either of these two things:

  1. Two orders being placed, and the stock quantity of that item being reduced to -1 :(
  2. One of the orders failing for one of the customers, after she has made the payment, leading to a frustrated customer. >.<

Both these were big problems, and we couldn’t live with either.

Race condition
Photo by Jonathan Chng on Unsplash

The solution

We started brainstorming on what we could do. Since there are multiple machines processing these requests, there’s no single state that all the machines have. We could queue all these requests, but that would just lead to a slow customer experience.

We decided on building a holding system for putting items on hold for the customers entering the payment gateway. So, you’re only allowed to go to the payment gateways, if we have enough stock of that item, essentially holding an item for a customer till she completes the payment.

We needed something to share state between the machines, and decided that Redis could be a good tool for this. From the Redis website:

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

We already use Redis aggressively for caching our API requests(and I love it!). Our solution was to keep a quantity per itemID key in Redis, and reduce it whenever someone went to the payment gateways. If the quantity is less than 0, not allowing the customer to go through to the payment. Since Redis is single threaded, we can be sure that each single Redis call is an atomic operation, and will always return correct data.

We soon realized though, we’d need to execute a set of Redis calls inside a single transaction to be able to successfully hold an item for a customer. Lua scripting came to our rescue. Redis allows executions of scripts written in Lua, which basically makes them one atomic function. So, checking if we have enough stock, to increasing the expiry of the holding period is all done in one single atomic block. This function takes less than couple milliseconds on an average to execute, so we had no worries on performance here.

We kept an expiry of 5 minutes on this key, essentially allowing a holding time for 5 minutes.

So, now if two customers want to buy the same item which has just a single quantity left, and they click on make payment, several things happen:

  1. Customer 1’s call goes to Redis, which decides it can book the item for this customer for 5 mins, and reduces the quantity of the item to 0.
  2. Customer 2’s call goes to Redis, which sees the quantity of the item, and decides it can’t let this user through.
  3. Now, customer 1 can complete the payment and buy the item. In the case they can’t, the Redis key eventually expires after 5 minutes of holding time, and customer 2 can go ahead and go through the payment process.

This gives us consistency in the inventory management backend and a better user experience.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store