Load testing to prepare for Black Friday and the holiday season
By Jonny Johansson, Data Systems Director
Black Friday is a big event here at Boozt and we normally dedicate 3 months leading up to Black Friday for preparations and have multiple dedicated teams assigned. As with many IT systems, our Platform is good at automatically scaling for the load we see during normal campaigns and activities. But Black Friday is very different. We can see huge spikes, sometimes 10X more traffic than during normal campaign days, related to emails, SMS or push notifications, or that people just wait until 00:00 to enter the site for checking the best offers. To make sure we are ready for the spikes, Load Testing is an important part of the preparations we do.
Goal
The goal with Load Testing is to prepare ourselves for the expected load during Black Friday and Holiday Season, and of course to have some extra buffer. As with all systems, we need to balance the expected load with costs and resources. For example, we could spin up lots of Databases and Compute Instances, but it will not help if the application can’t handle the load — and cost-wise it will just be expensive.
Testing customer journeys and running load tests
We use several tools and processes to perform Load Testing. One of those things is simulating actual user behaviour, but intensified — searching/filtering for products, adding them to the cart, going to the checkout, etc. When we pre-configure a customer journey it can look something like the flow below;
→ Enter Home Page → Go to Women Dresses → Select 2 Random Filters → Place 2 random items in the cart → Go to Checkout → Set shipping and payment method.
This customer journey is then set up based on our configuration to run multiple times. One scenario can be that we start small with 500 users and ramp up with 500 every 5 minutes — resulting in 1000 users after 5 min, 1500 after 10, etc. We can then scale up the number of users performing the journey simultaneously until we hit a predetermined threshold or until the page starts to perform slow or in the worst case is unresponsive. Our internal KPI is to handle the minimum of 2X the expected traffic during Black Friday, meaning the Load testing will test for much higher traffic than the one we are expecting to see during the event.
Production vs Stage
We have chosen to perform Load Testing on the Production environment. We could of course do it on stage servers, but if we can’t handle traffic and load during testing, how could we expect to handle it during Black Friday?
Learnings
During and after each test we analyse logs and monitor both infrastructure and applications carefully. This will find bottlenecks in our application and infrastructure. For each area where we see that we could improve something we create tickets and distribute this to assigned teams. We have done load testing during the last couple of years and each year we learn new stuff.
Some typical improvement areas:
- Slow End-Points: End-points that have long response times, maybe due to misconfigured cache or heavy business logic.
- Locking queries: When load is heavy and multiple processes want to write/edit the same information there could be lock on Database level.
- Infrastructure: Reaching Max connections, slow startup times, reaching network throughput.
After a test run, we normally spend some time improving based on the findings, then we schedule new tests and make sure our improvements are working as expected so that we are all set to handle load during peak times like Black Friday and the holiday season.
Some of the other Black Friday preparations we do:
Front-End Performance
We constantly, but even more before Black Friday, make sure that our Site is fast and responsive for our users. We measure several KPIs and make sure that users get as good of an experience as possible.
3rd Party Dependencies.
We have several 3rd party dependencies like distributors, payment gateways, etc. We work on having a backup for each of them and to have a quick or automatic failover in case there are any issues on their end.
Code freeze
One week before Black Friday we have a code freeze, which means that we don’t allow new features to be pushed to Production. This to avoid mistakes and bugs.
To celebrate the big event this year and get everyone in the team spirit we had special edition t-shirts made for everyone in the Platform team.
If you enjoyed this article and want to read more great stories from the Boozt platform team, be sure to subscribe to the Boozt Tech publication!
Or perhaps you’re interested in joining our platform team? Then check out our careers page.
