Black Friday in an Online Department Store
It’s that time of the year. You know it. You love it and you hate it. It’s the Black Friday season — an activity that used to be one day but now more or less has converted into a crazy month. There are many feelings related to a Black Friday campaign — but let me cut my purpose in this article down to the fact, that running a succesful Black Friday is all about preparations and execution. It’s about letting the busy family into a modern shopping facility and allowing them efficiently to make the majority of their pre-Christmas shopping without too much fuzz.
It wasn’t always like that. As you probably know, the Black Friday tradition originates in the idea that the family would go shopping for good deals on the day after Thanksgiving. It turned into a big event and we have all seen the videos of people fighting on the floor over that last flatpanel TV or tearing the last pair of sneakers out of the hands of someone else. Fortunately, it does not have to be like that any longer. With modern scalable technology it can all be done quietly at home in the sofa with the family around.
The first year we in Boozt were confronted with Black Friday we didn’t really believe it would take off. It did. It was way bigger than we thought, and it put new limits to what we in the tech team would have to provide to keep it being a convenient shopping experience.
Fact is, the online Department Store experience is perfect for the challenge that Black Friday put on all processes. You have all the good offers at hand across all categories and you don’t have to fight for the items — although you may have to hurry up to get it in time. We have earlier called Black Friday for “The Superbowl of E-commerce”, and we will in the article take you through some of the preparations and challenges we phase as a tech team during the BF period.
Winning the Superbowl
Winning the Superbowl takes talent — of course — and it takes a lot of preparations. But the main thing to win the Superbowl is to have a good team. We all have to be well prepared, know our role, and how to collaborate. This part we do really really well at Boozt.
Months before Black Friday we start up the “war room” mentality and start distributing the plans between all teams:
- Buying & Merchandising
Goes through the stock composition in all (sub-)categories and plans the offers. They also make additional buying plans where we expect to be able to make extraordinary good offers. - Marketing
Makes a detailed marketing plan for both Boozt and Booztlet, where the campaigns start earlier in November. All timings regarding communication per channel is shared with the other teams, so all are alert when the pressure comes. Visual material is prepared in good time to be able to execute the campaigns with flexibility where needed. There is a special close collaboration with Buying & Merchandising to identify and promote the really good deals. - Customer Service
Customer Service is our main contact to the consumers throughout the entire Black Season. They help and support the customer before, during, and after the sales, and they are also the best channel for catching potential issues on the web shops. - Warehouse and Supply Chain
This is where it gets physical; where we actually touch the items that we sell. To be able to handle the extreme loads we have, we go into a special “Black Friday mode”, where some processes (like the photo process is paused or run on non-peak times). Most of the operator ports in the three Autostore cubes we have are used for “Pick&Pack” process so we really optimise the output of the orders. Having the biggest Autostore in the world bring lots of automation power to an online department store like Boozt, but all processes have to be scalable to keep the performance we want to offer our customers.
Platform Team Preparations
Yes, we do also prepare in the Platform Team . Actually, Black Friday is typically the period where we raise the bar for next year’s top performance level (read more from last year here). We know by history, that this is where we get the most traffic, but we also know that this is where our customers are most engaged with us: they visit the full department store — all categories. They click on more products, they use more filters and search, put more things in the basket, and they often place several orders — not just one. There is also a tendency to much higher peaks in server load than during normal campaign events where the traffic is floating in more steadily. This is mainly due to much higher responses to the notifications we send out; our customers are really looking for the best offers in all our categories.
Load Testing
The difference in the behaviour was one of the reasons why we chose to build our own performance load testing tool (described in another post here). So, in the months up to Black Friday we make a number of load testing scenarios where we put extra load on the servers on top of the normal traffic we get. By monitoring this, we have good knowledge on how servers and applications respond to the irregular behaviour, and we can improve application and infrastructure step wisely and try again. With the approach of the repetitive improvements, we know that we can handle at least 400k server hits per minute, which corresponds to around 50k users simultaneously.
The Pre-Mortem
One important exercise before Black Friday is a guessing challenge of “WHAT CAN POSSIBLY GO WRONG?”, and then make a plan for getting around it. This is the case for all critical internal applications, integration between these and all external systems we are dependent on (mainly payment systems and distributors). This forms the Pre-Mortem guide with potential failure events, criticality, and fixes to get around it. Many highly skilled engineers are engaged in helping out with this which means that the SRE team has a great tool to use in case something critical occurs during the busy days — and nights.
Scaling the Infrastructure
A couple of years ago, we moved our entire infrastructure to Google Cloud from a hybrid setup with physical servers and a private cloud. We have of course had to rewrite a lot of the applications to be more cloud native, but it has paid off many times. The cloud instances scales up automatically as traffic increases. Together with the traffic and load management provided to us by Cloudflare is gives us an amazing power to scale with the traffic and the on site behaviour we experience when everything heats up. Before Black Friday, we scale up to 20 concurrent servers for Boozt and 6 for Booztlet. From the load testing we have run, we know that this will take us a long way into the busy hours where we then can scale up further.
Warehouse Technology
We also run the technology used at the warehouse, and as mentioned above we top tune the processes around the Autostore so we keep the high performance in this period. Therefore, we have introduced three major improvements just before Black Friday: 1) improved our refill processes to store what is needed in the Autostore boxes — but not more. Thereby, we can do the majority of the picking from the Autostore ports and keep a high performance, 2) the option to balance the order distribution between all three Autostore cubes to keep a high output without waiting for robots, 3) doing the distributor sorting directly at the packing ports cut one complicated step in the proces. If it sounds complicated — it’s because it is complicated.
The Game Starts
So, all the preparations are done and that takes us half way there. Execution of the Black Friday really start at 18:00 on Thursday. Thats where we release the official campaigns on site and send out the notification over app as well as the emails. So, how many visitors can we attract to the Online Department Store, how many can we convince to buy and how is the shopping experience both before, during, and after the sale.
Visibility and Eyeballs
We ran campaigns — big time. We did it online and offline (like on TV’s, billboards, radio, bus posters, etc.) — to attract both new customers and activate existing. Marketing works with “Number of Eyeballs” as a KPI — that means how many times the Boozt brand or a campaign have been seen by a potential customer. Just to share some number with you:
- Eyeballs: 1.000.000.000 views across all the different marketing channels. Yes, it’s one trillion.
- New customers: 100.000 customers shopped with us for the first time. We hope they will stay.
- Overall we produced and used marketing material in 1.100 different formats across 10 different languages. It paid off.
Commercial
We are a commercial company. We live by selling items — not only by people visiting the department store. They do their shopping with us if we have the right products, if it’s a good offer, and if the convenience is good. That all ends up in the conversion rate — that is one of the essential KPI’s for an online shop. And if the overall experience is good they come back.
- The conversion rate went considerable up compared to last year. But so did the average order size, which increased with 18% better and 15% more items where in the order when the customer checked out.
- In a tough season where most of our competitors either backed down or were happy with keeping last years results we managed to grow considerably compared to last year. Much more than we had budgeted with.
- Number of customers shopping across multiple departments went up with 45% compared to last year. The department store really took off this year.
Infrastructure Stats
We were well prepared with the infrastructure when we launched the Black Friday campaigns. Not only for the web shop but also for all the internal systems that also experience a drastic increase in load on these days. There are more orders, more customers, and more data being pushed around between the systems. And both customers and business users are definitely less patient than in normal circumstances 😊 We have previous years, we have had situation where we were hit by unexpected side effects, like direct database queries from an internal sales dashboard that took down the master database. That did not happen this year. The Pre-Mortem was used once when things seemed to heat up, but it was balanced well and our infrastructure performed flawless on the peak situation where we had many thousand very active visitors as well as in the longer streams of a very heavy load.
- We started with a cloud configuration of 20 preheated servers on Boozt and 6 servers on Booztlet. This was scaled up to a maximum of 48 servers for Boozt and an average of 25 servers throughout the days around Black Friday.
- Throughout the week, we registered more than 50M page views — and on Black Friday alone we had around 50M requests to the servers within one hour (each page visit can contain hundreds of server requests).
- Replication lag from the master to the slave databases was low (few seconds) most of the time. Only in one situation, it grew to around 2 minutes but would then recover quickly. This has been the best peak performance for our databases in peak situations ever. The good preparation work had helped.
Warehouse Stats
At the warehouse is where the physical part of the online department store takes place. We change the processes so all operations are fully optimised for getting out the orders that are quickly piling up. As mentioned, we have the biggest AutoStore installation in the world with almost 1 mio boxes spread in three logistic cubes and a total of 1.300 robots moving the boxes to the packing ports where they are combined and bagged.
- To optimise this flow we change the usual routines and configure up to 150 ports for handling orders. For simplicity we try to distribute the order handling so all items from one cube. Otherwise we have to consolidate the items picked in several cubes, which adds complexity.
- With this optimisation we were able to handle almost 200k items in one day. This record was broken three times during Black Friday.
- Most of the move of the items are done by the robots. In total during the 5 days around Black Friday, the robots moves a total distance of 143.000 km — representing the same distance as 3.5 times around the world.
Web Shop Performance
So the campaigns worked — the customers came and the servers were stable and well performing. So let’s look at how the experience was for the consumer. Some of the best definitions for that are the Core web vitals that you can read more about here: https://developers.cloudflare.com/analytics/web-analytics/understanding-web-analytics/core-web-vitals
- First Contentful Paint (the perceived speed): performance was good in 83% of the time and poor in 4% of the time. This corresponds to normal performance statistics for the Boozt web shop.
- First Input Delay (user response performance): this KPI was good in 96% of the time and only poor in 1% of the time corresponding (or maybe even a little better) than the rest of the month as shown on the graph below.
- Cumulative Layout Shift (represent the visual stability of the rendering): strong performance by the web shop as well with 91% good rating and 5% poor performance rating.
- Another great KPI is the App engagement time, which actually went up from 4 to 5 1/2 minute. So, the customer spent more time on the app during Black Friday and still the crash rate for the app was on the same level as normally (<0.2%). It worked well.
Summary
In my view, we won the Superbowl this year. At least if we were competing with ourself during the previous years. Probably also, when comparing to competitors and especially comparing to dedicated niche shops, which are focused on one category only. Commercially, we saw the online department store strategy taking off this year — customers shopping more categories and putting more items in the basket.
Execution of the Black Friday season has been a success for our culture. Tremendous collaboration across all teams whether you work with products, customers, campaigns or technology this is the peak of the year. The cliche that no chain is stronger than the weakest link comes to show these days.
So, it’s a pleasure to represent the tech team, delivering technology for the colleagues and the customers in this phase. Knowing that proud colleagues look at the result of their work and happy customers right now are picking up the last packages shopped on Black Friday. Maybe these are even their first Christmas presents.
In Boozt, we are on a journey to build the best online department store in the Nordics. Everything we have seen in the past week proves that this is the right journey. We have just started.
