How we made new PizzaPortal mobile website
PizzaPortal is biggest polish food ordering service – our local Grubhub. When taking up the role of the chief of product there, one of my first tasks was to verify the status of the current sites and applications, and to prepare a plan for their development.
I decided to start from creating a completely new mobile site. It was then (still) a relatively small in percentage sales channel so it allowed for bolder experiments – even if the final result were poor, the impact on the bottom line would be possible to swallow. On the other hand, I had a conviction that mobile web is the key to unlock the potential of this part of our, still developing market.
In mid June 2015 I took over the control over the development of a new mobile website, setting its creation as the basic priority for 2015. 21 December, that is almost six months later, the service publicly started and after a month it generates 2.5–3 times more orders than the previous version. You can read below how it happened.
Situation at hand
The very concept of a new mobile site appeared in the organization even before I came. However, the lack of clear understanding caused the shape of the product to result from immediate needs of the departments involved. When there is no real owner filtering the needs of stakeholders, the effect is „design by committee”. That led to a situation where ten people commented on one supplied graphics, everyone putting in their two cents’ worth. There were many opinions and few decisions.
The effect was such that the homepage mockup was overloaded with information to the extent that it did not fully fit into our customers’ smartphones. But then, we have a large logo! And to that, we say how easy it is to order from us, instead of showing it on the facts.
The first step as a Product Owner was to confront what was already done with what actually had to be done. Each stakeholder learned that I was now responsible for the product, which also gave me the right (and obligation) to decide on the product development. I also became a SPOC (Single Point of Contact) for all those who in any way were involved in the development of a new mobile site – whether they wanted to add a function important to them or asked about the color of a button.
In practice it meant that I constantly said „no” in all possible versions („not now”, „a great idea for future dev”, „no, as long as X is not done”, etc.). Although it sounded cold sometimes, it was needed in order to clarify what actually was of a real value and had to be done first.
Sprints and a marathon
The rhythm of work was determined by weekly sprints. Every Monday one sprint ended and another one started. During that time, we did a Review – a summary of changes from the last week and Planning, that is identifying work for the new week.
These activities took us half a day, but thanks to that, the rest of the week was quiet – although originally communication load seems large but with getting into the rhythm of the work, it makes communication much easier and gave me control over the fate of the product.
Weekly iterations have one more advantage: if works go in a wrong direction, it is possible to react promptly and correct the course minimizing wasted time and energy.
Each team member had access to backlog in the Pivotal Tracker that was our main planning tool. During these 6 months of work 35 userstories were delivered, divided into numerous tasks. I connected Pivotal to our company Slack so anyone interested in the organization saw immediately if a specific userstory was taken to a sprint or changed its place in the backlog
Dead ends
In the course of works we fell into many pitfalls. The biggest one for me is thinking about ordering as about a homogeneous and one way process – just as I was used to with mobile applications. However, the site has its own rules – the user can appear on it in a different place than the homepage – for example, when typing „Bobby Burger Warsaw” in the search engine, they go to the restaurant site in our system. Then the website must, in a clever way, take the user through the process taking into account that earlier they did not provide information important for us.
The problem was that we did not have their address so we could not tell them if they could order from it – it was necessary to think about how to handle such a situation, without them closing the site upset. This is easier in the application: You will not see the list of restaurants if you do not enter your address.
That aroused in us a need for „non-linear” thinking when using the service. We planned once more the individual stages and the basic logic of ordering, supplementing it with external traffic that comes on page somewhere during the designed process. On top of that, we add the possibility to jump between stages and our abilities to move data along with the user.
Such a visualization was important for us to know where we should keep the content of the user cart and where we can already delete it. Or, as I wrote above, when to request an address – before showing the restaurant or only after an attempt of adding food to the cart. We argued for a long time on what would upset them more: lack of possibility to check the menu without giving the address or the sense of lost time after choosing a dish when it turns out that the restaurant does not deliver to the place of user residence.We had many of such discussions – for example, whether to enable removing products at checkout or what icon should represent filtering.
Adding, deleting and compromises
Each Planning is a clash of priorities: Is it better to improve design or work on SEO? To improve the service of errors or maybe add filtering? And maybe sorting earlier?
Discussions on this long stretched during our Monday meetings. Sometimes their effects were contrary to intuition. For example, we decided to start the site without any option of sorting and filtering the list of restaurants. Much more important for us was to launch the new service, which already at the initial tests gave better results than delaying until a new function appears. And you know what happened? Nobody complained about the lack of filtering.
We decided to start the site without any option of sorting and filtering the list of restaurants. Much more important for us was to launch the new service, which already at the initial tests gave better results than delaying until a new function appears. And you know what happened? Nobody complained about the lack of filtering.
We tried to cleverly accelerate the entire ordering process. One of the biggest difficulties is to enter an address on a smartphone keyboard. For this reason, we always show hints with cities and streets based on entered letters. However, now we took a step ahead and even with no entering any letter we show five cities with the greatest number of orders in our system. There is more than 75% chance that order will come from one of those places.
During the work on the mobile web we conducted, known from Google Ventures, a Design Sprint (a book on this topic) of a completely different product in our portfolio. One of the conclusions that we did not expect (these are the best) was that users often are afraid to order if they cannot give their apartment number at the beginning. They presume no one will ask them about that in the further ordering process, so eventually their food will go to the neighbor or another floor.
Intuition suggests that the fewer fields to fill out the better but our users made us realize that there is a limit beyond which the user loses control and as a result, confidence in the process. A solution to this dilemma was to add a non-mandatory field „apartment no” on the first screen.
In the „middle” months of our work little was changing visually. I myself as the Product Owner began to frustrate when I was asked how the project is going and the only thing I could say was that „developers are developing.” Meanwhile, I had nothing to show – visual changes in the prototype do not reflect the huge work done at the level of the system logic, which gives the impression that nothing goes forward. However, finally, when the time came to combine together the elements visible for the user with the actual engine, the speed of changes was more than satisfying.
One of our bottlenecks screen is a dish configuration screen. Due to the system logic, technology issues and, above all, the number of possible configurations of food, this screen could load and transfer data with a significant delay. That, from the user perspective, could mean the site was not working. To overcome this impression, we added … well, we added what everyone likes most – GIFs. One for menu load and one for adding meal to checkout.
Testing
Even if the mobile web is not the most important sales channel in PP, the process of substituting the old site with the new one must be carried out in a predictable and safe way.
On 9th December we started the cooperation with the A/B testing team from our Berlin office. So far the team dealt with optimization of particular elements, so as to increase conversion by small steps (e.g. by changing the color of buttons or the size of pictures). We, on the other hand, gave them a task at a completely new level: awhat if we want to do split tests of two completely different pages to check if what we created actually works on a large scale?
That was a serious challenge that required scientific tests and coordination of several systems (Visual Website Optimizer and Advanced Segmentation in Google Analytics) to test the statistical effects taking into account the appropriate size of a sample and the level of significance. In the production environment, we put an alternative site with a beta prefix and daily watched the report with fluctuating results % Delta eCom CVR
We wanted to do our tests in the least invasive manner so we began at a low level and with time we increased the sample. Initially, from the entire traffic, we determined and flagged 10% users who will see the new mobile page. The next 10% will see the old site but will also be flagged – as a control group. The remaining 80% of the traffic did not participate in the study.
With the positive effects, we increased the load to 20% – 20% and finally 50% – 50%. That was useful also for our developer team that could test, in a controlled environment, the load and look for errors that can become visible only with a greater group of real users.
Ship it!
The effects of test completely surpassed our presumptions – our new site was generating 2.3–3 more orders then previous one. When Lech (CEO of PizzaPortal) saw the test results, asked how much money we lost a day not having a new site implemented yet. When I calculated that I went slightly pale.
What is understandable, Lech firmly reasoned, that we should not wait anymore and make the site public as soon as possible. I wanted to still give myself and the team some time for further corrections and the rest of functions, but finally I speeded up the start from 28.12 to 21.12.
And it is good that I let myself be convinced. Only the pressure by Lech enabled me to see that I could be improving the product by adding „another last feature” in infinity. Besides, I have such tendencies, looking at the experiences with previous products.
Deployment day – Monday 21.12 – it is also the time of our Christmas Eve party in the company. When others were having fun we nervously peeked at our phones under the table, observing live if everything was going well, coordinating the Warsaw developer team, the Swedish backend team and our Łódź – Berlin marketing department.
Here is one more lesson: the product is not only the product, but also its infrastructure. We underestimated how much confusion can swapping the API key, changing domains and reconfiguring thirdparty services in the new product cause. The worst thing is that each time it is an extremely individual issue and so rarely performed that there is no universal formula to make all the steps in proper order.
Effects
A month from the start of the site has passed already – historically it is both the worst (Christmas) and the best time (1–2 January) for us when it comes to orders. Thus we had the possibility to test extreme situations of the site’s operation. And what happened?
From the worst working channel, the new Mobile Web advances very quickly in our internal ranking of the number of orders per day – it took over iOS straightaway and now, along with strengthening of SEO, catches up with Android. Consistently, day by day, it generates 2.5–3x better conversion from the previous version. There is still one more exceptional thing: the new Mobile Web has also the best average cart from all channels. These two facts cause that the new mobile web suddenly became a noticeable part of the stream of revenue.
And although in terms of the optimization of loading time, there is still a long way ahead of us, it is very nice to look at the results of automatic tests of Google that show the 100/100 score in terms of convenience for users.
My original plan was to create a new mobile site in three months. Eventually, it took almost twice as long, which also translates into much higher costs. That, however, is still acceptable considering that in the meantime the developer team broke up (of the initial four people there is only one) and it had to be rebuilt almost from scratch – in the middle of work. In most cases, such a situation leads to killing of a product and the need to start from scratch. Fortunately, Scrum and transparency of the process allowed the new team to quickly get accustomed with the product and largely wiped out the delay.
What next
I have a strong impression that we managed to build good foundations for further work and with each new function, new possibilities emerge. And although in these six months we completed 35 userstories, we still have 32 more to do (and new ones still come!). At the same time the experience gained with the creation of the new Mobile Web is for us the basis to work on further projects – this time larger by a scale of size. Keep your fingers crossed!
I thank both teams thanks to which the project was possible and for the faith of Lech and the whole PizzaPortal in my methods.
