How we built the biggest homepage in Poland. A story about looking for answers
Adam Plona (@adlon)
According to experts from National Oceanic and Atmospheric Administration, the summer of 2014 was the hottest in 130 years. It was equally hot at Wirtualna Polska (Virtual Poland — wp.pl). It was near the end of that summer, on 1 September 2014, that we launched our new homepage, which just several months later became the most frequently visited in Poland. But if you think that my today’s story ends on the date of implementation, I have a surprise for you. The most important things happened later.
Let me tell you a story about looking for answers.
In 2014, the Polish internet market went through two major shifts. Wirtualna Polska and o2.pl, two large online media in Poland, joined forces and established the Wirtualna Polska Group, which is now the biggest online media player in Poland. Another great consolidation of media players happened one year earlier — Onet.pl was acquired by Ringier Axel Springer. The contest for leadership on the Polish internet has just started.
At Wirtualna Polska we knew that in order to become a leader we need to show Poles that we’re making a new start, with new blood coursing in our veins. We knew for sure that our competitors are working on their own offensive and that if we are late, we will have much worse place in the leadership race.
So it was the end of June, when we set ourselves a bold and risky goal. We decided to launch a huge marketing campaign on the 1st September that would let all Poles know that there is something new and great in Polish internet. But if we wanted to do it, we needed to be credible. We had to show the start of the change. And what better way to start than by changing the most important page in our whole group?
A few words of explanation here. The portal’s homepage is an intersection of three forces: big traffic, big money and powerful but conflicting interests.
1) Traffic, because Wirtualna Polska’s homepage is visited monthly by 10 million Poles who generate more than 450 million pageviews. On a standard day, users on site click nearly 300 000 times per hour.
2) Money, because for each of the four largest horizontal portals in Poland, the homepage is the key revenue line, representing 20% to 40% of the entire business.
3) Conflicting interests, because it’s the source of 90% of traffic for vertical websites like news, sport, finance or tech. So brand managers fight like lions for exposure on the homepage. And the same do sales managers, for whom every pixel represents a chance for additional budgets.
So the homepage is like a superheated pressure cooker of conflicting interest. And we decided to heat the atmosphere up even more. We decided to change everything. In two months.
But let me be clear.
Our case is not a textbook example and I should say: don’t try it at home. Big portals usually change in a very different way. By evolution, which is often unnoticed by the average user. Such changes are very safe, because they are tested for many days before they’re available to all users. Below you see the examples of BBC, New York Times, Amazon, and even an example from the Polish market — this is the previous redesign of Onet.pl, our main competitor.
Evolution.
Of course we knew this. But sometimes you just don’t have that kind of convenience. We didn’t, because we had just 2 months. When your window of opportunity is that narrow, with only one chance to make it work, you have no alternative. You fly or die.
And we definitely wanted to fly.
We kicked it off. We were working on this project following the agile development methodology. The workgroup consisted of leaders of nearly all areas of the organization, because changes on the homepage affected the entire company. The whole team were meeting on daily basis. We organized 51 meetings in total, plus dozens of smaller ad-hoc groups. And its easily double that number. The fact that Wirtualna Polska is based on two remote cities – Warsaw and Gdańsk — didn’t made it any easier.
We drew the first lines in Axure on 29 June, but it wasn’t very similar to our final project. The project had nearly 80 iterations. Several graphic designers worked on it simultaneously. In consequence webmasters and programmers worked on almost constantly transforming project. We weren’t even creating specification because it was pointless. Sometimes an element had changed twice a day, because something designed in the morning was no longer that attractive by the end of the day.
During these two months, change was an immanent feature of our every workday. By the end of the project we were so used to it, that when I personally changed the central element of the homepage — the main topic — with just ten hours left to launch, the programmers just said „yeah, we’ll do it“. And its pretty weird if you know how developers usually react to sudden changes.
Our tempo was impressive. The whole mobile version of the site was created from scratch in just 20 hours, including all nuances of typography, responsiveness, breakpoints, java script transformations and the behaviour logic of the individual elements. Without a single line drawn in any graphic or UX software. All it took was some paper, a fair dose of mathematics, a great graphic designer and an exceptional webmaster. Like a real life hackathon.
In the same time our UX specialists went out to meet the users. We didn’t have time to organize stationary usertests, so we chose a quicker methodology: ambush guerilla user testing, invented by Andy Budd from The Guardian. To refresh your memory: the method involves testing prototypes ‘in the street’ through highly spontaneous interviews in city parks, shopping malls and coffee shops.
One of our designers decided to mix business with pleasure, and that’s how we conducted our first usertests on the beaches in Sopot and Gdańsk. We had nothing to show at the time, so we tested our old site. At the and of interview we asked our interviewees to describe Wirtualna Polska as if it was a human being. Lets see what our respondents said:
A guy in his late fourties, very formal
An old guy with no style, a miserable bore
An aged woman, a gossip who likes to talk a lot
And old stiff
We started to ask ourselves what will they say when we ask the same question after the launch of the new homepage? In other words: are we really going in the right direction?
I’m talking about these doubts, because we knew exactly how big was this revolution. The new homepage almost completely changed the long-established way of exploring our hompage. The division into the left magazine section and the right news section was replaeced by a division into horizontal topic sections, like news, finance, sport etc. We were planning to replace the sea of blue links with a grid of photographs and vibrant colours. To the current 20 photos on the page we were going to add an additional…few hundred. We removed most of the navigation items. We wanted to revolutionize not just the editorial parts, but also change the locations and forms of a large number of advertising formats.
Naturally, we were trying to limit the potential risk of error. With coming beta versions, we were conducting ad-hoc user tests. In total we ran four such series. We also managed to do several A/B tests on the old homepage. We tested the most controversial elements that we were planning to implement in the final project. This gave us the peace of mind that the company would not go bankrupt on 1 September, and we would not be fired.
However, we still had many questions that we couldn’t answer within those two months. Let me show you some examples.
As you can see we had many concepts for our right column. But which of them would be best? With mini pictures or without them? With blue links, or with black?
We also wanted to choose the best navigation layout.
But which one will be working best? Red? White? With fewer or more links? With links on the right or on the left? What about the right column — is it better with more pictures or not? Should the weather be on the header or would it work better in the right column? What about the main topic? What about the left column with the colour cables? Will they work better with photos or without them? And the automotive section — perhaps it should be above, not below the women’s section?
But there came the launch date, and all these questions remained unanswered.
On 1 September, exactly two minutes past midnight, we launched the new homepage of Wirtualna Polska. Champagne bottles were opened and joy filled the corridors. We were successful. We launched a homepage, a huge campaign, and everybody talked about us, mostly in a very positive way. In theory, we could have ticked that box and started a new chapter.
But a sense of uncertainty kept bothering us. What about all these question that we didn’t answered? It seemed that some parameters could have been higher if we only knew what to change. The questions that we kept asking ourselves before we started were now joined by new ones. But the most important question was: could our page be better?
And that’s where the real fun started.
In the middle of September we started to actively look for answers. We verified hypothesis after hypothesis by conducting multi-variant A/B tests. By the end of the month we accomplished 13 such tests.
One of the hypotheses under investigation concerned the right column with the links. And well, we found out that our initial decision had been wrong. One of the variants, the green one, was improving the conversion of the homepage by several percent. At our scale, this is huge.
So we quickly implemented it and immediately saw an improvement of parameters.
And it was like a drug.
A moment to prepare the test, a short waiting time and — sniff! — real, perceptible change. We got hooked really quickly and we started testing right and left.
In fact, this process of seeking answers sometimes gave us surprising results. We found this out when testing, for example, the structure of the photo tiles, the basic building block of our homepage.
Our hypothesis was that the current structure, on the left, was probably not easy to read. So we developed several variants with backgrounds and other fonts. It seemed super-logical that the variant with the uniform background will work best. How very wrong. Much better proved to be the variant with changed font, the green one on the right.
Equally surprising was one of the last tests. We tested 9 different headers but I will show you only three variants. We noted something interesting in that test. The most popular element here is the mail icon. Tens of thousand of users click it every day. In the current layout, mail icon is always next to search bar. But in these three variants we changed the position of this icon.
And what the test showed? The icon positioned in the previous location of the mail, next to search bar, in any given variant always generated the most clicks. This means that users used this element from memory. They clicked a familiar location, not noticing that both the icon and the title had changed.
As you can see tests like this teach us more than many books or some speech by a guy at a UX conference.
We have been looking for answers, stating hypotheses and verifying them for 7 months. We have introduced 60 bigger and smaller improvements. Each month we test several dozens of new variants. And we continuously look for better solutions. Is it worth the trouble?
Let me show you a graph illustrating how the CTR of the new homepage changed since the first week after implementation. I am proud of this slide.
Our painstaking work succeeded in improving the basic conversion parameter of our homepage by more than 20%. But we also achieved something more. An amazing source of knowledge. Every A/B test adds another ingredient to our secret sauce. This is how we build our competitive advantage. Thanks to it we know much more and much faster than our competitors. This value is priceless.
I bet you’re wondering whether we are happy in general with the whole redesign. I could of course tell you a great deal about how well it was received by media houses, how satisfied our customers are and how various parameters have grown. But business is about money, right? So let me show you just one figure. It represents the year to year increase in advertising revenue on the homepage.
And this number is 100% increase.
I encourage you to not be satisfied with your initial ideas even if you’ve already implemented them. Explore your own hypotheses. Seek answers. Challenge your judgment. Always dig deeper than others. It will yield you an ocean of knowledge, and your pride will swell. These don’t have to be A/B tests or focus groups. The point is to always question your decisions. To test hypothesis after hypothesis, even those that look illogical at first sight. The tenacious search for answers is a feature of every good UX designer, product manager and entrepreneur.
Keep looking for answers.
It’s worth it.
This is a transcript of my speech at UX Poland 2015 conference. Polish version is available on my blog. If you like it, feel free to share on Facebook or Twitter. I’m @adlon at Twitter.
