Wanted: An Automated Performance Tool That Measures Ads

…since they are part of your website, right?


Update

Since I wrote this article we have been in touch with SpeedCurve. They have now added a nice feature which allows us to have ads included in our performance tests.

I have written an article about this update and how to do it yourself:


Who wrote this?

Allow me to introduce myself, since this is my first Medium post in English which means it can reach a wider audience and people I haven’t been in touch with before.

I’ve been a journalist for almost a decade but I have spent most of the years working with things behind and around journalism. Currently, I work with performance (both technical and organizational) in the development department behind Danish tabloid Ekstra Bladet, one of the biggest websites in the country.


It might sound weird, but earlier performance wasn’t really a ‘thing’ in our circles. The reason that it is now is, let’s be honest, Facebook’s Instant Articles. Sure, the developers cared about fast websites, but the rest of us really didn’t care about performance — that is, until we every now and then came across a slow loading website (especially if it was our own).

To sum it up: It was more about being not slow than it was about being fast. Facebook has changed that. And thank you for that.

Curve your enthusiasm

What is the first thing you do when you want to improve something like website performance? You measure. You find out how your website performs in the here and now, you set up a budget, you tweak and change things — and then you measure again.

This requires automated performance measuring tools. Otherwise you’ll be doing multiple manual tests every day. Which is not something you want to do.

When I first started working with our performance, we had decided on a tool called SpeedCurve. A developer had been asked to look at various tools and SpeedCurve is what he came back with. So naturally we went with that, since it’s often unnecessary to question recommendations like that. Plus, SpeedCurve is sort of the creme de la creme of these things, so it was the right choice.

The first thing we did was add our front page and the front pages from some of our competitors and ‘friends’ in the Danish online news business into SpeedCurve. And we liked what we saw. We had a fast website. Woohoo, job done.

But as the automated tests kept on rolling and rolling and the results kept coming in, we started to wonder. We were performing a little too good compared to what we had seen with our own eyes. Of course, factual measurements beats human experience but what if the measurements are performed on false conditions?

We dug deeper into the tests in SpeedCurve and there we found our problem: The ads weren’t there! For some reason they weren’t included in the tests in SpeedCurve. Instead we were looking at the performance of our own HTML, images, scripts etc., mostly. And it did good compared with the competition. That’s a pat on the back for our developers — but our users really don’t care about that. They get the full page, ads included.

We knew what had to be done: The ads had to be measured. We suspected them to be a major culprit, so not having them in there was a serious flaw.

And thus began the story that is….

‘The Quest for the Performance Measuring of Ads’.

A developer at Jyllands-Posten pointed me to a setting in the performance measuring tool, WebPageTest.

(WebPageTest is what you use, when you do performance tests. SpeedCurve is actually based on WebPageTest — and the most important things in SpeedCurve are the automated tests and a much better design/UI, at least some of the parts — I’ll get back to that.)

What you have to do, before you do a WebPageTest test, is to ask WebPageTest to remove the letters ‘PTST’ from the user agent string (which every browser uses to identify itself):

(I’ve written a blog post in Danish about this nifty little feature.)

‘PTST’ is the culprit in all of this. When our ad technology provider AdTech (now a part of AOL) sees a browser with these four magic letters in the user agent, it withholds the ads from rendering. The reason: To avoid wasting ad displays on tests. Which makes sense, when you think about it.

Run a test on WebPageTest with this checkbox checked and you get everything. And that’s what we want. I’ve seen tests where the ‘fully loaded’ time (the browser is saying “I’m totally done with loading this site now”) multiplied by 5; that’s a 400% increase! In the same test the total number of requests was multiplied by 3 (200% increase).

Oh, and our SpeedIndex value (an expression of how fast the first screen view/viewport loads) increased by 30% in a test.

But while WebPageTest can give us the correct data, it can’t automate it. We could do something via the WebPageTest API, but this is something we want to avoid, so as to not have too many products and service to monitor and maintain.

We then went back into SpeedCurve, but there was no feature to allow this. But… in the ‘Enterprise’ edition of SpeedCurve you are allowed to use the WebPageTest scripting language. One of the things you can do here is set the user agent, which is exactly what we wanted to do.

Documents were written, meetings were held, decisions were made. And we (across JP/Politikens Hus, that is Ekstra Bladet, Politiken and Jyllands-Posten) signed up for SpeedCurve Enterprise. O, how we thought we had it made.

We now saw SpeedCurve rendering the entire frontpage. Just like we wanted. And we started lacking in the comparisons in SpeedCurve, just as we had expected. Especially compared to the Danish Broadcasting Corporation (which has no ads, since they are funded through Public Service).

And the good times kept on coming. SpeedCurve announced that they would now support the same browsers as you can choose between in the developer tools in Google’s Chrome browser. A developer at Politiken tested this and yes, it meant we no longer had to script our user agent. This was a huge plus.

Just look at what happened once SpeedCurve updated the browsers and started including ads:

As you can see, ads have a…certain influence on our front page.

These two screenshots from SpeedCurve shows how big a percentage third party stuff (here; ads) take up of the front page:

Notice those percentage numbers…

When something takes up almost 80 percent of a websites requests and sites shouldn’t it also receive about 80 percent of the attention?

PSTS back in, ads back out

Alas, it wasn’t to last. SpeedCurve changed the browsers and reintroduced ‘PTST’ into the user agent string. Therefore; no ads. We noticed this and went back to scripting the user agent. But that didn’t work either. Though it had earlier.

I got in touch with the SpeedCurve folks. They told me they had fixed a ‘bug’ and that a test browser should always label itself as such, as Mark from SpeedCurve told me in an email:

WPT should always be identifying itself, even if the UA string has been set via scripting.

Instead he created an issue with WebPageTest to allow the user to set the user agent (without ‘PTST’) in the scripting language. Nothing has happened since April 25th. Steve Souders (who is the closest you’ll come to a ‘Mr. Performance’) who also works at SpeedCurve has created an issue with SpeedCurve itself to allow us to remove PTST via a checkbox, like in WebPageTest. This issue was created on March 1st.

We still had one shot left though: Whitelist a browser with a ‘PTST’ user agent with our ad technology provider to to allow the SpeedCurve test browsers to see the entire page rendered. Unfortunately, this is not possible since it is a “global setting across all client networks”. That means, it would have to be changed across all of the sites that use their technology. According to their own website they have 74 countries with active clients.

I then asked if we could allow the browser through if we scripted the user agent to include the word “SpeedCurve”. In effect, their block functionality would allow a browser through if both the words ‘PTST’ and ‘SpeedCurve’ are in the user agent string. But no dice:

As long as PTST is in the UA we will block it.

Alternatives?

This is, obviously, a precarious situation for us to be in. We can’t measure the performance of our entire site automatically.

The logic step is to look at alternatives. So far I’ve only tried one: Calibre (which was suggested to me by the same colleague who suggested SpeedCurve). I even wrote to the guy behind Calibre up front to be sure that it would include ads. But the same result: A fast, lean website. Which just isn’t the truth ;-)

Until SpeedCurve (or WebPageTest) comes up with a change we might look at the initial no-no: Running automated WebPageTest tests through their API. As Jyllands-Posten’s developer suggested, we might be able to get it up and running pretty fast using Amazon.

So… here we are. Thinking about what to do. Since we can’t automatically measure our entire page render, we can’t really do any performance budgets. We can’t measure any tweaks or changes, either. We could do it via manually tests but that is the last way out.

(Also note: Performance budgets are really hard to do, once you’ve got ads in the mix. The load and performance of them vary a lot; week to week, day to day, hour to hour, even banner to banner. Also, the biggest influence on your performance is outside of your control. So ask yourself if a performance budget is the way to go.)


If you made it all the way through this article and have either a trick (or a fully fledged automated performance test tool which include ads…) up your sleeve, please leave a comment.

Banner ads (and for us; the way they are found, delivered and rendered) are a huge performance culprit but we can’t automate the measurements of that fact. We are stuck with manual tests in WebPageTest — or browser developer tools like those in Google Chrome.

(I you found this post by Googling your own frustrations, know this: You are not alone.)


Update on June 14th, 2016:

Apparently this isn’t a problem will all ad tech providers: