HTTPS is Hard
This is an abridged version of my article HTTPS is Hard, which you can find, in full, on the Yell Blog. You can also see my reflections on the whole thing on my personal blog.
“We’re a business directory, why do we need to be secure?” I said to Daniel Applequist (W3C Technical Architecture Group co-chair) in the pub after London Web Standard’s January 2015 event (full disclosure, I help to run it) where he spoke about HTTPS. Daniel replied,
“Think about what information your users put through that every day, searching for legal counsel and family planning clinics, as well as regular plumbers and hairdressers. They search for it locally, and that is all personally identifiable. If I were a hacker intercepting this traffic I could work out some pretty interesting stuff about you.”
It’s a solid standpoint — in this post-Snowden world, the person snooping on you may not be reaching you through that insecure coffee-shop router that you’ve logged on to whilst waiting for your latte. Add back a line re Yell security: When traffic is unencrypted, it opens the potential for anyone to identify you and your internet footprint, and Yell could have been a part of this, so we felt it was important we looked at increasing web security across Yell.com.
Having HTTPS is also proving increasingly important as a search ranking factor. According to Daniel Applequist:
“Google is pushing hard on this, they made it a ranking factor to encourage the big guys to change. If you’re selling this transition to your boss, that’s what you’ll major on.”
I mean, how hard can it be?
In 2015, the year of hoverboards, this should have been simple; it’s just a few certificates in the right places. Put those on the servers, update your links and add some redirects to point to the HTTPS URL , right?
Right?
Implementing HTTPS
The process
Before starting this journey, I expected I would have to talk to the internal teams, listed in the diagram below to get HTTPS implemented on Yell.com. In smaller organisations, it could even be fewer teams than this, but for Yell, by the end of the journey, this model couldn’t have been further from the truth.
I began by identifying all of the different components of our system that used HTTP as a hard-coded link, and tried to switch them over to HTTPS. The easiest way to do that was to set up a self-signed certificate on my local development machine and see what failed. There are lots of good tutorials on setting up self-signed certificates on different web servers to help with this. I used the Mozilla SSL Configuration Generator to get going, as well as this handy article on getting the green padlock locally on OS X.
With those set up, I ran the site and looked for errors, and there were a lot of them, including:
Adtech
AOL’s Adtech platform is one of the largest advertising networks in the world, along with Google’s DoubleClick, and they both support serving adverts over HTTPS.
The issue here was that even though Adtech serves images and JavaScript libraries over HTTPS, the adverts that they inject into your page may not be served over HTTPS. This produced a mixed content warning in the best case. In the worst case, there was a 300x250 pixel gap where the advert should have appeared.
Solving this was a human problem — I sat down with Yell’s advertising manager and requested that all adverts be served over HTTPS. They were more than happy to oblige, and ticked all the appropriate boxes for us. They also changed the processes for the advertising sales team to ensure that new advertising is served over HTTPS by default.
My team released this update by March 2015, well before our main HTTPS transition. By this point in the process, the diagram had grown to include:
Adobe Analytics
Yell makes great use of Adobe’s analytics solution Omniture, (an alternative to Google Analytics). It’s a really important tool for Yell, and there are a lot of people in our team working with it.
I filled out a form with a request for a certificate and a recommendation that Yell upgrade our implementation as well as move to a closer data-centre. This all made sense, especially the regional data collection, meaning we’d be sending data 50 miles, rather than 5,000 miles.
This part of the process took time — nearly two months. It involved testing, trialling, upgrading libraries across multiple systems and prioritising this amongst other changes and priorities.
At the end of the process, Yell.com is better off in many ways: the upgrade reduced page load times by around 100 milliseconds (ms) and thanks to a new library, our overall analytics capabilities have improved.
This change went live to Yell.com in April 2015 — three months after we began our transition to HTTPS. By now, the departments involved included:
By this stage in the transition, we’d reached the point where the bulk of the technical challenges were out of the way and are live (continuously deploying and testing this was one of the best decisions we made on this project). The last thing to do was get the certificates onto the servers and redirect our users.
Obtaining certificates
Buying an SSL certificate is relatively simple for an individual to do. For a single website I’d recommend going to the LetsEncrypt project, which provides free certificates for services through a simple command-line interface. We wanted an extended validation (EV) certificate, for a few reasons:
- It’s a mark of trust in the organisation
- It’s not much more expensive than a regular certificate
- It’s the only type of certificate that turns the padlock green in Microsoft’s Edge web browser, and provides larger amounts of green in other browsers, which is important for the perception of security
The process for obtaining an EV certificate involves not only proving that you are the owner of the site digitally, but that you exist physically too. For Yell, this meant asking our legal team to align our domain name registration (under Yell) with our Companies House records (under our parent company, hibu (UK) Limited), This was relatively simple to solve, but the whole process took four weeks.For most companies this may not be necessary, but it is worth noting if your domains are registered under alternative group or legal names.
By June 2015, the project and the number of departments consulted had grown significantly:
Other technology providers
Yell also work with an anti-scraping service and a content delivery network for our video products. Involving these suppliers in the process required a business case for sign off by the company. For a lot of engineers, writing a business case can be a daunting process. One of the few resources I found to help with this was Making the Software Business Case by Donald Reifer (available on Amazon). Writing the case, taking input from all of the stakeholders, and presenting it to the budget holder, came in at a total of three weeks from start to finish.
By June, my sphere of influence for this had grown again
Go-live
On launch day itself, we updated the following items, all of which are critical to the site’s SEO:
- Our sitemaps, and every URL inside them (> 10M links)
- Robots.txt file, pointing to the updated HTTPS sitemaps
- Canonical URLs for every page, with HTTPS links
- Register Google Search Console for HTTPS and all of the sub-properties, so we can monitor the transition
- 301 redirects to HTTPS for HTTP traffic
My team spent pretty much all day waiting for things to fall over, but they didn’t! The transition was a success, and there is now a green padlock for every user.
The aftermath
With a transition of this magnitude and a product this large it was no surprise that we didn’t think of everything right at the start. After the launch, we spent a couple of days fixing bugs like e-mail sending, publishing premium adverts and our performance testing suite. The main culprit here was Java.
Java
Depending on the version of Java you are using, certain Certificate Authorities aren’t trusted and therefore the system throws up a silent security error if it tries to get a resource from a HTTPS URL with an untrusted certificate. Here we discovered that not all of our systems had our certificate provider in their key stores.
It should have been a simple matter of updating the keystore, but for a while this didn’t work. We noticed this only after we had started redirecting internal traffic from HTTP to HTTPS, and at that time, we were redirecting our users too. There was a key lesson here: you may need an internal HTTP service that isn’t upgraded, to deal with any applications that simply don’t want to work with HTTPS. Chalk that one up for experience.
TLS Performance
We have been told that HTTPS is fast, and that it makes very little difference for modern browsers. You no longer need dedicated hardware to perform decryption within a reasonable amount of time — but how much of a difference does it actually make to your bottom line? Well, with real user monitoring, I can tell you.
HTTPS costs 150ms for desktops and 200ms for smartphones at the median, and over half a second at the 95th percentile
That’s a relatively large gap to have to make up, especially at the extremes where 5% of users are connecting in an additional 2 seconds as their clients negotiate the HTTPS connection.
To mitigate this, we performed some TCP Profile tuning on our load balancer, with Ilya Grigoik’s High Performance Browser Networking guiding us, which saved us about 20ms at the median, and a little more at the extremes. Frankly, that’s not a lot of benefit from the profile — but it’s better than nothing.
HTTP Referrer
You may not know this, but a lot of analytics information relies upon HTTP referrers to identify where traffic came from. It’s useful for businesses of all sizes to identify if a marketing campaign is working, or their links around the web.
The HTTP 1.1 spec defines when to pass a referrer:
HTTP => HTTP Pass referrer
HTTP => HTTPS Pass referrer
HTTPS => HTTP Do not pass referrer
HTTPS => HTTPS Pass referrer
So, with Yell.com now on HTTPS, for the referrer to be passed, the destination site must also be on HTTPS. Whilst the internet transitions to HTTPS, things aren’t looking good. 99% of our customer’s websites are served over HTTP, and of that 1%, a quarter of those are Facebook pages.
This has wide-ranging ramifications for us.
Yell provides click-through metrics to our merchants directly, and so I naïvely believed they would only be using that to check their performance. But Google Analytics is the tool of choice for many of our customers, and if we’re not sending referrer data to them there, it looks like we’ve just stopped working. This isn’t the case, as their overall traffic hasn’t dipped, but looking broken is sometimes worse than actually being broken.
I spent a few weeks working with our sales team, our customer services team and the marketing team to help customers through this transition. We’re not going back to HTTP, and that option has never been raised in meetings I have attended, partly thanks to Google’s position on HTTPS as a ranking factor. We’re still working together as a business to communicate the benefits of the transition to our customers, and to encourage them to use campaign tags in their URLs, which give much more referrer and data accuracy, especially in GA.
So, after all that, my simple change has led me to talk to the majority of the company.
HTTPS is Hard
This article is written with the best of intentions, to highlight how difficult it can be to implement HTTPS across a large organisation such as Yell and the challenges other businesses can encounter through the process.
For me, the whole exercise was worth it. For all the times my head hit the desk at the stupidity of some technology, and the times where things just took far too long, seeing that shiny EV certificate in my browser is something I’m proud of bring to Yell.com. We resolved far more small issues with all of our systems than I’ve detailed here — in fact, I probably made changes to every single component in our stack to complete this change. For all those challenges, Yell is now in a better position that we’ve ever been in terms of resilience and security.
In summary: implementing HTTPS isn’t just hampered by technology problems, it also has a people problem — namely a lack of incentives.
How can we make HTTPS easier
Based on everything Yell has been through to implement HTTPS, I suggest the following:
- Eliminate the cost of standard certificates. LetsEncrypt is showing how this can be made easier and free, Dreamhost is undercutting themselves and Amazon Web Services is providing free certificates for their East-Coast data centre. All of these companies are setting the trends that others need to follow.
- Reduce the cost of setup: I understand that private IP addresses can be costly, but this step needs to get cheaper (or become free) to be successful. The alternative to this, SNI, is not broadly available from service providers at the moment.
- Google, Bing and Yahoo need to make HTTPS a stronger ranking factor. Having it as a tiebreaker at the moment is nice, but it doesn’t have as big an effect as it could. If Google really wants the web to go HTTPS, there needs to be a stronger signal from them on this.
- Improve education on what HTTPS will actually do for you — especially in regards to referrers, performance and certificate issues. Google have recently created some HTTPS transition guidelines, and released their HTTPS transparency report, though currently, there’s not a lot of information out there.
- Implement the privileged contexts spec for developers — there are plans to deprecate access to geolocation, user media and others, whilst service worker will be HTTPS-only. This is known as “privileged contexts”, or more commonly, “powerful features”. Deprecation of features may not be enough here, and this is only viable once the first few items in this list have been made a success or this may have the opposite effect. I look forward to this advancing more in the near future.
- Finally, browser manufacturers need to be sending strong signals to the public, and online companies, that HTTP is insecure. Both Google and Mozilla have plans to action this in 2016, though to what extent it is still unknown.
You can find my full article, HTTPS is Hard, on the Yell Blog
I’d love to hear your experiences and comments on Twitter: @steveworkman or @yelldev
