What’s up with Airbnb’s slow-loading photos?
In late April I began planning a road-trip for myself and a few friends around Sicily. Having used Airbnb successfully in the past, we expected to again turn to Airbnb for the majority of our stays. What we didn’t expect was the trouble we’d encounter using their website this time.
Almost as soon as I began browsing Airbnb locations near Palermo, I noticed that the photos of the houses were taking a very long time to display. A glance in Chrome’s status bar showed:
Waiting for a0.muscache.com…
“Okay”, I thought, “their CDN is just being a bit slow tonight. No big deal.” I persevered, booked the first day’s Airbnb and forgot about it.
A few days later I needed to book another Airbnb stay and I again encountered the same image loading delays. These were not small delays — it would often take over a minute for photos to be displayed, and sometimes they wouldn’t display at all (with an HTTP 598 error code).
This time I did some digging. Naturally, I ensured I was connected via Ethernet cable, ran a long-lived ping to 8.8.8.8, ran some speed tests, and ultimately satisfied myself that my broadband connection was not at fault. I spoke to a friend going on the same trip and asked her to give it a try. She encountered exactly the same issue from a different location completely.
Turning to Twitter
So I turned to Twitter to see if others were experiencing this issue, and boy were they!
It’s hard to pin down when it began, but one of the earliest and most popular tweets was back in December 2014:
Of course, a search for Airbnb slow will yield countless other results from all around the world.
@AirbnbHelp, whilst very responsive to these complaints, sticks to the same stock answer for most tweets:
Sometimes they’ll point the finger at your browser, or tell you to try incognito mode, or even indicate some of their regional image servers are at fault.
What’s really going on and how widespread is it?
First thing’s first. A quick look at Chrome’s developer console shows that the delays were all occurring for a{0,1,2}.muscache.com. What is muscache.com? A quick WHOIS lookup will tell you it’s owned by Airbnb themselves, so you might think they’ve spun up their own photo-serving CDN. However, a couple of short DNS queries later you’ll note that it is just an alias onto Akamai’s CDN infrastructure.
$ host a0.muscache.com
a0.muscache.com is an alias for evsan.airbnb.com.edgekey.net.
evsan.airbnb.com.edgekey.net is an alias for e864.b.akamaiedge.net.
e864.b.akamaiedge.net has address 23.214.167.187
Ah, so can we blame Akamai? Almost certainly not. The URL structure of an Airbnb photo on a property’s image carousel is as follows:
The query string parameters indicate that the content is almost certainly being reverse proxied to some backend Airbnb servers for image reprocessing, and Akamai’s job is just to cache the response at the edge. This points the finger at Airbnb’s backend.
Executing the same request with the ever-awesome cURL from a very well connected server, we can eliminate any possibility of the issue being related to a client’s browser or cookies:
$ curl -w “TCP connect: %{time_connect}s\nTotal time: %{time_total}s\n” -k -v -o /dev/null -s “https://a0.muscache.com/ic/pictures/11850408/f7b273ea_original.jpg?interpolation=lanczos-none&size=large&output-format=jpg&output-quality=70"
> GET /ic/pictures/11850408/f7b273ea_original.jpg?interpolation=lanczos-none&size=large&output-format=jpg&output-quality=74 HTTP/1.1
> User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 libssh2/1.2.7
> Host: a0.muscache.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Akamai Image Server
< Last-Modified: Mon, 10 Dec 2012 02:00:18 GMT
< ETag: “b276e578ec87e1fba5617265986c4200”
< Content-Type: image/jpeg;charset=UTF-8
< Content-Length: 61385
< Cache-Control: public, max-age=2394832
< Date: Mon, 25 May 2015 19:35:19 GMT
< Connection: keep-alive
<
{ [data not shown]
TCP connect: 0.023s
Total time: 66.281s
66 seconds for an image to be fetched!
Interestingly, sometimes it seems Akamai isn’t happy with Airbnb’s very sluggish backend and times out after exactly 60 seconds with an HTTP 598 error:
< HTTP/1.1 598 Unknown
< Server: Akamai Image Server
< Content-Length: 31
< Date: Mon, 25 May 2015 19:32:02 GMT
< Connection: keep-alive
<
{ [data not shown]
TCP connect: 0.024s
Total time: 60.233s
Handily, cURL is very scriptable. So to test the claim that the delays were isolated to certain regions, I ran five cURL requests from a number of servers I have access to around the world and took the median result from each. Each server is very well connected, with at least a 1Gbps uplink, and is a physical dedicated server (no VMs here). Each request was unique; I altered the output-quality query string parameter to ensure that Akamai couldn’t deliver a cached copy.
London: 61s
Amsterdam: 3.4s
New York: 21.2s
Texas: 1.8s
Atlanta: 64.1s
Los Angeles: 2.1s
Sao Paulo: 88.2s
Singapore: 3.4s
Paris: 61.1s
Melbourne: 2.6s
Hong Kong: 2.2s
Toronto: 18.9s
The results are, frankly, all over the place. Even the very best case of 1.8s from the Texas location is pretty poor. The worst case of 88.2s in Sao Paulo is just embarrassing.
However, rerunning the results for Texas a bunch more times revealed some very high loading times as well. So changing your hosts file to point a*.muscache.com to Akamai’s Texas IPs is not a good solution.
Conclusion
Airbnb have had extensive problems with slow loading photos for at least a month now, and most likely a lot longer than that based upon Twitter. The issues are experienced globally.
The @AirbnbHelp Twitter account acknowledges the issue and says their engineering team is hard at work on rectifying it. This gets harder and harder to believe as time goes on.
I hope this post will help shine light on a issue that perhaps even Airbnb’s own management don’t realise is very widespread and seriously affecting user experience. More importantly for Airbnb, this will almost certainly be affecting their bottom-line too. Selfishly, I just want them to fix it before I go away again!
