CDN change and its effects on GoogleBot’s behaviour

Couple of months ago, we, at Practo, decided to switch the CDN for our website and services from Akamai to CloudFlare. Akamai is one of the largest and most popular content delivery platforms out there. However, we had to move out of it and here’s why.

• Akamai’s configuration update system, Luna Property Manager is a very elaborate and complex system. It is also extremely configurable and gives you enough power to change and modify the behaviour to the maximum possible extent. Because of the same reason, the learning curve and maintenance of these configurations is fairly time consuming. For a startup like ours, investing so much time and efforts into fine tuning the system was too much of an overhead.
• Secondly, the kind of performance boost and security that Akamai provides, doesn’t come at a small price.

On the other hand, in CloudFlare, the number of things that we could control and configure were far fewer but that was exactly what we wanted. The amount of time that we had to spend managing the setup had reduced drastically. Also, with simple dashboards where configuration changes are click-of-a-button away, simple to understand graphs to monitor the setup and great documentation to help us integrate without any need of professional support, CloudFlare was an obvious choice.

Other than a few minor hiccups like, inability to activate couple of features at subdomain level, inability to configure url rewrite rules, WAF false positives, etc. the migration proceeded as per the plan. After few days of efforts, we had all our services migrated to CloudFlare. This was followed by a relaxed weekend, since the target was met — or so we thought.

Very soon, we discovered a drastic drop in GoogleBot’s crawl rate. What followed was an anxious day, hoping our SEO ranking doesn’t get affected by this drop. By the end of the day, we had switched back, our consumer facing crawl-able site, to Akamai to avoid any side effects (Thanks to the Akamai team for saving the day with a service extension).

Monday morning blues!

The decrease in crawl rate could directly be correlated to increase in time spent in downloading the page. The culprit seemed to be the increased response time, which was a result of increased latency. It was also observed that the response times of the requests served to our users weren’t as bad as it was to GoogleBot. It was obvious that the cause for increased latency was because of our origin servers and GoogleBot servers being in two different continents. Since majority of our end users are in the South and Southeast Asian countries, which is also where our origin servers are hosted, end user response times weren’t affected at all. But why hadn’t we noticed these differences till date? Because of Akamai! Akamai’s routing took care of the latency issue for us. Our next step was to figure out how CloudFlare could solve this for us. Details on our attempts with CloudFlare offerings is a story for another time.

While we were concentrating on figuring out our latency related issues, a new fact came into picture. GoogleBot’s algorithm had a big role to play in the change in these numbers.

So, what was the main cause of drop in crawl rate?

IP address change !!
With the CDN change, the IP address had changed.

Why would a new IP address result in drop in crawl rate?

Whenever there is a change in the IP address, GoogleBot has to ensure that it doesn’t overwhelm the servers since it is unaware of the capacity of the new system. GoogleBot also takes into account what the hosting provider can sustain. With CloudFlare, we possibly had more neighbours and hence got a lower rate for ourselves. Hence, GoogleBot starts slow and takes time to relearn how much it can crawl. The crawl rate is ramped up, post some experimentation. Phew! That’s one smart spider!

Summarising the causes of the drop

• Change in IP.
• Change in CDN.
• Increased latency.

After letting GoogleBot understand the capacity of our “new” system for a couple of days and with a boost of CloudFlare’s offering to reduce latency, we were back to getting millions of our pages crawled by GoogleBot, at rates higher than before!

Follow us on twitter for regular updates. If you liked this article, please hit the applause icon to recommend it and help other Medium users find it.