googlebot & logs
Data I found checkin the logs. In order to not forget what better than to share it in Medium?
When: around April 2014
What: around 10 sites split in 3 different countries
How long: I don’t know if for a week or a month
What I found:
User-Agent
23% of the Googlebot calls where done by the smartphone user-agent: https://developers.google.com/webmasters/mobile-sites/references/googlebot
3,5% done by the Feature phone user-agent
83,8% by the classic Googlebot
0,1% done by the (rarely used): Googlebot/2.1 (+http://www.google.com/bot.html) as in https://support.google.com/webmasters/answer/1061943?hl=en
Robots.txt & sitemap
0,94% of the calls were done to the robots.txt file
0,05% of the calls were done to the sitemap file
0 % of pages blocked with Robots.txt were crawled
Where from:
99% of the times the bot came with an american IP
Getting deeper:
In Germany:
1,5 % KL
33,59 % SB
40 % BR
0,17 % Glossar
0,225 % Lexikon
17,46 % track.y
0,92 % homepage
1,03 % blog
0,2% optiker
0,05% service
