It looked like this:
Eric was pretty excited. “Look at the User Agent!” he said. I didn’t see the big deal. Mozilla/5.0 (compatible; Googlebot/2.1). So? I thought.
Could it be?
Eric did some quick digging and he found a couple of other interesting things.
- We did IP address lookup on the errors and it resolved to Mountain View, CA.
After data mining the server’s log files we found some interesting information in relation to the activity of Googlebot/2.1.
GET xxx/GetEventsAndNews - 220.127.116.11 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)
POST xxx/GetEventsAndNews - 18.104.22.168 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)
An ip lookup of 22.214.171.124 verifies it is from Google’s Mountain View based headquarters.
We saw something even more intriguing though; a GET request for an image file from the same ip and query string.
GET /assets/xxx.jpg - 126.96.36.199 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)
Why is this interesting? Because the only way the Googlebot would have known about this was from the returned json data from their ajax call of the GetEventsAndNews function.