International Googlebot is Smarter: Is Your Site Setup Correctly?

Googlebot has for many years used only IP addresses based in the United States for their crawling, however in a change few noticed back in January they changed to add crawler IPs outside the US. This change helps them more accurately detect location aware websites that change content based on user IP addresses. Pages which change content based on a user's IP address or the Accept Language setting in the browser are known as Locale-Adaptive pages.

The new multi-location Googlebot is also now generating an Accept-Language field in the request header when they attempt to load a web page. These changes more closely emulate actual users, but may cause issues with some Locale-Adaptive websites that have not been prepared for Google to make these changes. Sites that change URL structure for alternate user locations and/or languages are not affected by this change. If you DO change content without redirecting or changing URLs, then you need to make sure to test that your site is functioning correctly when an International Googlebot visits.

Merkle has launched a Local Adaptive Page Testing Tool that lets you generate an Accept-Language HTTP request header manually and can also use a proxy IP to help simulate a visit from a different location. This is a great tool to help you test your site if you vary content on the same URL based on user location. According to Google's Site Console Help Page they highly recommend doing the following to help Google correctly identify your content when using Locale-Adaptive pages -

• Google highly recommends using separate URLs for alternate languages, dynamically changing content based on user IP and/or accept language header can be problematic.
• The use of rel="alternate" hreflang=""...