At the beginning of the year, Google made some changes to the way that it crawled pages. The index now looks at how the content on a page changes depending on the IP address of the user, and based on the user’s preferred language settings. A growing number of sites are opting to change the content that they serve to users based on information about their location, and Google admits that it has encountered some problems trying to find and index this kind of international content.
In fact, Google may sometimes fail to crawl international content because the default IP addresses that Googlebot uses are all based in the United States. In addition, Googlebot does not send the Accept-Language flag in its request header.
Going Global
In a bid to resolve this problem, Googlebot has started to add some new IPs to its pool. Google will use these IPs to try to determine whether the content for a given page changes when the users are based in foreign countries. It will then look at the content, and assess whether the new content is more relevant for users in that country. In theory, this will help to improve the search experience for non-US based users, because they will be served up the version of the page that was designed specifically for them.
Googlebot has also started to send some requests using the Accept-Language header to see if foreign languages are being served up for various pages. If you have a local-adaptive website, and want to make sure that Googlebot will see it the way it is supposed to be seen, then you can test it using a crawling tool with the most common languages, to get an idea of how Google views your site.
Google Prefers Locale URLs
While Google is making an effort to understand local-adaptive websites, it still encourages webmasters to use separate locale URLs, and to use the rel=alternate annotation to flag that these are alternate URLs. Using this method of separating URLs will limit the number of crawl configurations that Google will need to use. The technology exists for Google to parse dynamically changing content, but as the international web grows, the scalability problems this will bring are clear. By using Locale URLs, you will be helping Google to understand the content that exists on your website, and exactly who that content is aimed at.
