Google has released an update to its official Googlebot documentation which now provides guidance on visits from IPs that are linked with GoogleUserContent.com. This update includes documentation on user-triggered bot visits, which was previously missing from Googlebot documentation, leading to confusion among publishers. As a result, some publishers blocked IP ranges of legitimate visits, and this update aims to clarify and categorize the different kinds of Google bots that publishers can expect. The three categories of Google Bots include the Googlebot – Search crawler, Special-case crawlers, and User-triggered fetchers (GoogleUserContent). The last category has been a source of confusion for publishers because Google didn’t have explicit documentation about it.
GoogleUserContent refers to tools and product functions where the end-user triggers a fetch. For instance, Google Site Verifier acts on a user’s request. Because the fetch was requested by a user, these fetchers ignore robots.txt rules. The documentation states that the reverse DNS mask will show the domain ” –––.gae.googleusercontent.com.”
The updated documentation now categorizes user-triggered fetchers in their own group, and some of the user-triggered fetchers include Feedfetcher, Google Publisher Center, Google Read Aloud, and Google Site Verifier.
Previously, it was believed that bot activity from IP addresses associated with GoogleUserContent.com was triggered when a user viewed a website through a translate function, which is no longer available in Google’s SERPs. However, the new documentation clarifies that bot activity from IP addresses associated with GoogleUserContent.com can be triggered by the Google Site Verifier tool.
In addition to this, Google has retired its Mobile Apps Android crawler, and the company now advises users to verify that the domain name is either googlebot.com, google.com, or googleusercontent.com. The updated documentation finally provides clarity on bots that use IP addresses associated with GoogleUserContent.
Previously, search marketers were confused by these IP addresses and believed that those bots were spam. However, it is now clear that bot activity from IPs associated with GoogleUserContent is not spam or hacker bots, but rather from Google. Publishers who had previously blocked IP addresses associated with GoogleUserContent should probably unblock them. The current list of User-Triggered Fetcher IP addresses is available on Google’s website.
