The Ultimate Guide to the New InspectionTool Crawlers

On May 17 2023, Google added new crawlers called InspectionTool crawlers to their list. These crawlers come whenever we use the Rich Result Test and the URL inspection in Search Console. You can find this here, look for the section on Google-InspectionTool.

Two crawlers are listed -

A mobile version -

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible;Google-InspectionTool/1.0)

And a desktop version - 

Mozilla/5.0 (compatible; Google-InspectionTool/1.0)

Google’s documentation is partially accurate. Upon examination of multiple sites’ server logs - more InspectionTool Crawlers are in active service.

Below I’m listing them with chrome builds as of June 28 2023.

DESKTOP (2)

Mozilla/5.0 (compatible; Google-InspectionTool/1.0;)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.179 Safari/537.36 (compatible; Google-InspectionTool/1.0)

MOBILE (2)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.179 Mobile Safari/537.36 (compatible;Google-InspectionTool/1.0;)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.179 Mobile Safari/537.36 (compatible;Google-InspectionTool/1.0)

* Note how a semi-colon character distinguishes these two mobile crawlers.

Below are examples of the types of files each crawler appears to be responsible for crawling.

The simple html crawls relate to the first pass crawl referenced above, where the simple keyword is found. The crawling of the script files and css files is the rendering crawl. That rendering is when the javascript is activated and once that has completed is when Google first can assign the javascript keyword to the page.

Any discussion of WHY Google introduced and developed these new crawlers is by nature going to be conjecture. Google does not give us detailed plans and tell us how or why they test. We know that they do test search results and that includes all their data streams -( onpage content, the link graph, click validation, images etc) and it is in the multiple thousands of tests during any 12 month period. 

My first question is what problem does it solve by going to the added expense of creating and testing these InspectionTool crawler agents? 

Prior to the introduction of these new crawlers, there was no way to distinguish at least through the server logs and I suspect the same to be true when it came to the data processing on the other side of search console, between content that was discovered and content that WE told google about either through search console or the various other tools that show us what google sees. 

However, isn’t it now highly likely it is easy for Google to identify the content that THEY discovered and distinguish that from the content that we called attention to via these tools telling Google proactively about our new content? 

In the documentation, Google confirms that these crawlers come when indexation requests are made through the search console and for the Rich Result Tests.

Independent tests have confirmed that they also send these crawlers for tests via the mobile friendly test tool. (Another reason why an update of the documentation is justified.) 

Perhaps it is in Google’s benefit to be able to sort out search results that THEY discovered and assigned keywords to vs search results that are a blend of their discoveries and our requested new content to be crawled, indexed and assigned a keyword to be findable in the results for that term/s. 

Based on the data gleaned from server logs, the Google Developer crawler page needs updating as of July 3 2023 on the section describing the InspectionTool Crawlers from the two listed to the four (4) that are in active duty as well as what tools are using these new crawlers. 

About the Author: Carolyn Holzman is Vice President of Digital Marketing Strategy at Strategic Marketing House. Her specialty is forensic SEO, if a site is failing to perform in search results, she knows where to dig and solve the problems. Since August 2021 she has been testing the indexation systems of Google daily and sharing her findings on the YouTube channel @CrawlorNoCrawl. She is the host of the weekly podcast, Confessions of an SEO, now in its 3rd season.