.Google.com's John Mueller addressed a question regarding why Google.com marks webpages that are actually prohibited from crawling through robots.txt and why the it is actually secure to ignore the similar Look Console records regarding those creeps.Crawler Web Traffic To Concern Criterion URLs.The person asking the concern recorded that robots were making hyperlinks to non-existent concern guideline Links (? q= xyz) to web pages along with noindex meta tags that are actually also shut out in robots.txt. What prompted the concern is that Google is actually creeping the hyperlinks to those web pages, getting shut out by robots.txt (without noticing a noindex robots meta tag) then receiving reported in Google Browse Console as "Indexed, though blocked by robots.txt.".The person talked to the adhering to inquiry:." But right here's the significant inquiry: why would certainly Google.com mark webpages when they can not even find the content? What's the conveniences because?".Google's John Mueller validated that if they can't creep the webpage they can't observe the noindex meta tag. He likewise produces a fascinating mention of the site: search driver, advising to ignore the end results given that the "ordinary" consumers won't observe those outcomes.He composed:." Yes, you're correct: if our experts can't crawl the webpage, our company can't view the noindex. That said, if we can't crawl the web pages, at that point there is actually not a great deal for our company to index. So while you may see a number of those pages along with a targeted web site:- concern, the normal consumer won't see them, so I would not fuss over it. Noindex is likewise fine (without robots.txt disallow), it only indicates the URLs will certainly wind up being actually crawled (as well as wind up in the Explore Console file for crawled/not catalogued-- neither of these statuses create problems to the rest of the site). The integral part is actually that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's response validates the restrictions in using the Internet site: hunt advanced search operator for diagnostic causes. One of those factors is actually due to the fact that it's certainly not hooked up to the routine search index, it is actually a distinct trait entirely.Google's John Mueller commented on the internet site search operator in 2021:." The brief solution is actually that a website: concern is certainly not meant to become complete, nor made use of for diagnostics purposes.A website question is actually a details type of search that restricts the end results to a specific web site. It's primarily merely the word site, a digestive tract, and afterwards the site's domain.This query confines the outcomes to a details internet site. It is actually certainly not suggested to be an extensive selection of all the pages coming from that internet site.".2. Noindex tag without using a robots.txt is fine for these kinds of circumstances where a bot is linking to non-existent pages that are actually acquiring found by Googlebot.3. URLs along with the noindex tag will certainly generate a "crawled/not catalogued" item in Explore Console which those will not possess an adverse impact on the remainder of the web site.Check out the inquiry and answer on LinkedIn:.Why would certainly Google mark pages when they can not also see the material?Included Photo by Shutterstock/Krakenimages. com.