.Google's John Mueller answered an inquiry concerning why Google marks web pages that are forbidden coming from creeping through robots.txt as well as why the it is actually secure to ignore the related Explore Console files about those creeps.Crawler Visitor Traffic To Query Specification URLs.The individual asking the concern recorded that crawlers were producing hyperlinks to non-existent query specification URLs (? q= xyz) to pages along with noindex meta tags that are also blocked in robots.txt. What motivated the question is actually that Google.com is actually creeping the hyperlinks to those webpages, getting shut out by robots.txt (without envisioning a noindex robotics meta tag) at that point obtaining reported in Google.com Search Console as "Indexed, though blocked by robots.txt.".The individual inquired the observing question:." But listed here's the big concern: why will Google index web pages when they can not also view the information? What is actually the benefit during that?".Google's John Mueller verified that if they can't creep the page they can't find the noindex meta tag. He additionally helps make an appealing mention of the site: hunt operator, urging to overlook the results because the "ordinary" consumers won't view those results.He wrote:." Yes, you're appropriate: if we can not creep the webpage, our company can't view the noindex. That said, if we can't creep the webpages, at that point there is actually not a whole lot for our company to index. So while you could see a number of those web pages with a targeted web site:- inquiry, the average individual won't view them, so I definitely would not fuss over it. Noindex is additionally alright (without robots.txt disallow), it just suggests the URLs will definitely wind up being crept (as well as wind up in the Search Console report for crawled/not catalogued-- neither of these statuses lead to issues to the rest of the internet site). The essential part is actually that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the limitations in operation the Internet site: search evolved hunt operator for diagnostic factors. One of those explanations is actually considering that it's not linked to the normal hunt index, it is actually a separate thing completely.Google's John Mueller commented on the internet site hunt operator in 2021:." The short solution is actually that an internet site: concern is actually not implied to become total, nor utilized for diagnostics objectives.An internet site query is actually a specific sort of search that confines the end results to a certain internet site. It's generally only words web site, a bowel, and then the internet site's domain name.This question confines the end results to a particular website. It is actually certainly not implied to be an extensive assortment of all the pages coming from that web site.".2. Noindex tag without making use of a robots.txt is alright for these type of scenarios where a crawler is connecting to non-existent pages that are actually obtaining found through Googlebot.3. Links along with the noindex tag will produce a "crawled/not indexed" entry in Browse Console which those won't possess an adverse impact on the rest of the website.Review the concern and respond to on LinkedIn:.Why would certainly Google.com mark webpages when they can't also find the material?Included Picture by Shutterstock/Krakenimages. com.