Help! My robots.txt blocked URLs are still showing in Google Search.

• I have several URLs that I have blocked in my robots.txt file as I do not want them to appear in Google search. Unfortunately, they are still showing up. Why is Google not abiding by my robots.txt file and how can I get these URLs out of the index completely?

Answer: More likely Google IS reading your robots file; the problem is that Google can and does return a link to a page in its index even if it didn't crawl the URL originally.

What appears to be happening in your situation is quite common. You blocked the pages and Google did adhere to your robots protocols. However, they found another link to the page but didn't fetch the result itself. When this occurs Google lists just the URL and no other snippet information.

Take a look at these current examples from Microsoft and Google itself:

Google and Microsoft specifically blocked both these pages within their respective robots.txt files. Nevertheless, these pages are currently viewable in the Google index due to the large number of inbound links tied to each. Google followed the links and noted the pages but didn't fetch the full results.

An extremely detailed tutorial on this very issue was covered by Google Engineer Matt Cutts in a YouTube video back in October. You can watch the full video below.