Help! My robots.txt blocked URLs are still showing in Google Search.
 by Casey Markee

Help! My robots.txt blocked URLs are still showing in Google Search.

  • I have several URLs that I have blocked in my robots.txt file as I do not want them to appear in Google search. Unfortunately, they are still showing up. Why is Google not abiding by my robots.txt file and how can I get these URLs out of the index completely?

Answer: More likely Google IS reading your robots file; the problem is that Google can and does return a link to a page in its index even if it didn't crawl the URL originally.

What appears to be happening in your situation is quite common. You blocked the pages and Google did adhere to your robots protocols. However, they found another link to the page but didn't fetch the result itself. When this occurs Google lists just the URL and no other snippet information.

Take a look at these current examples from Microsoft and Google itself:

blocked-result-1.jpg

blocked-result-2.jpg

Google and Microsoft specifically blocked both these pages within their respective robots.txt files. Nevertheless, these pages are currently viewable in the Google index due to the large number of inbound links tied to each. Google followed the links and noted the pages but didn't fetch the full results.

An extremely detailed tutorial on this very issue was covered by Google Engineer Matt Cutts in a YouTube video back in October. You can watch the full video below.

...

Already a member? Sign in here

Read the rest of this article,
and get all this for only $1.

  • The Search Engine Strategies Updates for September 2016
  • Ultimate Guide to Avoiding Google Penalties
  • The Complete Site Audit Checklist
  • The Definitive Local Search Audit Checklist
  • 100's of Strategic SEO Articles and Q&As
  • The Professional Engine Master's Chart
  • The Internet Marketing Glossary
  • The Ultimate Directory Submission List
  • The Pro SEO's Local Search Directory List
  • 16 years of SEN Archives
  • PLUS, as a Full SEN Basic Member, you'll be eligible for hundreds of dollars of discounts on SEO courses ranging from beginner to master on a variety of topics including organic search, local search, and social networking.

  • Your $1 Trial is good for 7 days at which time your card will be charged $29/mo unless you cancel before October 4th, 2016

 This form is encrypted for your security