The following info is intended to assist you in identifying the search engine spiders that visit your site. Details are based on information you can obtain by viewing your site's visitor logs.
Use the User Agent to identify a particular search engine in your site's visitor logs, then use the Robots.txt Identifier in your robots.txt file if you'd like to block that spider from accessing parts of your site. For more information on blocking any of these spiders using your robots.txt file, see http://www.robotstxt.org/wc/exclusion.html.
Read the rest of this article,
and get all this for only $1.
The Search Engine Strategies Updates for October 2016
Ultimate Guide to Avoiding Google Penalties
The Complete Site Audit Checklist
The Definitive Local Search Audit Checklist
100's of Strategic SEO Articles and Q&As
The Professional Engine Master's Chart
The Internet Marketing Glossary
The Ultimate Directory Submission List
The Pro SEO's Local Search Directory List
16 years of SEN Archives
PLUS, as a Full SEN Basic Member, you'll be
eligible for hundreds of dollars of discounts on SEO courses
ranging from beginner to master on a variety of topics
including organic search, local search, and
Your $1 Trial is good for 7 days at which time your card will be charged $29/mo unless you cancel before November 1st, 2016