Are you scraping Google results for client rankings? Be careful!
 by Casey Markee

Are you scraping Google results for client rankings? Be careful!

  • For the last couple of months we have been using a self-written tool to scrape the Google database for client ranking results. We have been running the tool a couple of times a week but after a few hundred queries we get blocked. Can you provide some tips on how to get around this or maybe an alternative we can use?
Answer: If you are using a "few 100 queries" then you are probably using the tool too much. Google is clear that they DO NOT like automated querying of their database and have said as much in their Google Webmaster Guidelines:
"Google's Terms of Service do not allow the sending of automated queries of any sort to our system without express permission in advance from Google. Sending automated queries absorbs resources and includes using any software (such as WebPosition Gold) to send automated queries to Google to determine how a website or webpage ranks in Google search results for various queries."

That being said this is going on and has been going on well before Google entered the scene. Nevertheless, those that do this (and do this effectively) run the queries across hundreds of different IP addresses. So if you can't do this at your end, then you need to consider using a tool or service to do it for you, and safely.

For example, our Rank Monitor runs across dozens of different IP addresses. This guarantees no IP has an abnormal strain of automated queries to tip-off Google that the resulting queries are anything but natural. If you aren't a member of, this is a great reason to sign-up.

Here's a VIP link to that will allow to "kick the tires" and try...

Already a member? Sign in here

Read the rest of this article,
and get all this for only $1.

  • The Search Engine Strategies Updates for October 2016
  • Ultimate Guide to Avoiding Google Penalties
  • The Complete Site Audit Checklist
  • The Definitive Local Search Audit Checklist
  • 100's of Strategic SEO Articles and Q&As
  • The Professional Engine Master's Chart
  • The Internet Marketing Glossary
  • The Ultimate Directory Submission List
  • The Pro SEO's Local Search Directory List
  • 16 years of SEN Archives
  • PLUS, as a Full SEN Basic Member, you'll be eligible for hundreds of dollars of discounts on SEO courses ranging from beginner to master on a variety of topics including organic search, local search, and social networking.

  • Your $1 Trial is good for 7 days at which time your card will be charged $29/mo unless you cancel before November 4th, 2016

 This form is encrypted for your security

  • Bing's Keyword Planner Adds Product Keywords in Latest Update

  • What's New in Local Search for March 2016

  • Your 2016 Professional Local Search Audit Checklist

  • Understanding & Unraveling Google's 9 Most Commonly Played Penalties

  • The Ultimate Guide to Avoiding Google Penalties

  • GWT adds Redirect Errors, Mobile Only Stats & Improved Keyword Data

  • Google's New Image Mismatch Penalty Explained

  • 8 Steps to Insulate Your Web Site from Future Spam Updates

  • Publishers Can Lighten the Load with Google Media Tools

  • AdWords Keyword Tool Being Replaced by New Local Friendly 'Keyword Planner'

  • Register for New 'Structured Data' Dashboard

  • Your Guide to the New Google Places for Business Dashboard

  • The Ultimate Keyword Primer, Part 2

  • Twitter Closing Posterous Blogging Platform

  • Raven Releases a Schema Microdata Markup Tool That Rocks!

  • EMD Update Smacked your Site? Here's How to Recover!

  • Updates Rich Snippets Tool to 'Structured Data Testing Tool'

  • Panda Refresh Arrives on Schedule

  • Help! My PageRank just Dropped from 6 to 3. Did I get a Penalty?

  • Will using Guest Post Content Trigger a Panda/Penguin Slap?

  • Bing Webmaster Tools Report Metrics Change

  • Bing Discloses Secret SEO Recipe

  • AdWords Announces Hangouts on Air

  • Learn with Google Webinars

  • Google Buys Facebook Competitive Data for $400 Million

  • Webmaster Tools add Traffic Change Notices and Structured Data Dashboard

  • 'Fetch as Googlebot' Should NOT be Ignored

  • Webmaster Tools Adds 'Crawl Error' Alerts & Latest Links

  • Taking Strides to Compete with Siri and Become Your Assistant