Crawling google SERPs - will G ban me?

[justo_tx]

If I run a batch of queries a day (say anywhere from 1,000 to 10,000 or maybe more) from a single IP address will google end up blocking the requests after a few days?

If there is a ban possibility does anybody have a ballpark number of queries that can be run a day before pissing them off?

 

[Rascagua]

how long does it take to try and figure it out yourself? and most likely get a much more accurate answer then all the assumptions you are about to hear.

 

[justo_tx]

how long does it take to try and figure it out yourself? and most likely get a much more accurate answer then all the assumptions you are about to hear.

I'm trying to avoid pissing away however many hours I would take to end up writing the crawler if it's just going to get banned in two days, but whatever, this is about the type of answer I expected.

 

[emp]

err...

yes... you are going to get blocked pretty fast (a capctha) so either:

a) Simulate human behaviour.
This means slowing down your crawler, adding pauses randomly, etc...

b) Use proxies.

::emp::

 

[jryan21]

Most humans cannot do 15 searches a second, so don't make your crawler do 15 searches a second.

Don't make your crawler do dictionary searches in alphabetical order. Most humans do not do 15 searches a second in alphabetical order with perfectly spelled dictionary words.

Also, don't waste your time trying to spoof user-agent strings on google. They usually know when you are doing that.

 

[justo_tx]

emp / jryan, thanks for the insight guys.