Searching New Content?

[aeisn]

So I need something that can search new comments on digg, reddit etc for a certain keyword. Google doesn't index stuff fast enough, by the time they get it it's already a day too late. So my question is there a pre exisiting tool that can do this? And if not how hard would it be to write one myself. Thanks for the help.

 

[guerilla]

Did you try Google blogsearch?

 

[aeisn]

Yeah I tried blogsearch, just as bad as the normal search.

 

[guerilla]

Doh! I didn't read close enough

Try this

site:digg.com "minutes ago" - Google Search

And this

site:reddit.com "hours ago" - Google Search

 

[erect]

Doh! I didn't read close enough

Try this

site:digg.com "minutes ago" - Google Search

And this

site:reddit.com "hours ago" - Google Search

well done guerilla! ... any idea how hard you can hammer google's cache? Should I throttle back like when scraping with operators (allintitle or scrape away like when hitting real queries.

I know I can test & I know I can use proxies .. would prefer to do neither if I have the option.

 

[guerilla]

You can't hammer Google too hard with site: but harder than allinurl or inurl

I usually get captcha stopped but not outright blocked.

site: is my favorite operator for the stuff I do.

You should PM me.

 

[aeisn]

Anything in google, even site: searching stuff in the last 24 hours isn't picking up comments fast enough, or at all.

 

[guerilla]

aeisn, if you can scrape reddit for the latest discussions, you might be able to pull the feeds for those discussions. That should have the latest comments per article.

I got no other suggestions short of writing your own spider.