Web Scraping Tools - Please Advise

[dchuk]

Hit up mattseh, he is both an expert web scraper and a tool

 

[mattseh]

Hit up mattseh, he is both an expert web scraper and a tool

import magicrequests as requests
import time
import string
import collections
import json

url = 'http://www.warriorforum.com/warrior-special-offers-forum/'

views_counter = collections.Counter()
posts_counter = collections.Counter()

while True:
    print url
    try:
        r = requests.get(url)
    except:
        continue
    counts = [int(c.replace(',', '')) for c in r.xpath('//tr/td[5]/text()') if not any(char in c for char in string.lowercase)]
    posters = r.xpath('//tr/td[2]/div[2]/span/text()')
    views_counter.update(dict(zip(posters, counts)))
    posts_counter.update(posters)

    next = r.xpath('//a[@rel="next"]/@href')
    if len(next):
        url = next[0]
        time.sleep(5)
    else:
        break

    print 'most common', views_counter.most_common(10)

json.dump(views_counter, open('wafo-views.json', 'w'))
json.dump(posts_counter, open('wafo-posts.json', 'w'))

 

[Grindstone]

Waste of time bro, your service doesn't have a 7 in the price and doesn't promise the next one click shiny button to 100% guaranteed success and eternal happiness.

 

[matt3]

Just pay hehejo... His turn around time is faster than you building anything.

Always did a great job for me.

 

[j0hnsmith]

A great tool is mattseh, has voice, email and skype interfaces.

 

[DanTe_0101]

Seems you're asking for a ubot alternative that doesn't exist AFAIK

ZennoPoster 5| Get rid of routine with the help of ZennoPoster 5!

Didn't used but i heard that its better than ubot and they also offer free demo.

 

[Avalanche]

I don't know how complicated of a scraper you want, but check out import.io. Little buggy since it's in beta, but free & pretty easy to use.

 

[Hammi]

just hire a coder.. seriously. most little jobs like your talking about can be done in an hour or 2.

 

[Xdreamer]

Web Scraper, Web Extractor, Screen Scraping, Web Ripper, Web Spider

 

[Suddenly_Ass]

 

[AdHustler]

just hire a coder.. seriously. most little jobs like your talking about can be done in an hour or 2.

Sounds good in theory but when you have clients and something breaks, that coder isn't at your beck and call to fix it.