Web Scraping Tools - Please Advise

[AdHustler]

I'm looking for some tools in which I can fairly easily build bots to scrape data off of some of our clients websites. We have permission to perform the scrapes as we are using some of their data in marketing for them. Please do not suggest ubot. It's just a terrible piece of software. Any other suggestions? Thank you!

 

[mattseh]

Why not just pay a programmer to actually do the scraping for you? Real programming languages are a lot better (in the right hands) than ubot etc IMO.

 

[ShadowCaster]

Why don't they just give you DB access? Would save lots of trouble.

 

[CCarter]

CasperJs, if you know DOM and have an understanding of javascript, I found it rather easy to put a couple of Google scraper bots together - Hit 6,500 threads before crashing, I limit it to 4,500 now a days (Also depends on your system's capabilities).​

 

[momchil9]

Scrapy | An open source web scraping framework for Python

 

[edward_b]

PHP Simple HTML DOM Parser

 

[JulianTrueFlynn]

Scrapy | An open source web scraping framework for Python

I checked that out before, and I'm very nooby dev/programmer (understatement of the year) but it is very bloated and I think OP can get by with a much leaner framework.

 

[momchil9]

I checked that out before, and I'm very nooby dev/programmer (understatement of the year) but it is very bloated and I think OP can get by with a much leaner framework.

I take that "fairly easy" is a key part of his suitable solution.

I'm far from good at coding, but was able to do whatever I've wanted so far with scrapy.

 

[AdHustler]

I don't want a programmer to do it because i'd like to be able to whip something up fairly simply on the fly.

They can't provide database access because they don't have it. This is an industry where the clients use locked down third parties for their web development.

 

[AdHustler]

I don't mind paying either, monthly or outright for a point and click type software to get the job done.

 

[Andrew Scherer]

a point and click type software to get the job done.

Seems you're asking for a ubot alternative that doesn't exist AFAIK

 

[HurrDurr]

I don't mind paying either, monthly or outright for a point and click type software to get the job done.

When was the last time you used uBot? I ask because it is way better now then it was a couple years ago. Scraping is very easy. I love the click and drag.


[ame="http://www.youtube.com/watch?v=e09XGjRBpTM"]UBot Studio In-Depth Video Training - First Steps - Part 5 - YouTube[/ame]

 

[Celtik]

Auto Friend Adder and Mass Messenger for ANY Social Network | webDOMinator

It's very similar to ubot as it allows you to create plugins to do whatever you want it to do. Scraping with it should be very easy. I can put you in touch with the creator if you want, just PM me.

 

[edward_b]

Here's a couple more:
Data Extraction, Web Screen Scraping Tool, Mozenda Scraper (web based, paid)

https://www.webharvy.com/ (windows based)

Web Data Extraction Software | Data Extraction Software | Visual Web Ripper (windows based)

I'm also far from a coder, but managed to whip up a few small things using simple html DOM using their examples and docs as none of the above really did exactly what I wanted or seemed slow/overly bloated for what I needed.

 

[dandylion]

Not exactly sure what type of scraping you need, but Imacros is really simple. Can make quick scripts within minutes.

iMacros - Wikipedia, the free encyclopedia

 

[flaw3d]

webharvy sucks I used that shit but I would not recommend it. Mozenda isn't that user friendly and they charge too much for simple shit.

Visual Web Ripper I never used that before.

 

[jryan21]

I think most people just PM dchuk.

 

[erect]

i m a c r o s

 

[greenleaves]

Why not just pay a programmer to actually do the scraping for you? Real programming languages are a lot better (in the right hands) than ubot etc IMO.

mattseh builds awesome scrappers, he has built two for me and it was worth EVERY penny (not google scrappers, custom for a specific purpose)

His code is sexy. 10/10 would bang.

 

[hehejo]

Requests > Ubot etc