Need a large scrape done.....

[jaykhicks]

Hey all....im not a programmer or anything fancy at all...but I enjoy learning, but what i have in mind is way WAY out of my league!

I would like to scrape Thomasnet.com

I would like the following data:
Company
Address
Phone
Fax
Company URL
company profile text
Sales
employees
activities
year founded
certifications
officials

Basically I want their database....

How much for data in CSV?

 

[Enigmabomb]

Are you in the glue industry?

 

[TheMoMan]

WOW - I'm familiar with PHP and have written several scrapers for my own personal use and took a look at the data you wanted to scrape. That's a HELL UV a lot of data! Can be done, but a bit of work (researching the data formats, and writing the appropriate GREP/string matching).

Have you tried elance or rentacoder? You should be able to find someone there to do it for a few hundred dollars.

Give them your requirements, they'll give you a script that could be run from your server. For the amount of data to scrape you may want to also look at using proxies. You may not want to run such a script from a shared hosting provider - I'm assuming you may have your own server or semi-dedicated.

Good luck!

 

[Caisy]

Do you have an example company profile url with all the data you are looking for? I've been looking about and can't find, Sales, for example. I'ed be interested in working on this project.

 

[SpamHat]

You should know that directory companies like this, yellow pages and even maps have seeded databases - meaning they put fake profiles in there that if marketed to they may go after you.