Scraping web pages on the fly

Status
Not open for further replies.
lucab

lucab

New member
Jan 4, 2007
677
3
0
I am designing a site that allows users to enter a product they are looking for: for example, "Widget 3000." Now this happens to be a blue widget. When the user enters the product, I want to display this product with some details, along with other blue widgets.

Learning different methods and trying to decide what would be the best way to go - curl/php, or ruby/scrubyt!, or mechanize/perl, or... Undecided as of now.

Anyway, would this be feasible to do on the fly? User enters query, script submits to webpage, scrapes product details and related products and then displays it.

Or is this crazy and the only way to make this work is to have previously scraped the data and call it up from a db?
 


Have you seen the esrun ask.com scraper videos? Sounds like you want to do something similar. Go to youtube and search for esrun.
 
i watched those videos a while ago, but lost the link. +rep when i have some to give out.

anyone have any experience with employing esrun? i am beginning to think that I may have to outsource this - I am learning how to scrape data, it is just the navigating around that is making this project complicated.
 
I've duplicated what esrun did and built a scraper to get article urls from ezine articles. I may be able to help you out if you want. I don't have a ton of time though so if it gets too involved...

Basically the search and scrape part should be fairly easy.
 
Yeah this is pretty easy to do. If you like Perl, you can use LWP.

I'm not sure what kind of products you need to do this for, but you might also look into using Amazon's API. It can do all of this including product titles, descriptions, images, size, etc. for almost every product in their inventory.
 
Status
Not open for further replies.
Top Bottom