Data Extraction from a site.

Status
Not open for further replies.
doug102938

doug102938

Cartoon Character
Oct 17, 2007
346
3
0
Just curious... how hard would be it to write a little program that would. take every game on xbox 360s or another site and extract the image, description, rating, genre and title and provide those to me in an easy to copy and paste fashion? Or even have them insert themselves into a post?

Is this to many details to go after and to complicated? I'm sure I would have to get someone to do this for me just curious if its even realistic. thanks
 


If they have an RSS feed, extremely easy.

Otherwise you'd use some kind of script, like if I wrote one, I'd have to pre-program it to recognize the pattern or section. The problem with that method is, if they change the coding/html much at all, it could break the script.
 
fair enough i only need the script to really be able to do the job once as i can do updates myself.. i just need the past items as there are hundreds... My other solution is to maybe hire some lost cost data entry labor... Not sure which is more cost effective yet.
 
You can use LWP module to do this, if you know Perl.
Or you can use PHP and get this done with regex.
If you don't know either of these, just outsource it.
 
indianman said:
You can use LWP module to do this, if you know Perl.
Or you can use PHP and get this done with regex.
If you don't know either of these, just outsource it.
Click to expand...

As in outsource the creation of a tool to do this?.. or the manual entering of the data?.. I'm fairly sure you mean the manual route but just checking.
 
Both will work.
If you get somebody to write this tool for you:
1. You can use it again and again, to extract the data, at no additional cost.
2. You can tweak it (or ask the same guy who wrote it to change it for you) to extract data from similar sites, at minimal additional cost.
 
kblessinggr said:
If they have an RSS feed, extremely easy.

Otherwise you'd use some kind of script, like if I wrote one, I'd have to pre-program it to recognize the pattern or section. The problem with that method is, if they change the coding/html much at all, it could break the script.
Click to expand...

Which is why you fetch & save all data immediately after extraction.. or setup file caching.

Also, If you are scraping data, always make sure the pattern is consistent across data as you do not want to fuck yourself over down the line (wink wink personal experience).
 
So my luck with hiring people that I would trust with the manual data entry seems to be going badly.. Anyone here willing to work out a deal in creating the code for me? Or do i need to post in the BST section. Thanks
 
doug102938 said:
Just curious... how hard would be it to write a little program that would. take every game on xbox 360s or another site and extract the image, description, rating, genre and title and provide those to me in an easy to copy and paste fashion? Or even have them insert themselves into a post?

Is this to many details to go after and to complicated? I'm sure I would have to get someone to do this for me just curious if its even realistic. thanks
Click to expand...


I know an Indian chap who can do this with ease . . . I use him to rip a few databases in the past for things like wifi locations in the USA.

If you want, PM me and I will kick you over his Skype info.
 
Not sure what exactly you are looking for but here you go it's a good step in the right direction

name, price, description, image & more ... but it doesn't separate out xbox360, you'll be on your own for that. That service has reviews you can snag also and match up with the games ... but the db is massive
 
1) This data is available via the amazon api; images, prices, et cetera.
2) Even easier, get a phpbay (see my sig for link) plugin and once you've bought it, buy into the phpZon alpha program.
 
Status
Not open for further replies.
Top Bottom