php + js help please

[jlknauff]

Hey guys, I am using cURL to grab content from a page but the roadblock I'm up against is that the page initially displays 100 results. In a browser, you can click a button to show more, but it doesn't affect the URL, so I can't figure out how to get all the data w/ cURL.

Before you ask, the additional results aren't on the page until you click the js button, at which point, the page reloads on the same URL and adds the new results.

Can someone point me in the right direction?

 

[dchuk]

welcome to the world of scraping ajax. time to think outside of the box.

Install HTTP Fox and start watching the headers/data being sent when you make that ajax request. Then go from there.

 

[jlknauff]

Thx, I'll give that a try!

 

[kblessinggr]

Ajax – jQuery API

Basically that. A simple $.get(...) could add content to an existing page when the results come back.

 

[Rage9]

Ajax – jQuery API

Basically that. A simple $.get(...) could add content to an existing page when the results come back.

We are talking about scraping here bud. I'm not sure where your trying to go.

Anyways dchuk is on the right track. Watch the headers. Decipher where the ajax query is being made. Then query that/those urls for the data.

 

[kblessinggr]

We are talking about scraping here bud. I'm not sure where your trying to go.

Anyways dchuk is on the right track. Watch the headers. Decipher where the ajax query is being made. Then query that/those urls for the data.

I was going off this:

Before you ask, the additional results aren't on the page until you click the js button, at which point, the page reloads on the same URL and adds the new results.

Sounded like he wanted it to show results upon clicking a "js button". Anywho I read it again and yes he's on the right track, though in FireFox I'd use FireBug or in Safari the Inspect item option to watch the header requests that way.

 

[Rage9]

Sounded like he wanted it to show results upon clicking a "js button". Anywho I read it again and yes he's on the right track, though in FireFox I'd use FireBug or in Safari the Inspect item option to watch the header requests that way.

FireBug is OK but I prefer the Live HTTP Headers plugin.

 

[gutterseo]

FireBug is OK but I prefer the Live HTTP Headers plugin.

+1 livehttp headers are very useful when scraping. It also has a nice replay function you can use to determine exactly what request is causing the page to load.

 

[nvanprooyen]

httpfox is pretty nice too - https://addons.mozilla.org/en-US/firefox/addon/6647

 

[uplinked]

TamperData. actually, it kind of sucks for just watching the headers, but you can stop and step-by-step replay each request, one at a time, changing any GET/POST fields [or even HTTP headers] that you want. also cooks breakfast in the morning.