hey guys,
I have setup a working scraper as the guide in this thread.
http://www.wickedfire.com/design-de...le-keyword-tool-willing-share.html#post900237
everything works fine but when I Download the keyword data it seems something has changed with the url data since this post was made.
when i sniff the packets and add the url to the app it will work for a few hours then it seems to expire.
this is the step by step method I'm using with vb.net.
Basically it break down like this;
NOTE: All request should save the returned cookie data and send any previously saved cookie data.
1) Login to Google. I use this URL 'https://www.google.com/accounts/ServiceLoginAuth', but there are a few different ones.
2) Request this url "https://adwords.google.com/um/StartNewLogin?sourceid=awo&subid=ww-en-et-gaia" and follow all the redirects, there are 6 (this is setting cookies need for Adwords).
3) When the last redirect complete and the HTML data is returned, extract the values for __u= and __c=. I use regex but any way will do.
Code:
u=(.*?)&__c=(.*?)&
4) Request this URL "https://adwords.google.com/o/Targeting/Explorer?stylePrefOverride=2&__u=&__c=" adding the values extracted for 'u' and 'c'. The URL should look something like "https://adwords.google.com/o/Targeting/Explorer?stylePrefOverride=2&__u=1654564568&__c=87 94964564"
5) Extract the value for token, again I use regex
Code:
token:'(.*?)'
6) Send a POST request to the URL 'https://adwords.google.com/o/Targeting/file/DownloadAll' with the POST data outlined below.
7) (OPTIONAL) Decompress the data and format it with a CSV function.
POST data;
$u is the __u value you extracted in step 3
$c is the __c value you extracted in step 3
$format can be CSV, GZIPPED_CSV, CSVFOREXCEL, XML or TSV
$county is a uppercase two letter country code, US, GB, DE, etc
$language is a lowercase two letter language code, en, jp, etc
$keyword is the keyword (do not url encode the keywords, leave a space as a space, don't change it to %20 or +)
$token is the token you extracted in step 5
This is the POST string for keywords and this is the url in question.
1:
0BC91872D430CA67A5237BB62909C258
2:
|106|17t|110|en_US|ul|12g|11f|17k|11i|
I can not find where these values are coming from it seems they are set via an ajax call but I cant pin point how or when.
If anyone out there can give me some hints or tips on this method of scraping the new keyword tool or even other methods if there are any that would be great.
thanks.
I have setup a working scraper as the guide in this thread.
http://www.wickedfire.com/design-de...le-keyword-tool-willing-share.html#post900237
everything works fine but when I Download the keyword data it seems something has changed with the url data since this post was made.
when i sniff the packets and add the url to the app it will work for a few hours then it seems to expire.
this is the step by step method I'm using with vb.net.
Basically it break down like this;
NOTE: All request should save the returned cookie data and send any previously saved cookie data.
1) Login to Google. I use this URL 'https://www.google.com/accounts/ServiceLoginAuth', but there are a few different ones.
2) Request this url "https://adwords.google.com/um/StartNewLogin?sourceid=awo&subid=ww-en-et-gaia" and follow all the redirects, there are 6 (this is setting cookies need for Adwords).
3) When the last redirect complete and the HTML data is returned, extract the values for __u= and __c=. I use regex but any way will do.
Code:
u=(.*?)&__c=(.*?)&
4) Request this URL "https://adwords.google.com/o/Targeting/Explorer?stylePrefOverride=2&__u=&__c=" adding the values extracted for 'u' and 'c'. The URL should look something like "https://adwords.google.com/o/Targeting/Explorer?stylePrefOverride=2&__u=1654564568&__c=87 94964564"
5) Extract the value for token, again I use regex
Code:
token:'(.*?)'
6) Send a POST request to the URL 'https://adwords.google.com/o/Targeting/file/DownloadAll' with the POST data outlined below.
7) (OPTIONAL) Decompress the data and format it with a CSV function.
POST data;
$u is the __u value you extracted in step 3
$c is the __c value you extracted in step 3
$format can be CSV, GZIPPED_CSV, CSVFOREXCEL, XML or TSV
$county is a uppercase two letter country code, US, GB, DE, etc
$language is a lowercase two letter language code, en, jp, etc
$keyword is the keyword (do not url encode the keywords, leave a space as a space, don't change it to %20 or +)
$token is the token you extracted in step 5
This is the POST string for keywords and this is the url in question.
its nothing to do with the token as I have checked that. It seems to these values changing.
1:
0BC91872D430CA67A5237BB62909C258
2:
|106|17t|110|en_US|ul|12g|11f|17k|11i|
I can not find where these values are coming from it seems they are set via an ajax call but I cant pin point how or when.
If anyone out there can give me some hints or tips on this method of scraping the new keyword tool or even other methods if there are any that would be great.
thanks.
