Hey guys;
i want to scrape/automate & save searches to a public database and save that info to my site/db. So i can create a site that visitors can search on my site - everything i've scraped from these other public databases. I may be using scape word when unneeded. I just need to perform searches on that database(100% free and public), starting from like A and going to Z - saving all the info to my db.
Primarily though, I want to scrape/crawl they're NEWEST database entries. What is the laws/general ethics behind that? They add new people into their database daily/weekly. It's a public database; Public site; all free public info...still legal?
Can i have something crawl them in the early AM each day, and not be worried? Can they see they're being scraped? I want to create a SEARCH feature on my site, where visitors can search for these scraped profiles?
How often can i scrape a site? is it legal? its basically just performing MANY MANY seaches, and saving the info right?
tl;dr I am looking to scrape/perform searches on certain public databases to save that info to my site - thus creating a search portal on my site for these records, instead of having 100 diff sites that you can search - its all accessible from my search. whats the general rules for this? and how can i get this done?
Thanks in advance fellas, tips, advice, suggestions; all highly appreciated.
jt
i want to scrape/automate & save searches to a public database and save that info to my site/db. So i can create a site that visitors can search on my site - everything i've scraped from these other public databases. I may be using scape word when unneeded. I just need to perform searches on that database(100% free and public), starting from like A and going to Z - saving all the info to my db.
Primarily though, I want to scrape/crawl they're NEWEST database entries. What is the laws/general ethics behind that? They add new people into their database daily/weekly. It's a public database; Public site; all free public info...still legal?
Can i have something crawl them in the early AM each day, and not be worried? Can they see they're being scraped? I want to create a SEARCH feature on my site, where visitors can search for these scraped profiles?
How often can i scrape a site? is it legal? its basically just performing MANY MANY seaches, and saving the info right?
tl;dr I am looking to scrape/perform searches on certain public databases to save that info to my site - thus creating a search portal on my site for these records, instead of having 100 diff sites that you can search - its all accessible from my search. whats the general rules for this? and how can i get this done?
Thanks in advance fellas, tips, advice, suggestions; all highly appreciated.
jt