Collecting URLs

[Atomm]

I am looking for a cheap program to spider a specific URL and report all the link addresses. If it can filter, that would be great!

I know I can do this with script, but if there is a good program out there already, I'd rather just get it.

Thanks!

 

[bluebobbo]

Go Google it, or hotscripts.com

Maybe something like "site spider" or even a sitemap tool would work, they do basically the same thing, just have a different output.

 

[Palmerville]

If you want cheap and know a little regex, you can use "pscrape". http://www.webscrape.com/

To grab all the links from http://www.dmoz.com/Business/Investing/Brokerages/ for example, you would:

pscrape -i -m -u"http://www.dmoz.com/Business/Investing/Brokerages/" -e"<a href=\"(.*)\">"

You can scape pretty much whatever you want however. Since it's a command line program it's infinitely scriptable and will take whatever input and give you whatever output. If you need an automated easy method of say grabbing album song titles every week *cough*ringtones*cough* I couldn't image living without something like the above.

 

[Atomm]

Well, I haven't done much in the way of RegEx for about 5 years. I've pretty much forgot everything I use to know. Guess it's time to bone up again.

I've actually downloaded a bunch of programs trying to find something good and I don't like any of them. I was hoping someone might have a suggestion on a program I missed.

Thanks for the help.