The WF Ruby Functions War Chest

hehejo

hehejo

Developer
Sep 22, 2009
803
12
18
Switzerland
www.peakinformatik.com
I know there are quite a few ruby coders on here. Why is there only a PHP and Python thread?

Checks for available exact match domains.
.org domains return not found which is unreliable so only .com and .net are checked.

As I am new to ruby feel free to improve/correct my script
Code: 
class Emds
  def available(keyword)
    kws = Array.new
    res = Array.new

    kws << keyword.gsub(/ /, '')
    kws << keyword.gsub(/ /, '-')

    tlds = ['.com','.net']
    
    kws.each do |kw|
      tlds.each do |tld|
        domain = kw + tld
        whois = `whois #{domain}`
        res << domain if whois.include?( "No match" )
      end
    end

    return res
  end
end

# how to use
emds = Emds.new
puts emds.available("insert keyword here")
 


Great addition, I've moved to Ruby recently, enjoying it far more than any previous language.

Anyway, don't have much to contribute yet.

Here's a proxy manager I made, that limits which proxies are used, based on when they were last previously used.

https://github.com/JakeAustwick/Ruby-Proxy-Manager

(linking to GitHub because it has syntax highlighting and the README, and we all know WF syntax highlighting sucks)
 
alright, so first, instead of calling whois directly on your own system, use Ruby Whois - Ruby Whois Gem ... it's awesome.

Second, I created and launched this gem a few weeks back, it's a super fast and efficient domain crawler: http://www.github.com/dchuk/Arachnid ... you give it a single url on a domain, it will crawl every single page on that site. You can give it patterns to look for in the block that is returned for each page (just check out the docs). I'm going to add page limiting today to it.

I have a shit ton of other ruby code, I'll dig through it and see what other stuff I can clean up and post here.
 
I can confirm the awesomeness of Anachnid, used it a few times now, and it's pretty damn fast.

Just submitted a pull req on it with the small tweak we talked about on skype.
 
dchuk said:
alright, so first, instead of calling whois directly on your own system, use Ruby Whois - Ruby Whois Gem ... it's awesome.

Second, I created and launched this gem a few weeks back, it's a super fast and efficient domain crawler: http://www.github.com/dchuk/Arachnid ... you give it a single url on a domain, it will crawl every single page on that site. You can give it patterns to look for in the block that is returned for each page (just check out the docs). I'm going to add page limiting today to it.

I have a shit ton of other ruby code, I'll dig through it and see what other stuff I can clean up and post here.
Click to expand...

Thanks.

Has anyone wrote a scraper for the google adwords kw tool yet? If not I might give it a try soon. Looks like it should be possible with watir.
 
hehejo said:
I know there are quite a few ruby coders on here. Why is there only a PHP and Python thread?

Checks for available exact match domains.
.org domains return not found which is unreliable so only .com and .net are checked.

As I am new to ruby feel free to improve/correct my script
Code: 
class Emds
  def available(keyword)
    kws = Array.new
    res = Array.new

    kws << keyword.gsub(/ /, '')
    kws << keyword.gsub(/ /, '-')

    tlds = ['.com','.net']
    
    kws.each do |kw|
      tlds.each do |tld|
        domain = kw + tld
        whois = `whois #{domain}`
        res << domain if whois.include?( "No match" )
      end
    end

    return res
  end
end

# how to use
emds = Emds.new
puts emds.available("insert keyword here")
Click to expand...

Pastebin link

Code: 
class String
  def available?
    `whois #{self}` =~ /No match/
  end
end

class DomainChecker
  attr_accessor :domains, :available_domains

  def initialize(keyword)
    @keyword = keyword
    @tlds = %w(.com .net)
    @domains = []
  end

  def process
    keywords = []
    keywords << @keyword.gsub(/ /, '')
    keywords << @keyword.gsub(/ /, '-')
    keywords.product(@tlds).each {|domain| @domains << domain.join}
    @available_domains = domains.select(&:available?)
  end
end

checker = DomainChecker.new "my cool niche"
checker.process

puts "\nAVAILABLE DOMAINS (%d of %d)" % [checker.available_domains.count, checker.domains.count]
puts checker.available_domains

#=> Output:
# AVAILABLE DOMAINS (4 of 4)
# mycoolniche.com
# mycoolniche.net
# my-cool-niche.com
# my-cool-niche.net
I don't have that much Ruby experience, but from what I understand looking at pro code, you want to pollute the namespace of at least one high level class no matter how trivial the app. :food-smiley-002:

When working with URLs, Addressable is pretty cool.
 
Finished the above. Just ran though another 10,000 questions, managed to solve them all with 100% success rate.

Link is in above post.
 
Nice thread.

I found I had to round-robin across multiple whois servers when checking the availability.
That's doesn't work too well for .org, but at least it cuts through .com and .net quickly.

Putting these things on github is an interesting trend, I will check out the repos you guys listed and see what I can add.
 
This thread seems to have died. I'm working on a few things currently, will update when they're done.

Anyone else got anything interesting to share?
 
You must log in or register to reply here.
Top Bottom