robots file to block all backlink crawlers.

J

joe_brannaman

New member
Jan 7, 2015
33
1
0
Hi

I need to consider blocking our backlinks from being shown on the backlinks crawlers. Is there a stock robots file out there that I can download which will have me covered for all backlink engines?
 


If I understand this correctly you want bot scrapers to be blocked in your site. Target is for your competitors not to see your links like Moz and duplicate your strategy.

You may find the following link helpful - Robots.txt and Meta Robots - SEO Best Practices - Moz

With what I know,
Downside of this is you block all the robots like google bot. These crawlers and google uses the same logic, block one you block all of them. Use <meta name="robots" content="noindex"> to block Google indexing these pages. This also block MOZ and other backlink scrapers from crawling these pages.
 
  • Like
Reactions: Member8200
TheWriteOne said:
If I understand this correctly you want bot scrapers to be blocked in your site. Target is for your competitors not to see your links like Moz and duplicate your strategy.

You may find the following link helpful - Robots.txt and Meta Robots - SEO Best Practices - Moz

With what I know,
Downside of this is you block all the robots like google bot. These crawlers and google uses the same logic, block one you block all of them. Use <meta name="robots" content="noindex"> to block Google indexing these pages. This also block MOZ and other backlink scrapers from crawling these pages.
Click to expand...

Well you can specificy exactly which bots to block such as

Mozbot
Majesticbot
alexabot etc.

I was looking for a source that will specify all the bots.
 
joe_brannaman said:
Well you can specificy exactly which bots to block such as

Mozbot
Majesticbot
alexabot etc.

I was looking for a source that will specify all the bots.
Click to expand...
Source like??

If you want to block all bots you can use this

Useragent: *
Disallow: /

The user-agent: * means "this section applies to all the robots.
The Disallow: / "tells the robots that is should not ANY of the visit any of the pages on the site
 
if you had some common sense you'll realise you can replace * with a list of specific bots you want to disallow.
 
a robots.txt will not block the crawlers, at all. robots.txt simply tells the "good" robots that actually respect robots.txt file what and how they will crawl. unfortunately 99% of web bots are "bad" and DO NOT respect robots.txt - especially backlink crawlers. they are some of the worst offenders.

to truly block any robots, you need to use .htaccess or PHP scripts to completely block them off of your server. of course this is on apache servers, which is all i deal with - i simply would not deal with anything else.

to do this you need to write the lines of code and you need to understand things like IP ranges, the difference in ISP & datacenter IP ranges, User Agents, Referrers, Malicious file requests, and more.

This is truly the ONLY way to block them from ever accessing your server or any content at all.
 
^ He's right. You need to block them at server level. A robots.txt file is not enough.

Have you searched other websites robots.txt files? Sometimes you have luck there.

Of course, keep DuckDuckGo, Google and Bing out of the blocking list if you want to be found.
 
You must log in or register to reply here.
Top Bottom