Robots.txt - Disallowing files with no extension

[rockhard]

Is it safe to use:

User-agent: *
Disallow: /*.$

to disallow all files with _no extension_, without fucking anything up up?
Will http://www.site.com/ still get indexed?

 

[jryan21]

I tested the line at google webmaster tools against domain.com/file and it didn't block it. Also:

domain.com/ -Allowed
domain.com/.file -Allowed
domain.com/file. - Blocked (I think this is the specific pattern the line is trying to match)

If you have filenames with no dot, I'm not sure what you would need to do to distinguish a file from a directory.

 

[Drake]

o Ya. New Wildcards and robots.txt files. Always an exciting combo.

If this is a high value site and you have time might want to try with some less valuable domain. But might have to take time to upload strange files and have them indexed.

 

[audax]

You'd probably have more luck with adding a mod_rewrite rule that adds the X-robots HTTP header to any requests for files without an extension. I don't believe regular expressions are universally accepted in robots.txt.

 

[rockhard]

Adding a custom file extension seems like the best option then.

 

[Traffic-Bug]

Thanks for the inputs