[csw-maintainers] robots.txt

Philip Brown phil at bolthole.com
Mon Aug 31 20:33:11 CEST 2009

I am against putting in "limit" directives, unless there is a very
clear, specific benefit.
Unless i missed something, you mentioned only general case things, and
a potential of broken robots.

Any robot that cant figure out to properly prune   /page  vs
/page.php, in this day and age, is a broken robot.
We should not modify our configs to make dumb robots look better than they are.

(that being said, if we have any references to page.php anywhere,
instead of the native "page" name, we should update our links. But
that is besides the point of the changes you are proposing)

More information about the maintainers mailing list