[csw-maintainers] robots.txt

[Philip Brown]

I am against putting in "limit" directives, unless there is a very
clear, specific benefit.
Unless i missed something, you mentioned only general case things, and
a potential of broken robots.

Any robot that cant figure out to properly prune   /page  vs
/page.php, in this day and age, is a broken robot.
We should not modify our configs to make dumb robots look better than they are.

(that being said, if we have any references to page.php anywhere,
instead of the native "page" name, we should update our links. But
that is besides the point of the changes you are proposing)