[csw-maintainers] robots.txt

[Philip Brown]

2009/8/31 Trygve Laugstøl <trygvis at opencsw.org>:
>
> It very well might be old links that has gotten index at some point in time,
> and since we still return 200 OK on those URLs, Google will keep on
> re-indexing them. Try adding a permanent redirect from "page.php" to "page"
> and they will disappear after a while.

Seeing as how the .php page is the "real" page, that might be a little
challenging.

Or perhaps, since we currently have apache "fancy[somethingorother]"
to auto-dereference "page", to "page.php" on the back end... there
might be an apache setting to automatically redirect "page.php" to
"page"?

We should only do that for certain subtrees, though. not mantis ones.
ugh.