[csw-maintainers] robots.txt

[Maciej (Matchek) Blizinski]

On Tue, Sep 1, 2009 at 1:37 AM, Philip Brown<phil at bolthole.com> wrote:
> 2009/8/31 Trygve Laugstøl <trygvis at opencsw.org>:
>>
>> It very well might be old links that has gotten index at some point in time,
>> and since we still return 200 OK on those URLs, Google will keep on
>> re-indexing them. Try adding a permanent redirect from "page.php" to "page"
>> and they will disappear after a while.
>
> Seeing as how the .php page is the "real" page, that might be a little
> challenging.
>
> Or perhaps, since we currently have apache "fancy[somethingorother]"
> to auto-dereference "page", to "page.php" on the back end... there
> might be an apache setting to automatically redirect "page.php" to
> "page"?
>
> We should only do that for certain subtrees, though. not mantis ones.
> ugh.

mod_rewrite[1] could help. There could br a rule along the lines of:

RewriteRule ^/packages.php(.*) /packages$1 [R=301]

We'd need to test it, of course.

[1] http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteRule