[csw-maintainers] robots.txt

[Maciej (Matchek) Blizinski]

On Sun, Aug 30, 2009 at 11:08 PM, Dagobert Michelsen<dam at opencsw.org> wrote:
> Hi Maciej,
>
> Am 30.08.2009 um 10:24 schrieb Maciej (Matchek) Blizinski:
>>
>> Can we set up a robots.txt file on www.opencsw.org? I would suggest
>> the following content:
>>
>> User-agent: *
>> Disallow: /search
>
> Do you really mean www.opencsw.org? Or mirror.opencsw.org? The awstats
> are for mirror.opencsw.org only.

I meant www.opencsw.org specifically, but it really concerns all of
our (sub) domains that are being visited by indexer bots.

Maciej