Using conditional GET or wget timestamping for the catalog files

[Maciej (Matchek) Bliziński]

Hey Peter (B) and maintainers,

I spoke to Dago a few days ago, and we had a chat about a large portion of
traffic from our main mirror being just the catalog files, that is, the
files named 'catalog' that are downloaded and re-downloaded a countless
number of times. The mirror can withstand it, but it's a constant stream of
a few megabytes per second, day and night. Perhaps this can be helped by
using the conditional GET with the possible HTTP 304 Not Modified response,
or timestamping. wget has an option to timestamp files, and it can issue
just a HEAD request to skip downloading the whole file. Here's some
information I found:

http://www.gnu.org/software/wget/manual/wget.html#Time_002dStamping

Have we considered this in the past? I don't recall it. Maybe we should
take a look, it could be simple to implement, and we would save some
bandwidth on our main mirror and on other mirrors worldwide.

Maciej
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opencsw.org/pipermail/maintainers/attachments/20131028/749b5f96/attachment.html>