Using conditional GET or wget timestamping for the catalog files

[Dagobert Michelsen]

Hi folks,

Am 28.10.2013 um 10:37 schrieb Maciej (Matchek) Bliziński <maciej at opencsw.org>:
> Hey Peter (B) and maintainers,
> 
> I spoke to Dago a few days ago, and we had a chat about a large portion of traffic from our main mirror being just the catalog files, that is, the files named 'catalog' that are downloaded and re-downloaded a countless number of times. The mirror can withstand it, but it's a constant stream of a few megabytes per second, day and night.

Some numbers: we have constantly 3-4 MB per second. This is not a problem ATM as we
have a direct gigabit uplink to the internet, but summing this up it is roughly
10 TB. Just as a comparison: Amazon would charge $0,120 per GB resulting in 1200$ !!
So I would like to take the initiative and see that we save bandwidth now that we still
have the cheap mirror.

> Perhaps this can be helped by using the conditional GET with the possible HTTP 304 Not Modified response, or timestamping. wget has an option to timestamp files, and it can issue just a HEAD request to skip downloading the whole file. Here's some information I found:
> 
> http://www.gnu.org/software/wget/manual/wget.html#Time_002dStamping
> 
> Have we considered this in the past? I don't recall it. Maybe we should take a look, it could be simple to implement, and we would save some bandwidth on our main mirror and on other mirrors worldwide.

Just adding --timestamping would already be a great benefit.

Peter, what do you think?


Best regards

  -- Dago


-- 
"You don't become great by trying to be great, you become great by wanting to do something,
and then doing it so hard that you become great in the process." - xkcd #896

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2351 bytes
Desc: not available
URL: <http://lists.opencsw.org/pipermail/maintainers/attachments/20131028/dee3b729/attachment.p7s>