Yesterday, whilst testing our new public Debian Mirror (http://debian.anexia.at/) I noticed that my system would re-download the Packages.gz files every time I did an apt-get update.
Now that’s not what you expect and my first impression that a configuration issue on our side is causing this behaviour was right.
As the title of this entry already suggests, the reason for apt-get always re-downloading the package list was a missing Last-Modified header on the server’s end.
After quite a while of debugging without finding any real results I fetched the lighttpd source code (using apt-get source lighttpd – which got me version 1.4.31-4) and started digging.
5 minutes later I found an interesting piece of code in mod_staticfile.c of lighttpd 1.4.31 , line 449.
Without quoting the code itself, the short version of what’s there is the following: if the Content-Type is NULL, then set the Content-Type to “application/octet-stream” and disable caching.
In this case disabling caching means that both ETag and Last-Modified are neither calculated, nor sent with the response.
But how to fix that? There were multiple approaches as lighttpd processes /etc/mime.types on Debian. One solution that works for each and every file is defining a fall-back value inside the configuration file.
Why? Because the Content-Type will not be NULL when it reaches the check.
tl;dr:
Long story short, the fix was adding these three lines to the bottom of our lighttpd.conf:
mimetype.assign += (
“” => “application/octet-stream”
)