The tale of lighttpd not sending the Last-Modified header
As the title already suggests, the reason for apt-get always re-downloading the package list was a missing Last-Modified header on the server's end.
Yesterday, whilst testing our new public Debian Mirror (http://debian.anexia.at/) I noticed that my system would re-download the Packages.gz files every time I did an apt-get update. Now that's not what you expect and my first impression that a configuration issue on our side is causing this behaviour was right. As the title of this entry already suggests, the reason for apt-get always re-downloading the package list was a missing Last-Modified header on the server's end. After quite a while of debugging without finding any real results I fetched the lighttpd source code (using apt-get source lighttpd - which got me version 1.4.31-4) and started digging. 5 minutes later I found an interesting piece of code in mod_staticfile.c of lighttpd 1.4.31 , line 449. Without quoting the code itself, the short version of what's there is the following: if the Content-Type is NULL, then set the Content-Type to "application/octet-stream" and disable caching. In this case disabling caching means that both ETag and Last-Modified are neither calculated, nor sent with the response. But how to fix that? There were multiple approaches as lighttpd processes /etc/mime.types on Debian. One solution that works for each and every file is defining a fall-back value inside the configuration file. Why? Because the Content-Type will not be NULL when it reaches the check. tl;dr: Long story short, the fix was adding these three lines to the bottom of our lighttpd.conf:
mimetype.assign += ( "" => "application/octet-stream" )
