Comment 20 for bug 940825

This problem seems to appear when a file has been partially downloaded to /var/lib/apt/lists/partial and the server does not support request ranges. Here is the error I got when running "aptitude update" and the corresponding HTTP requests / responses:

[...]
Err http://de.archive.ubuntu.com precise-updates/universe i386 Packages
  406 Not Acceptable [IP: 141.30.13.20 80]
[...]
W: Failed to fetch http://de.archive.ubuntu.com/ubuntu/dists/precise-updates/universe/binary-i386/Packages 406 Not Acceptable [IP: 141.30.13.20 80]

E: Some index files failed to download. They have been ignored, or old ones used instead.

The HTTP requests and responses for this file are:

GET /ubuntu/dists/precise-updates/universe/binary-i386/Packages.bz2 HTTP/1.1
Host: de.archive.ubuntu.com
Connection: keep-alive
Cache-Control: max-age=0
Range: bytes=15556-
If-Range: Fri, 06 Sep 2013 23:35:00 GMT
User-Agent: Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.12)

HTTP/1.1 416 Requested Range Not Satisfiable
Date: Sat, 07 Sep 2013 17:51:48 GMT
Connection: keep-alive
Server: ATS/3.0.5
Content-Length: 0

GET /ubuntu/dists/precise-updates/universe/binary-i386/Packages.gz HTTP/1.1
Host: de.archive.ubuntu.com
Connection: keep-alive
Cache-Control: max-age=0
Range: bytes=15556-
If-Range: Fri, 06 Sep 2013 23:35:00 GMT
User-Agent: Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.12)

HTTP/1.1 416 Requested Range Not Satisfiable
Date: Sat, 07 Sep 2013 17:51:48 GMT
Connection: keep-alive
Server: ATS/3.0.5
Content-Length: 0

GET /ubuntu/dists/precise-updates/universe/binary-i386/Packages HTTP/1.1
Host: de.archive.ubuntu.com
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/*
Range: bytes=15556-
If-Range: Fri, 06 Sep 2013 23:35:00 GMT
User-Agent: Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.12)

HTTP/1.1 406 Not Acceptable
Date: Sat, 07 Sep 2013 17:51:48 GMT
Server: ATS/3.0.5
Alternates: {"Packages.bz2" 1 {type application/x-bzip2} {length 219035}}, {"Packages.gz" 1 {type application/x-gzip} {length 279396}}
Vary: negotiate,accept,Accept-Encoding
TCN: list
Content-Length: 576
Content-Type: text/html; charset=iso-8859-1
Age: 0
Connection: keep-alive

[... response body omitted ...]

In this case the file /var/lib/apt/lists/partial/de.archive.ubuntu.com_ubuntu_dists_precise-updates_universe_binary-i386_Packages had a size of 15557 bytes.

So I think the following thing happens: Apt finds a partially downloaded file, so it tries to continue downloading this file by sending a request with the "Range" header set. The server cannot serve this request (either because it does not support ranges or because the specified range is invalid). Thus it sends a 416 response. Now Apt makes a mistake: Instead of sending the request again without the "Range" header, it proceeds to the next file (Packages.gz after Packages.bz2 failed and finally Packages). Funnily it uses the same range there.

I think there are two bugs regarding the download of partially download files:

1) The start for the "Range" header is calculated from the file size on disk. However correct start depends on the type of the compression used. In my example, the file on disk was gzip compressed but the bzip2 version is requested first. Obviously Apt should only try to continue downloading a file with the same compression.
2) When a 416 error occurs, Apt proceeds to the next type of compression instead of trying to download the complete file again. This leads to problems when the server does not support ranges or the range specified is invalid for some reason. Apt should detect a 416 error and proceed with trying to download the complete file.

Maybe someone knowing the Apt code better can dig into it and find the place where these changes need to be made.

As a workaround, when you are experiencing a 406 error, you can delete the corresponding file(s) in /var/lib/apt/lists/partial. This fixed the problem in my case.