Re-publishing binaries with the same name and version confuses apt-ftparchive

It's know and desired that soyuz doesn't become very pedantic about binary versions, they are not strongly controlled as source versions. It's doesn't control uniqueness, installability or source coherency of binary versions.

There is no special reason behind this decision, expect the fact that that kind of check isn't trivial to implement and vast majority of the problems is already caught on source upload time.

However this cycle, this area was hit by a severe problem when a security upload was done in gutsy producing binaries with a version that was already used for a deleted gutsy-proposed upload.

If we look at https://edge.launchpad.net/ubuntu/+source/hplip/, we will see that there are two distinct source uploads with version 2.7.7.dfsg.1-0ubuntu5.1.

The fact is completely masked by the LP distribution UI, https://edge.launchpad.net/ubuntu/+source/hplip/2.7.7.dfsg.1-0ubuntu5.1 shows only the most recent set of publications (gutsy-security and gutsy-updates), but there were two of them.

The copy package infrastructure was designed to ignore deleted copies and copy the proposed source and binary over it and it seems to be okay from the installability point of view.

The problem is that apt-ftparchive is holding all caches for every file ever published using soyuz and by already having a cache for the published files it refused to generate new ones when the new set of binaries was published. It resulted in cheksum mismatch between the files on disk and the actual archive index.

There are three separated issues that should be addressed in this bug.

First, apt-ftparchive caches needs to be sanitized periodically, there is not point in keep all caches-ever on 500Mb bdb files, its performance is certainly reduced by that. A separated infrastructure similar to the one used to generate Content files has to be set up (it can't run simultaneous with the publisher) and apt-archive itself has a method to clean the caches up.

Second, we have to decide if what copy-package is doing right now is the correct policy. Ignoring deleted publications in the destination scenario is certainly handy for avoiding unnecessary version changes, but may not be the safest approach to take.

Third, and optional task, the distribution navigation design makes it really hardy to visualize those problems, if decided to keep the copy behavior, we will have to extend the UI to allow access to the 'hidden' deleted publications.