apt-ftparchive on-disk cache format changed between lucid and precise, results in Packages files with silently corrupted checksums fields

I will investigate this issue, but this smells more like a issue with libdb than with apt-ftparchive itself. But that might be wishful thinking, I will double check.

Its not the fault of libdb, its the fault of commit 650faab01603caac04494d54cf6b10a65c00ea13

Here is the reason for the bug:
--- apt-0.7.25.3ubuntu7/ftparchive/cachedb.h
+++ apt-0.8.16~exp12ubuntu10/ftparchive/cachedb.h
...
    struct StatStore
    {
       uint32_t Flags;
       uint32_t mtime;
- uint32_t FileSize;
+ uint64_t FileSize;
       uint8_t MD5[16];
       uint8_t SHA1[20];
       uint8_t SHA256[32];
+ uint8_t SHA512[64];
    } CurStat;
...

Adding the sha512 at the end is fine as there is a flag for the hashes so even if it reads garbage it will not use the not-yet-generated sha512 hash.

But the change of FileSize made it incompatibel.

Here is a fix for the bug. Its a bit heavy handed and will discard the record if the on-disk size and the struct size mismatch. This is not strictly needed if the struct StatStore is just appended and the FlagList is correctly used. It is in this case as the struct changed size right in the middle.

This patch adds compatibility with the previous on-disk format. The code is slightly more complex but the already calculated hashes can be re-used and do not need to be calculated again.

This will also affect debian squeeze to wheezy upgrades. Squeeze uses the old format with a 32bit FileSize and wheezy the new 64bit size.

See also https://github.com/mvo5/apt/commits/bugfix/lp1274466-cache for a updated version with regression test and all.

This is now in utopic

Hello Steve, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.8.16~exp12ubuntu10.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Steve, or anyone else affected,

Accepted apt into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/1.0.1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This SRU has been shadowed by a security update and needs to be re-merged.

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release