Please compress packages with zstd by default

# file perl-base_5.32.1-3ubuntu2_amd64.deb
perl-base_5.32.1-3ubuntu2_amd64.deb: Debian binary package (format 2.0), with control.tar.zs, data compression zst

Looks odd

# display 3rd character of file name extension like 2 of bz2 or m of lzma
>>>>>&-1 ubyte x \b%c
>>>>>>&0 ubyte !0x20
>>>>>>>&-1 ubyte !0x2f

is odd.

Debootstrap fix is merged in Debian:
https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/54

I don't understand what python-apt does here, it inherits zstd support from apt.

There seems to be a bug in bionic's apt.

root@bb:/home/jak/Projects# python3 -c 'import apt_inst, apt_pkg; apt_pkg.init(); apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2_zstd1_all.deb").control.extractdata("control")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
LookupError: There is no member named 'control'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 25, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 35, in <module>
    apt_pkg.init_system()
apt_pkg.Error: E:Cannot find a configured compressor for 'false'

I have not tested hirsute in a clean container; but touch /usr/bin/zstd makes it work, so the compressor lookup is broken.

@juliank Yes, python-apt works with just touched /usr/bin/zstd :

>>> apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()
True

@juliank Xenial's apt/python3-apt does not work even with zstd installed:

root@x-zstd:~# python3
Python 3.5.2 (default, Jan 26 2021, 13:30:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import apt_inst, apt_pkg
>>> apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: E:Internal error, could not locate member control.tar.{lz4gzxzbz2lzma}
>>>
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================-=======================-=======================-================================================================================
ii zstd 1.3.1+dfsg-1~ubuntu0.16 amd64 fast lossless compression algorithm -- CLI tool

Aside from the code being wrong - it's missing an apt_pkg.init_config(), it also does not work with the config being setup; it needs to be whitelisted.

So we definitely need to SRU APT. However, we are already preparing 1.2.35 at the moment for ESM enablement, I don't know if we want to stuff both things together.

@juliank IMO it would be better to add the zstd-related changes, too, to the planned SRU if those are not risky and the delay due to adding them is acceptable for ESM's perspective.

Split out APT / python-apt into bug 1923845

This bug was fixed in the package debootstrap - 1.0.124

---------------
debootstrap (1.0.124) experimental; urgency=medium

  [ Colin Watson ]
  * Add (Ubuntu) groovy as a symlink to gutsy.

  [ Samuel Thibault ]
  * functions: Turn for loops variables into locals, suggested by jhcha54008.

  [ YunQiang Su ]
  * stage1: re-mkdir /proc if it is a symlink instead of umount. (Closes: #968927)

  [ Luca Falavigna ]
  * Add (Ubuntu) hirsute as a symlink to gutsy.

  [ Tobias Koch ]
  * Add --inrelease-path option to allow specifying by-hash InRelease
    location.

  [ Dimitri John Ledkov ]
  * functions: add zstd control&data support to the ar extractor LP: #1923845
  * Add (Ubuntu) impish as a symlink to gutsy.
  * gutsy: sync changes from Ubuntu, to ensure that impish+ are always
    merged-usr.
  * gutsy: use --extractor=ar if dpkg-deb has no zstd support (i.e. on
    Debian & non-Ubuntu derivatives).
  * rules,control: adjust Recommends/Suggests to be appropriate on each
    distro.

 -- Dimitri John Ledkov <email address hidden> Mon, 26 Apr 2021 15:02:44 +0100

Verified 1.6.5ubuntu0.6 on Bionic.

root@bb-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:29:17-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.80MB/s in 1.1s

2021-05-13 19:29:18 (4.80 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@bb-lp1926437:~# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import apt_inst
>>> import apt_pkg
>>> apt_pkg.init_config()
>>> apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()
True
>>>
root@bb-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================-=======================-=======================-================================================================================
ii python-apt-common 1.6.5ubuntu0.6 all Python interface to libapt-pkg (locales)
ii python3-apt 1.6.5ubuntu0.6 amd64 Python 3 interface to libapt-pkg
root@bb-lp1926437:~#

Verified 2.2.0~ubuntu0.21.04.1 on Hirsute.

root@hh-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:42:59-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.78MB/s in 1.1s

2021-05-13 19:43:00 (4.78 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@hh-lp1926437:~# python3 -c 'import apt_inst; import apt_pkg; apt_pkg.init_config(); print(apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall())'
True
root@hh-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================-=====================-============-========================================
ii python-apt-common 2.2.0~ubuntu0.21.04.1 all Python interface to libapt-pkg (locales)
ii python3-apt 2.2.0~ubuntu0.21.04.1 amd64 Python 3 interface to libapt-pkg

Verified 2.1.3ubuntu1.4 on Groovy.

root@gg-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:51:34-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.70MB/s in 1.1s

2021-05-13 19:51:36 (4.70 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@gg-lp1926437:~# python3 -c 'import apt_inst; import apt_pkg; apt_pkg.init_config(); print(apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall())'
True
root@gg-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================-==============-============-========================================
ii python-apt-common 2.1.3ubuntu1.4 all Python interface to libapt-pkg (locales)
ii python3-apt 2.1.3ubuntu1.4 amd64 Python 3 interface to libapt-pkg

Verified 2.0.0ubuntu0.20.04.5 on Focal.

root@f-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:56:02-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.61MB/s in 1.2s

2021-05-13 19:56:03 (4.61 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@f-lp1926437:~# python3 -c 'import apt_inst; import apt_pkg; apt_pkg.init_config(); print(apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall())'
True
root@f-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================-====================-============-========================================
ii python-apt-common 2.0.0ubuntu0.20.04.5 all Python interface to libapt-pkg (locales)
ii python3-apt 2.0.0ubuntu0.20.04.5 amd64 Python 3 interface to libapt-pkg

Sorry, those comments were for LP: #1926437, which is at least related. :-)

The attached dpkg patch has been tested in the following Bileto ticket:

https://bileto.ubuntu.com/#/ticket/4583

Lintian needs to be fixed first to ensure smooth migration. The software-properties failure does not seem to be related at first glance.

dpkg patch looks good to me.

did lintian patch locally and started to run the regression test suite locally to see if anything else is tripped up.

working on reprepro changes too.

lintian should be good in impish-proposed, but we should migrate it first.

This bug was fixed in the package lintian - 2.104.0ubuntu3

---------------
lintian (2.104.0ubuntu3) impish; urgency=medium

  * Add zst as allowed compression for .deb & .udeb. Although we don't
    actually ship udeb package that supports zst. LP: #1923845

 -- Dimitri John Ledkov <email address hidden> Tue, 15 Jun 2021 10:42:12 +0100

@xnox Thanks! Please also SRU the reprepro changes. Ideally users of reprepro on stable releases would be able to just upgrade reprepro from -updates before the first zstd-compressed .deb lands in Impish to not break their setup even when they maintain a local repository of the devel series.

This bug was fixed in the package hello - 2.10-2ubuntu3

---------------
hello (2.10-2ubuntu3) impish; urgency=medium

  * Ubuntu Impish will be switching default dpkg compression format to
    Zstd. This upload is the first canary to signal the archive format
    change. dpkg in Ubuntu supports zstd in xenial and up. If any issues
    are identified with this package, please report them at
    https://pad.lv/1923845 LP: #1923845

 -- Dimitri John Ledkov <email address hidden> Fri, 18 Jun 2021 11:58:54 +0100

This bug was fixed in the package reprepro - 5.3.0-1.3

---------------
reprepro (5.3.0-1.3) experimental; urgency=medium

  * Non-maintainer upload.
  * Add support for Zstd compressed debs. (LP: #1923845)

 -- Dimitri John Ledkov <email address hidden> Mon, 21 Jun 2021 10:16:53 +0100

Proposed diffoscope fix: https://salsa.debian.org/reproducible-builds/diffoscope/-/merge_requests/83

Diffoscope already handles zstd compressed members well.

Hello Balint, or anyone else affected,

Accepted reprepro into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/reprepro/5.3.0-1.3~ubuntu21.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Balint, or anyone else affected,

Accepted reprepro into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/reprepro/5.3.0-1.3~ubuntu20.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Balint, or anyone else affected,

Accepted reprepro into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/reprepro/5.3.0-1.3~ubuntu20.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted reprepro (5.3.0-1.3~ubuntu20.04) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

dgit/9.10 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#reprepro

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

See reprepro verification comments at https://bugs.launchpad.net/ubuntu/+source/reprepro/+bug/1933363

@ubuntu-sru-bot

regressions triggered by reprepro have been retried, and have been now cleared.

This bug was fixed in the package dpkg - 1.20.9ubuntu2

---------------
dpkg (1.20.9ubuntu2) impish; urgency=medium

  * build: Switch default dpkg-deb compression from xz to zstd. (LP: #1923845)
    Keep compressing dpkg.deb with xz to help bootstrapping on non-Ubuntu
    systems.

 -- Balint Reczey <email address hidden> Mon, 14 Jun 2021 16:04:10 +0200

The verification of the Stable Release Update for reprepro has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package reprepro - 5.3.0-1.3~ubuntu20.04

---------------
reprepro (5.3.0-1.3~ubuntu20.04) focal; urgency=medium

  * Backport Zstd compressed debs support to hirsute. LP: #1933363

reprepro (5.3.0-1.3) experimental; urgency=medium

  * Non-maintainer upload.
  * Add support for Zstd compressed debs. (LP: #1923845)

reprepro (5.3.0-1.2) unstable; urgency=medium

  * Non-maintainer upload.
  * Flush stdout and stderr before execv of an end hook (Closes: #982423)

 -- Dimitri John Ledkov <email address hidden> Wed, 23 Jun 2021 17:01:44 +0100

This bug was fixed in the package reprepro - 5.3.0-1.3~ubuntu20.10

---------------
reprepro (5.3.0-1.3~ubuntu20.10) groovy; urgency=medium

  * Backport Zstd compressed debs support to hirsute. LP: #1933363

reprepro (5.3.0-1.3) experimental; urgency=medium

  * Non-maintainer upload.
  * Add support for Zstd compressed debs. (LP: #1923845)

reprepro (5.3.0-1.2) unstable; urgency=medium

  * Non-maintainer upload.
  * Flush stdout and stderr before execv of an end hook (Closes: #982423)

 -- Dimitri John Ledkov <email address hidden> Wed, 23 Jun 2021 17:01:44 +0100

This bug was fixed in the package reprepro - 5.3.0-1.3~ubuntu21.04

---------------
reprepro (5.3.0-1.3~ubuntu21.04) hirsute; urgency=medium

  * Backport Zstd compressed debs support to hirsute. LP: #1933363

reprepro (5.3.0-1.3) experimental; urgency=medium

  * Non-maintainer upload.
  * Add support for Zstd compressed debs. (LP: #1923845)

 -- Dimitri John Ledkov <email address hidden> Wed, 23 Jun 2021 17:01:44 +0100

This bug was fixed in the package diffoscope - 179

---------------
diffoscope (179) unstable; urgency=medium

  * Ensure that various LLVM tools are installed, even when testing whether
    a MacOS binary has zero differences when compared to itself.
    (Closes: reproducible-builds/diffoscope#270)

 -- Chris Lamb <email address hidden> Fri, 30 Jul 2021 09:05:01 +0100

dh-cmake currently FTBFS [1].

While investigating the issue, I realized that, now dpkg-deb defaults to compressing with zstd, python-debian can no longer decompress the compressed data into the binary package archive.

I am proposing a patch for python-debian in the following MP: https://code.launchpad.net/~athos-ribeiro/ubuntu/+source/python-debian/+git/python-debian/+merge/407413

A PPA with the proposed fix is available at https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp-1923845-python-debian/+packages

[1] https://launchpadlibrarian.net/552708462/buildlog_ubuntu-impish-amd64.dh-cmake_0.6.1_BUILDING.txt.gz

This bug was fixed in the package python-debian - 0.1.39ubuntu1

---------------
python-debian (0.1.39ubuntu1) impish; urgency=medium

  * Add support for zstd compression format (LP: #1923845)

 -- Athos Ribeiro <email address hidden> Thu, 19 Aug 2021 14:50:49 -0300

FYI I added initial support for Lutris here (https://github.com/lutris/lutris/pull/4021).
However this is really only cosmetics, it won't work until an stdlib zstd module is available in Python (see https://<email address hidden>/thread/VQIFA7WTNRAOYZGTVP4WZC2CD36KYIVY/#RMMZFYPKHDJHGSHCKHRKRDGKPZSCUHT2).

aptly is Fix Release in Kinetic in:

aptly (1.4.0+ds1-7) unstable; urgency=medium

  * Team upload.
  * Add support for zstd compression (Closes: #1010465)