python3-minimal should predepend on versioned version of python3-minimal

The dpkg --configure -a and apt-get -f install dance brought me to this point:

$ sudo dpkg --configure -a
dpkg: dependency problems prevent configuration of rhythmbox-plugins:
 rhythmbox-plugins depends on python3 (>= 3.6~); however:
  Version of python3 on system is 3.5.1-3.

dpkg: error processing package rhythmbox-plugins (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libgoa-backend-1.0-1:amd64:
 libgoa-backend-1.0-1:amd64 depends on libgoa-1.0-0b (>= 3.25.4); however:
  Version of libgoa-1.0-0b:amd64 on system is 3.18.3-1ubuntu2.

dpkg: error processing package libgoa-backend-1.0-1:amd64 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of python3-apt:
 gnupg (2.2.4-1ubuntu1) breaks python3-apt (<= 1.1.0~beta4) and is installed.
  Version of python3-apt to be configured is 1.1.0~beta1ubuntu0.16.04.1.

dpkg: error processing package python3-apt (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of rhythmbox-plugin-magnatune:
 rhythmbox-plugin-magnatune depends on rhythmbox-plugins; however:
  Package rhythmbox-plugins is not configured yet.

dpkg: error processing package rhythmbox-plugin-magnatune (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of python3-gdbm:amd64:
 python3-gdbm:amd64 depends on python3 (>= 3.6.4-1~); however:
  Version of python3 on system is 3.5.1-3.

dpkg: error processing package python3-gdbm:amd64 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of python3-reportlab-accel:amd64:
 python3-reportlab-accel:amd64 depends on python3 (>= 3.6~); however:
  Version of python3 on system is 3.5.1-3.

dpkg: error processing package python3-reportlab-accel:amd64 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of python3-cairo:amd64:
 python3-cairo:amd64 depends on python3 (>= 3.6~); however:
  Version of python3 on system is 3.5.1-3.

dpkg: error processing package python3-cairo:amd64 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of python-apt:
 python-apt depends on libapt-inst2.0 (>= 1.4~beta3); however:
  Version of libapt-inst2.0:amd64 on system is 1.2.26.
 python-apt depends on libapt-pkg5.0 (>= 1.4~beta3); however:
  Version of libapt-pkg5.0:amd64 on system is 1.2.26.

dpkg: error processing package python-apt (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of onboard:
 onboard depends on python3-cairo; however:
  Package python3-cairo:amd64 is not configured yet.
 onboard depends on python3 (>= 3.6~); however:
  Version of python3 on system is 3.5.1-3.

dpkg: error processing package onboard (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of software-properties-common:
 gnupg (2.2.4-1ubuntu1) breaks software-properties-common (<= 0.96.24.3) and is installed.
  Version of software-properties-com...

Liberal application of dpkg -i /var/cache/apt/archives/.... has gotten me much further, but this is a real sticking point:

225 upgraded, 122 newly installed, 315 to remove and 2028 not upgraded.
16 not fully installed or removed.
Need to get 0 B/254 MB of archives.
After this operation, 389 MB disk space will be freed.
Do you want to continue? [Y/n]
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 374078 files and directories currently installed.)
Removing systemd-shim (9-1bzr4ubuntu1) ...
Removing 'diversion of /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service to /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.systemd by systemd-shim'
dpkg-divert: error: rename involves overwriting '/usr/share/dbus-1/system-services/org.freedesktop.systemd1.service' with
  different file '/usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.systemd', not allowed
dpkg: error processing package systemd-shim (--remove):
 subprocess installed post-removal script returned error exit status 2
Errors were encountered while processing:
 systemd-shim
E: Sub-process /usr/bin/dpkg returned an error code (1)

That diversion does exist:

$ dpkg-divert --list | grep shim
diversion of /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service to /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.systemd by systemd-shim

and those files exist, with different contents:

ebf27f9c1e4b77939498616a1500e6a2cdb00862c116294d9844f5ddef8d0ca7 /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service
37a68d852e39114b9691001fc91c9db098e506e077a324329c46159020ab3f39 /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.systemd

On tsimonq2's advice (with proper caveats) deleting /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service let apt make progress.

I've had to install a lot of packages by hand to get apt and dpkg to continue making progress. I'm skeptical of the state of my system:

- ntp was still installed
- chrony was not installed
- netplan.io was not installed
- squid-deb-proxy was uninstalled
- rxvt-unicode was uninstalled

Thanks

Status changed to 'Confirmed' because the bug affects multiple users.

Thanks for your report.

Could you please attach the files from the directory /var/log/dist-upgrade/ ?

Do you know why this might have failed?

Preparing to unpack .../python3-apt_1.6.0_amd64.deb ...
/var/lib/dpkg/info/python3-apt.prerm: 6: /var/lib/dpkg/info/python3-apt.prerm: py3clean: not found
dpkg: warning: subprocess old pre-removal script returned error exit status 127
dpkg: trying script from the new package instead ...
/var/lib/dpkg/tmp.ci/prerm: 6: /var/lib/dpkg/tmp.ci/prerm: py3clean: not found
dpkg: error processing archive /var/cache/apt/archives/python3-apt_1.6.0_amd64.deb (--unpack):
 subprocess new pre-removal script returned error exit status 127
/var/lib/dpkg/info/python3-apt.postinst: 6: /var/lib/dpkg/info/python3-apt.postinst: py3compile: not found
dpkg: error while cleaning up:
 subprocess installed post-installation script returned error exit status 127

On Wed, May 02, 2018 at 04:40:44PM -0000, Brian Murray wrote:
> Do you know why this might have failed?
>
> Preparing to unpack .../python3-apt_1.6.0_amd64.deb ...
> /var/lib/dpkg/info/python3-apt.prerm: 6: /var/lib/dpkg/info/python3-apt.prerm: py3clean: not found
> dpkg: warning: subprocess old pre-removal script returned error exit status 127

Sadly, no :( I can't think of anything particularly special or different
about my Python packages. Aside from the huge number of deb-src lines
in my APT sources this system is pretty standard.

No pips, no virtualenvs, nothing fancy.

Thanks

I finished searching my audit logs and found nothing out of the ordinary -- the only AppArmor DENIED entries from around the time of the upgrade are from Firefox and Pulseaudio. (It was fun to see dpkg installing new timezone files tripping my audit logs for time handling.)

Thanks

I had the same problem, and I solved it in the same way, namely by renaming /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service

I'm hitting very similar nastiness here... I'm not very happy with the state of this system :-/

Hi Dustin, did you happen to notice where your upgrade went off the rails? We haven't figured out what might have been unique on my system to cause my upgrade failures but the py3clean error message seems most likely to be involved.

Are you currently stuck with an unhappy apt or dpkg?

Thanks

This bug looks like a duplicate of bug 1763844. My last comment there was:

 py3clean is only executed if which tells us it exists:

 if which py3clean >/dev/null 2>&1; then
         py3clean -p python3-apt
 else

 hence it's quite strange.

I don't see how that code can fail (py3clean is in the same package as python3, hence the interpreter exists if py3clean exists), but if it does, _all_ python packages are affected.

It's unclear to me if the systemd failure would appear normally, or if it's caused by fixing up the failures, as the system might be in an inconsistent state.

python3-minimal should predepend on whatever the current version of python3 minimal is e.g. python3.6-minimal.

please be careful with pre-dependencies. These tend to break other stuff ... why do you think that the pre-dependency will solve things here?

Minimal test case, unpacking python3-minimal and python3-apt from artful on xenial:

root@lp1768379:~# dpkg --unpack /home/jak/Downloads/python3-minimal_3.6.3-0ubuntu2_amd64.deb
(Reading database ... 25653 files and directories currently installed.)
Preparing to unpack .../python3-minimal_3.6.3-0ubuntu2_amd64.deb ...
Unpacking python3-minimal (3.6.3-0ubuntu2) over (3.5.1-3) ...
Processing triggers for man-db (2.7.5-1) ...
root@lp1768379:~# dpkg --unpack /home/jak/Downloads/python3-apt_1.4.0_beta3build2_amd64.deb
(Reading database ... 25653 files and directories currently installed.)
Preparing to unpack .../python3-apt_1.4.0_beta3build2_amd64.deb ...
/var/lib/dpkg/info/python3-apt.prerm: 6: /var/lib/dpkg/info/python3-apt.prerm: py3clean: not found
dpkg: warning: subprocess old pre-removal script returned error exit status 127
dpkg: trying script from the new package instead ...
/var/lib/dpkg/tmp.ci/prerm: 6: /var/lib/dpkg/tmp.ci/prerm: py3clean: not found
dpkg: error processing archive /home/jak/Downloads/python3-apt_1.4.0_beta3build2_amd64.deb (--unpack):
 subprocess new pre-removal script returned error exit status 127
/var/lib/dpkg/info/python3-apt.postinst: 6: /var/lib/dpkg/info/python3-apt.postinst: py3compile: not found
dpkg: error while cleaning up:
 subprocess installed post-installation script returned error exit status 127
Errors were encountered while processing:
 /home/jak/Downloads/python3-apt_1.4.0_beta3build2_amd64.deb

Hello Seth, or anyone else affected,

Accepted python3-defaults into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3-defaults/3.6.5-3ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Seth, or anyone else affected,

Accepted python3-defaults into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3-defaults/3.6.3-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

hello Łukasz Zemczak
i am new in Ubuntu. but i have the same problem. could you help me resolve it?

I have Ubuntu 18.04 and python3.6. And few apps which cant work without solving this bug.

При обработке следующих пакетов произошли ошибки:
 python3
 update-notifier-common
 gnome-shell
 update-notifier
 gnome-menus
E: Sub-process /usr/bin/dpkg returned an error code (1)

i have added -proposed repository. When i try upgrade i have this mistake: installed python3 package post-installation script subprocess returned error exit status 4
 and so on.

Could you help me please?

Hello Nataly, try this, all as root (sudo before each command, or one sudo -s before them all..)

apt-get update
apt-get install python3.6-minimal
apt-get install python3-minimal
apt-get install python3
apt-get install -f

Thanks

To verify this I actually had to edit my sources.list from xenial to artful so I could first install python3.6-minimal which depended on libc6 before I could unpack python3-minimal and python3-apt. After that it worked though.

bdmurray@clean-xenial-amd64:~$ sudo dpkg --unpack python3-minimal_3.6.3-0ubuntu3_amd64.deb
(Reading database ... 209665 files and directories currently installed.)
Preparing to unpack python3-minimal_3.6.3-0ubuntu3_amd64.deb ...
Unpacking python3-minimal (3.6.3-0ubuntu3) over (3.5.1-3) ...
Processing triggers for man-db (2.7.5-1) ...
bdmurray@clean-xenial-amd64:~$ sudo dpkg --unpack python3-apt_1.4.0~beta3build2_amd64.deb
(Reading database ... 209665 files and directories currently installed.)
Preparing to unpack python3-apt_1.4.0~beta3build2_amd64.deb ...
Unpacking python3-apt (1.4.0~beta3build2) over (1.1.0~beta1build1) ...

I also performed an upgrade of an Ubuntu Desktop system from xenial w/ -proposed enabled to artful so that python3-minimal from artful-proposed would be used and the upgrade proceeded without error.

bdmurray@clean-xenial-amd64:/tmp/ubuntu-release-upgrader-355quvv3$ grep -C10 python3-minimal /var/log/dist-upgrade/apt-term.log
Processing triggers for mime-support (3.59ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.15-0ubuntu1) ...
Processing triggers for cups (2.1.3-4) ...
Updating PPD files for postscript-hp ...
Setting up libssl1.0.0:amd64 (1.0.2g-1ubuntu13.6) ...
Setting up libpython3.6-minimal:amd64 (3.6.3-1ubuntu1) ...
Setting up libexpat1:amd64 (2.2.3-1) ...
Setting up python3.6-minimal (3.6.3-1ubuntu1) ...
Processing triggers for libc-bin (2.26-0ubuntu2.1) ...
(Reading database ... 209379 files and directories currently installed.)
Preparing to unpack .../python3-minimal_3.6.3-0ubuntu3_amd64.deb ...
Unpacking python3-minimal (3.6.3-0ubuntu3) over (3.5.1-3) ...
Setting up python3-minimal (3.6.3-0ubuntu3) ...
(Reading database ... 209379 files and directories currently installed.)
Preparing to unpack .../python3_3.6.3-0ubuntu3_amd64.deb ...
running python pre-rtupdate hooks for python3.6...
Unpacking python3 (3.6.3-0ubuntu3) over (3.5.1-3) ...
Preparing to unpack .../mime-support_3.60ubuntu1_all.deb ...
Unpacking mime-support (3.60ubuntu1) over (3.59ubuntu1) ...
Selecting previously unselected package libpython3.6-stdlib:amd64.
Preparing to unpack .../libpython3.6-stdlib_3.6.3-1ubuntu1_amd64.deb ...
Unpacking libpython3.6-stdlib:amd64 (3.6.3-1ubuntu1) ...
Selecting previously unselected package python3.6.

I preformed and upgrade of an Ubuntu Desktop system from xenial w/ -proposed enabled to bionic (after hacking DistUpgradeController.py not to disable -proposed) so that python3-minimal from bionic-proposed would be used and the upgrade proceeded without error.

bdmurray@clean-xenial-amd64:~$ grep -C10 python3-minimal /var/log/dist-upgrade/apt-term.log
Selecting previously unselected package python3.6-minimal.
Preparing to unpack .../python3.6-minimal_3.6.5-3_amd64.deb ...
Unpacking python3.6-minimal (3.6.5-3) ...
Processing triggers for libglib2.0-0:amd64 (2.48.2-0ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up libssl1.1:amd64 (1.1.0g-2ubuntu4.1) ...
locale: Cannot set LC_ALL to default locale: No such file or directory
Setting up libpython3.6-minimal:amd64 (3.6.5-3) ...
Setting up python3.6-minimal (3.6.5-3) ...
(Reading database ... 246397 files and directories currently installed.)
Preparing to unpack .../python3-minimal_3.6.5-3ubuntu1_amd64.deb ...
Unpacking python3-minimal (3.6.5-3ubuntu1) over (3.5.1-3) ...
Selecting previously unselected package libreadline7:amd64.
Preparing to unpack .../libreadline7_7.0-3_amd64.deb ...
Unpacking libreadline7:amd64 (7.0-3) ...
Selecting previously unselected package libpython3.6-stdlib:amd64.
Preparing to unpack .../libpython3.6-stdlib_3.6.5-3_amd64.deb ...
Unpacking libpython3.6-stdlib:amd64 (3.6.5-3) ...
Selecting previously unselected package python3.6.
Preparing to unpack .../python3.6_3.6.5-3_amd64.deb ...
Unpacking python3.6 (3.6.5-3) ...
Preparing to unpack .../libpython3-stdlib_3.6.5-3ubuntu1_amd64.deb ...

To verify that python3-minimal and python3-apt could unpack I had to do the same bit of editing my sources from xenial to bionic to get python3.6-minimal installed, after that the verification succeeded.

bdmurray@clean-xenial-amd64:~$ sudo dpkg --unpack python3-minimal_3.6.5-3ubuntu1_amd64.deb
(Reading database ... 246796 files and directories currently installed.)
Preparing to unpack python3-minimal_3.6.5-3ubuntu1_amd64.deb ...
Unpacking python3-minimal (3.6.5-3ubuntu1) over (3.5.1-3) ...
Processing triggers for man-db (2.7.5-1) ...
bdmurray@clean-xenial-amd64:~$ sudo dpkg --unpack python3-apt_1.6.1_amd64.deb
(Reading database ... 246796 files and directories currently installed.)
Preparing to unpack python3-apt_1.6.1_amd64.deb ...
Unpacking python3-apt (1.6.1) over (1.1.0~beta1ubuntu0.16.04.1) ...

I also performed an upgrade of an Ubuntu 16.04 lxc container to artful w/ -proposed enabled without issue.

ubuntu@artful-upgrade:~$ grep -C10 python3-minimal /var/log/dist-upgrade/apt-term.log
Unpacking python3-cryptography (1.9-1) over (1.2.3-1ubuntu0.1) ...
Processing triggers for systemd (234-2ubuntu12.4) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for libc-bin (2.26-0ubuntu2.1) ...
Setting up libssl1.0.0:amd64 (1.0.2g-1ubuntu13.6) ...
Setting up libpython3.6-minimal:amd64 (3.6.3-1ubuntu1) ...
Setting up libexpat1:amd64 (2.2.3-1) ...
Setting up python3.6-minimal (3.6.3-1ubuntu1) ...
Processing triggers for libc-bin (2.26-0ubuntu2.1) ...
(Reading database ... 26272 files and directories currently installed.)
Preparing to unpack .../python3-minimal_3.6.3-0ubuntu3_amd64.deb ...
Unpacking python3-minimal (3.6.3-0ubuntu3) over (3.5.1-3) ...
Setting up python3-minimal (3.6.3-0ubuntu3) ...
(Reading database ... 26272 files and directories currently installed.)
Preparing to unpack .../python3_3.6.3-0ubuntu3_amd64.deb ...
running python pre-rtupdate hooks for python3.6...
Unpacking python3 (3.6.3-0ubuntu3) over (3.5.1-3) ...
Preparing to unpack .../install-info_6.4.90.dfsg.1-1build1_amd64.deb ...
Unpacking install-info (6.4.90.dfsg.1-1build1) over (6.1.0.dfsg.1-5) ...
Setting up install-info (6.4.90.dfsg.1-1build1) ...
(Reading database ... 26272 files and directories currently installed.)
Preparing to unpack .../readline-common_7.0-0ubuntu2_all.deb ...
Unpacking readline-common (7.0-0ubuntu2) over (6.3-8ubuntu2) ...

Brian, thanks so much for testing these cases -- it's certainly more work than the "usual" SRU case.

I performed the same upgrade of an Ubuntu 16.04 lxc container but to binoic w/ -proposed enabled (after hacking DistUpgradeController.py) and there were no issues.

Thanks for all the testing Brian! As with any python3 upload, this also seems to come with quite a lot reported autopkgtest regressions. Could you take a look at those and see if they're related to the changes or not (both bionic and artful)?

The following tests have a history of failures and shouldn't be considered regressions:

dulwich [bionic/s390x]
snapcraft [bionic/all arches]
python-mechanicalsoup [bionic/all arches] These LinkNotFoundErrors also occurred in cosmic and when the bionic/s390x version was triggered by pytest-mock/1.7.1-1 so its not related to my SRU.
netplan.io [bionic/all arches] test_eth_dhcp6_off seems flaky to me as its failed with other triggers

The following tests have a history of failures and shouldn't be considered regressions:

xandikos [artful/ppc64el] it regularly has autopkgtest timeouts for all arches and release
meson [artful/i386] sometimes passes but sometimes fails always in simdtest
network-manager [artful/arm64] almost never passes
pyfai [artful/all arches] fails more often than it passes on most arches
unattended-upgrades [bionic/arm64] has failed repeatedly
python-docker [artful/s390x] client timeout errors on everything but armhf firewall?
translate-toolkit [artful/all arches] seems unrelated given that it fails on all arches and nothing has triggered translate-toolkit in 8 or 9 months so infrastructure?

It's not clear to me what is going on in the ubuntu-make autopkgtest for artful/s390x but the failure isn't related to my SRU. From the log:

Investigating (0) autopkgtest-satdep:s390x < 0 @iU mK Nb Ib >
Broken autopkgtest-satdep:s390x Depends on ubuntu-desktop:s390x < none @un H >
  Removing autopkgtest-satdep:s390x because I can't find ubuntu-desktop:s390x
Done

There is no ubuntu-desktop for s390x in artful, so the failure makes sense.

The verification of the Stable Release Update for python3-defaults has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python3-defaults - 3.6.3-0ubuntu3

---------------
python3-defaults (3.6.3-0ubuntu3) artful; urgency=medium

  * Switch python3.x-minimal to a pre-dependency for python3-minimal so that
    release upgrades will be more smooth. (LP: #1768379)

 -- Brian Murray <email address hidden> Thu, 21 Jun 2018 15:50:41 -0700

This bug was fixed in the package python3-defaults - 3.6.5-3ubuntu1

---------------
python3-defaults (3.6.5-3ubuntu1) bionic; urgency=medium

  * Switch python3.x-minimal to a pre-dependency for python3-minimal so that
    release upgrades will be more smooth. (LP: #1768379)

 -- Brian Murray <email address hidden> Thu, 21 Jun 2018 14:52:37 -0700

This is in cosmic-proposed, but hasn't migrated to -updates yet.

python3-defaults (3.6.6-1) unstable; urgency=medium

  [ Matthias Klose ]
  * Fix the idle desktop file. Closes: #901096.
  * Install the policy files in the python3 package. Closes: #901486.
  * python3-minimal: Pre-Depend on python3.6-minimal. Closes: #901001.
  * Bump version to 3.6.6.

  [ Scott Kitterman ]
  * Remove myself from uploaders

 -- Matthias Klose <email address hidden> Wed, 27 Jun 2018 18:18:20 +0200