do-release-upgrade fails if ESM enabled

Bug #1990798 reported by Ken Sharp

This bug report was marked for expiration 228 days ago. (find out why)

24
This bug affects 5 people
Affects Status Importance Assigned to Milestone
ubuntu-advantage-tools (Ubuntu)
Incomplete
Undecided
Unassigned
ubuntu-release-upgrader (Ubuntu)
Incomplete
Critical
Unassigned

Bug Description

Trying to run a do-release-upgrade from Focal to Jammy fails if ESM is enabled:

Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/a/ansible/ansible_2.10.7+merged+base+2.10.8+dfsg-1ubuntu0.1~esm1_all.deb
401 Unauthorized [IP: 185.125.190.23 443]
Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/o/openexr/libopenexr25_2.5.7-1ubuntu0.1~esm1_amd64.deb
401 Unauthorized [IP: 185.125.190.23 443]
Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/p/python2.7/python2.7-dev_2.7.18-13ubuntu1.1+esm2_amd64.deb
401 Unauthorized [IP: 185.125.190.23 443]
Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/p/python2.7/python2.7_2.7.18-13ubuntu1.1+esm2_amd64.deb
401 Unauthorized [IP: 185.125.190.23 443]
Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/p/python2.7/libpython2.7-dev_2.7.18-13ubuntu1.1+esm2_amd64.deb
401 Unauthorized [IP: 185.125.190.23 443]
Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/p/python2.7/libpython2.7_2.7.18-13ubuntu1.1+esm2_amd64.deb
401 Unauthorized [IP: 185.125.190.23 443]
Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/p/python2.7/libpython2.7-stdlib_2.7.18-13ubuntu1.1+esm2_amd64.deb
401 Unauthorized [IP: 185.125.190.23 443]
Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/p/python2.7/python2.7-minimal_2.7.18-13ubuntu1.1+esm2_amd64.deb
401 Unauthorized [IP: 185.125.190.23 443]
Failed to fetch
https://esm.ubuntu.com/apps/ubuntu/pool/main/p/python2.7/libpython2.7-minimal_2.7.18-13ubuntu1.1+esm2_amd64.deb
401 Unauthorized [IP: 185.125.190.23 443]

SERVICE ENTITLED STATUS DESCRIPTION
esm-infra yes enabled Expanded Security Maintenance for Infrastructure
fips yes disabled NIST-certified core packages
fips-updates yes disabled NIST-certified core packages with priority security updates
livepatch yes enabled Canonical Livepatch service
usg yes disabled Security compliance and audit tools

Unfortunately running `pro disable esm-infra` does, apparently, nothing and so the workaround is to comment out the relevant lines in the Apt sources. Then this has to be reversed when the upgrade is complete. This shouldn't be necessary.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: ubuntu-release-upgrader-core 1:20.04.39
ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210
Uname: Linux 5.4.0-128-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu27.24
Architecture: amd64
CasperMD5CheckResult: skip
CrashDB: ubuntu
CurrentDesktop: MATE
Date: Mon Sep 26 03:27:47 2022
PackageArchitecture: all
SourcePackage: ubuntu-release-upgrader
Symptom: ubuntu-release-upgrader
UpgradeStatus: Upgraded to focal on 2022-09-26 (0 days ago)
mtime.conffile..etc.apport.crashdb.conf: 2019-08-06T11:56:22.315382

Revision history for this message
Ken Sharp (kennybobs) wrote :
Revision history for this message
Thomas Mustafa (kismet22) wrote :

I have the exact same problem. I have attached a screenshot of the message that I received.

Unfortunately, I am not skilled with computers and could not understand your work around. Can you please detail it a bit more. What file did you change and how? Where can I find this file?

Much obliged.

Revision history for this message
José Relland (jrelland) wrote (last edit ):

Hi Thomas Mustafa, and the Ubuntu Team,

Exactly the same issue

This solution is here (worked for me) : https://ubuntu.com/pro/beta#
A Ubuntu Pro account has been open but any service.

Create a service (free) for this desktop.

And enjoy your 22.04 upgrade :)

See this: https://askubuntu.com/questions/1437617/22-04-upgrade-could-not-download-the-upgrades-unauthorized-ip-200167c1
thank you Thomas Ward :).

José from France.

tags: added: foundations-triage-discuss
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu-release-upgrader (Ubuntu):
status: New → Confirmed
tags: removed: foundations-triage-discuss
Changed in ubuntu-release-upgrader (Ubuntu):
importance: Undecided → Critical
Revision history for this message
kevin wang (nosocks) wrote :

I got the same 401 errors running apt-get upgrade from 18.04

The following packages will be upgraded:
  libnghttp2-14 libpython2.7 libpython2.7-dev libpython2.7-minimal
  libpython2.7-stdlib libpython3.6 libpython3.6-dev libpython3.6-minimal
  libpython3.6-stdlib python2.7 python2.7-dev python2.7-minimal python3.6
  python3.6-dev python3.6-minimal python3.6-venv sysstat vim vim-common
  vim-runtime vim-tiny xxd
...
E: Failed to fetch https://esm.ubuntu.com/infra/ubuntu/pool/main/s/sysstat/sysstat_11.6.1-1ubuntu0.2+esm1_amd64.deb 401 Unauthorized [IP: 91.189.91.47 443]

ran pro disable esm-infra

ran apt-get upgrade, saw that it no longer listed those packages.

ran pro enable esm-infra

ran apt-get upgrade, and now it is downloading the packages in question.

Note that I have not rebooted since adding the free Ubuntu Pro token.

tags: added: rls-jj-incoming
Revision history for this message
Julian Andres Klode (juliank) wrote :

This confuses me from the u-r-u/apt perspective because either the system is enrolled and the sources are there or it's not and the sources shouldn't be there.

Adding an ubuntu-advantage-tools task for further investigation and marking ubuntu-release-upgrader as Incomplete

Changed in ubuntu-advantage-tools (Ubuntu):
status: New → Incomplete
status: Incomplete → New
Changed in ubuntu-release-upgrader (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Renan Rodrigo (renanrodrigo) wrote :

Thanks Julian for the redirect
We do have upgrade-related integration tests in place and passing for all releases, so this seems indeed a non-catched bug. Will investigate further.

Revision history for this message
Renan Rodrigo (renanrodrigo) wrote :

I cannot reproduce this issue - I have executed upgrades ( X -> B -> F -> J ) with all esm services enabled and all worked.

From a theoretical perspective, 401s should be there only if the esm credentials are invalid. Note that apt update will work, as only the packages themselves need the auth to work to be accessed.

I will try it out on desktop though, to see if there is any problem or discrepance.

To further investigate, it would be nice to have some logs from our side to look at. Anyone having this problem could please run `pro collect-logs` and attach the resulting tarball?

Changed in ubuntu-advantage-tools (Ubuntu):
status: New → Incomplete
Revision history for this message
Renan Rodrigo (renanrodrigo) wrote :

Tried desktop, F-> J works fine

Revision history for this message
Ken Sharp (kennybobs) wrote :

Is it possible to check if those packages existed at the time this was logged?

What happens if a package is missing? Does it still give 401 and fail?

I can't do either of those myself.

Revision history for this message
Lucas Albuquerque Medeiros de Moura (lamoura) wrote :

Hi kennybobs,

I don't think the 401 are related to missing packages. It seems that your issue is related to
the esm-apps APT source file being present, but not together with its associated preference file.

However, as is, we cannot reproduce this issue and we would need the logs renanrodrigo mentioned to better investigate why this happened in the first place

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.