apport-retrace sandbox fails if mirror does not provide Contents-amd64.gz

Bug #1864995 reported by Esokrates
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apport
New
Undecided
Unassigned

Bug Description

If sources.list does contain a line such as

deb http://ddebs.ubuntu.com focal main restricted universe multiverse

then apport-retrace -v -s -R -S system _usr_bin_gnome-shell.1000.crash fails because it tries to access the following url:

http://ddebs.ubuntu.com/dists/focal/Contents-amd64.gz

Full output of above command:

Ign http://ddebs.ubuntu.com focal InRelease
Get:1 http://ddebs.ubuntu.com focal Release [34.7 kB]
Get:2 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal InRelease [20.8 kB]
Get:3 https://mirror.init7.net/ubuntu focal InRelease [255 kB]
Get:4 http://ddebs.ubuntu.com focal Release.gpg [819 B]
Get:5 http://ddebs.ubuntu.com focal/main amd64 Packages [526 kB]
Get:6 https://mirror.init7.net/ubuntu focal-updates InRelease [79.7 kB]
Get:7 https://mirror.init7.net/ubuntu focal-backports InRelease [79.7 kB]
Get:8 https://mirror.init7.net/ubuntu focal-security InRelease [79.7 kB]
Get:9 https://mirror.init7.net/ubuntu focal/restricted Sources [7032 B]
Get:10 https://mirror.init7.net/ubuntu focal/universe Sources [9784 kB]
Get:11 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal/main amd64 Packages [1604 B]
Get:12 http://ddebs.ubuntu.com focal/universe amd64 Packages [4435 kB]
Get:13 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal/main Translation-en [1084 B]
Get:14 https://mirror.init7.net/ubuntu focal/multiverse Sources [175 kB]
Get:15 https://mirror.init7.net/ubuntu focal/main Sources [839 kB]
Get:16 https://mirror.init7.net/ubuntu focal/main amd64 Packages [977 kB]
Get:17 https://mirror.init7.net/ubuntu focal/main Translation-en [503 kB]
Get:18 https://mirror.init7.net/ubuntu focal/main amd64 DEP-11 Metadata [517 kB]
Get:19 https://mirror.init7.net/ubuntu focal/main DEP-11 48x48 Icons [93.6 kB]
Get:20 https://mirror.init7.net/ubuntu focal/main DEP-11 64x64 Icons [164 kB]
Get:21 https://mirror.init7.net/ubuntu focal amd64 Contents (deb) [41.2 MB]
Get:22 http://ddebs.ubuntu.com focal/multiverse amd64 Packages [67.0 kB]
Get:23 https://mirror.init7.net/ubuntu focal/restricted amd64 Packages [21.5 kB]
Get:24 https://mirror.init7.net/ubuntu focal/restricted Translation-en [6136 B]
Get:25 https://mirror.init7.net/ubuntu focal/universe amd64 Packages [8623 kB]
Get:26 https://mirror.init7.net/ubuntu focal/universe Translation-en [5154 kB]
Get:27 https://mirror.init7.net/ubuntu focal/universe amd64 DEP-11 Metadata [3618 kB]
Get:28 https://mirror.init7.net/ubuntu focal/universe DEP-11 48x48 Icons [3000 kB]
Get:29 https://mirror.init7.net/ubuntu focal/universe DEP-11 64x64 Icons [7968 kB]
Get:30 https://mirror.init7.net/ubuntu focal/multiverse amd64 Packages [145 kB]
Get:31 https://mirror.init7.net/ubuntu focal/multiverse Translation-en [105 kB]
Get:32 https://mirror.init7.net/ubuntu focal/multiverse amd64 DEP-11 Metadata [45.0 kB]
Get:33 https://mirror.init7.net/ubuntu focal/multiverse DEP-11 48x48 Icons [17.6 kB]
Get:34 https://mirror.init7.net/ubuntu focal/multiverse DEP-11 64x64 Icons [198 kB]
Fetched 88.8 MB in 6s (7466 kB/s)
Fetched 0 B in 0s (0 B/s)
Extracting downloaded debs...
Debug:http://ddebs.ubuntu.com/dists/focal-proposed/Contents-amd64.gz
Debug:http://ddebs.ubuntu.com/dists/focal/Contents-amd64.gz
Traceback (most recent call last):
  File "/usr/bin/apport-retrace", line 322, in <module>
    sandbox, cache, outdated_msg = apport.sandboxutils.make_sandbox(
  File "/usr/lib/python3/dist-packages/apport/sandboxutils.py", line 220, in make_sandbox
    pkgs = needed_runtime_packages(report, sandbox_dir, pkgmap_cache_dir, pkg_versions, verbose)
  File "/usr/lib/python3/dist-packages/apport/sandboxutils.py", line 92, in needed_runtime_packages
    pkg = apport.packaging.get_file_package(l, True, pkgmap_cache_dir,
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 490, in get_file_package
    return self._search_contents(file, map_cachedir, release, arch)
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 1325, in _search_contents
    src = urlopen(url)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

Fully reproducible in Ubuntu 20.04 fully upgraded as of March 13 2020.
All info files attached in the tarball in comment #18.

WORKAROUND:

Move lines such as
"deb http://ddebs.ubuntu.com focal main restricted universe multiverse"
"deb http://archive.canonical.com/ubuntu focal partner"
into /etc/apt/sources.list.d

EXPECTED FIX:

Either fix the logic in apport-retrace or clearly document this somewhere.

Tags: focal
Revision history for this message
Esokrates (esokrarkose) wrote :

It fails on the following url:
http://archive.canonical.com/ubuntu/dists/focal/Contents-amd64.gz

It seems it tries to use a wrong url, since I guess it should be archive.ubuntu.com instead.

Esokrates (esokrarkose)
tags: added: focal
Revision history for this message
Esokrates (esokrarkose) wrote :
Download full text (19.1 KiB)

I tried again today, I simply cannot make it work:

user@ubuntu:~$ apport-retrace -v -s -R -S system /var/crash/_usr_bin_gnome-shell.1000.crash
Get:1 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal InRelease [20.8 kB]
Ign http://ddebs.ubuntu.com focal InRelease
Ign http://ddebs.ubuntu.com focal-updates InRelease
Get:2 http://ddebs.ubuntu.com focal Release [34.7 kB]
Get:3 http://archive.canonical.com/ubuntu focal InRelease [10.9 kB]
Get:4 http://ddebs.ubuntu.com focal-updates Release [34.3 kB]
Get:5 http://ddebs.ubuntu.com focal Release.gpg [819 B]
Get:6 http://ddebs.ubuntu.com focal-updates Release.gpg [819 B]
Get:7 https://mirror.init7.net/ubuntu focal InRelease [255 kB]
Get:8 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal/main amd64 Packages [1604 B]
Get:9 http://archive.canonical.com/ubuntu focal/partner Sources [1412 B]
Get:10 http://ddebs.ubuntu.com focal/main amd64 Packages [523 kB]
Get:11 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal/main Translation-en [1084 B]
Get:12 http://archive.canonical.com/ubuntu focal/partner amd64 Packages [1596 B]
Get:13 http://archive.canonical.com/ubuntu focal/partner Translation-en [804 B]
Get:14 https://mirror.init7.net/ubuntu focal-updates InRelease [79.7 kB]
Get:15 https://mirror.init7.net/ubuntu focal-backports InRelease [79.7 kB]
Get:16 http://ddebs.ubuntu.com focal/universe amd64 Packages [4426 kB]
Get:17 https://mirror.init7.net/ubuntu focal-security InRelease [79.7 kB]
Get:18 https://mirror.init7.net/ubuntu focal/multiverse Sources [176 kB]
Get:19 https://mirror.init7.net/ubuntu focal/main Sources [845 kB]
Get:20 https://mirror.init7.net/ubuntu focal/restricted Sources [7032 B]
Get:21 https://mirror.init7.net/ubuntu focal/universe Sources [9797 kB]
Get:22 http://ddebs.ubuntu.com focal/multiverse amd64 Packages [67.4 kB]
Get:23 https://mirror.init7.net/ubuntu focal/main amd64 Packages [984 kB]
Get:24 https://mirror.init7.net/ubuntu focal/main Translation-en [506 kB]
Get:25 https://mirror.init7.net/ubuntu focal/main amd64 DEP-11 Metadata [498 kB]
Get:26 https://mirror.init7.net/ubuntu focal/main DEP-11 48x48 Icons [90.9 kB]
Get:27 https://mirror.init7.net/ubuntu focal/main DEP-11 64x64 Icons [160 kB]
Get:28 https://mirror.init7.net/ubuntu focal amd64 Contents (deb) [41.3 MB]
Get:29 https://mirror.init7.net/ubuntu focal/restricted amd64 Packages [21.4 kB]
Get:30 https://mirror.init7.net/ubuntu focal/restricted Translation-en [6136 B]
Get:31 https://mirror.init7.net/ubuntu focal/universe amd64 Packages [8650 kB]
Get:32 https://mirror.init7.net/ubuntu focal/universe Translation-en [5161 kB]
Get:33 https://mirror.init7.net/ubuntu focal/universe amd64 DEP-11 Metadata [3618 kB]
Get:34 https://mirror.init7.net/ubuntu focal/universe DEP-11 48x48 Icons [3008 kB]
Get:35 https://mirror.init7.net/ubuntu focal/universe DEP-11 64x64 Icons [7962 kB]
Get:36 https://mirror.init7.net/ub...

Revision history for this message
Brian Murray (brian-murray) wrote :

Could you add your /etc/apt/sources.list file to this bug report? apport is using this file to determine which archive to use.

Changed in apport:
status: New → Incomplete
Revision history for this message
Esokrates (esokrarkose) wrote :

Thanks Brian, here is my sources.list.

Changed in apport:
status: Incomplete → New
Revision history for this message
Brian Murray (brian-murray) wrote :

Can you try commenting out the archive.canonical.com lines?

Revision history for this message
Brian Murray (brian-murray) wrote :

Additionally, could you attach the crash file to this report? You don't need to include the CoreDump section of the report.

Revision history for this message
Esokrates (esokrarkose) wrote :

Here is the crash file where I removed everything after the "CoreDump: base64" line.

Revision history for this message
Esokrates (esokrarkose) wrote :

Trying with commented out archive.canonical.com lines did not work, see the attached log.

Revision history for this message
Esokrates (esokrarkose) wrote :

As apport complained when invoked on the previously uploaded file, here the crash file without coredump that reproduces the problem for me.

Revision history for this message
Esokrates (esokrarkose) wrote :
Download full text (4.5 KiB)

Tried again on another focal install with identical sources.list where it failed because it tried to download http://ddebs.ubuntu.com/dists/focal/Contents-amd64.gz (the DEBUG lines in the following output come from me inserting a debug print statement in /usr/lib/python3/dist-packages/apport/packaging_impl.py).

apport-retrace -v -s -R -S system ./_usr_bin_gnome-shell-modified.1000.crash
Get:1 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal InRelease [20.8 kB]
Ign http://ddebs.ubuntu.com focal InRelease
Ign http://ddebs.ubuntu.com focal-updates InRelease
Get:2 https://mirror.init7.net/ubuntu focal InRelease [255 kB]
Get:3 http://ddebs.ubuntu.com focal Release [34.7 kB]
Get:4 http://ddebs.ubuntu.com focal-updates Release [34.3 kB]
Get:5 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal/main amd64 Packages [1604 B]
Get:6 http://ddebs.ubuntu.com focal Release.gpg [819 B]
Get:7 http://ddebs.ubuntu.com focal-updates Release.gpg [819 B]
Get:8 https://mirror.init7.net/ubuntu focal-updates InRelease [79.7 kB]
Get:9 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal/main Translation-en [1084 B]
Get:10 http://ddebs.ubuntu.com focal/main amd64 Packages [523 kB]
Get:11 https://mirror.init7.net/ubuntu focal-backports InRelease [79.7 kB]
Get:12 https://mirror.init7.net/ubuntu focal-security InRelease [79.7 kB]
Get:13 https://mirror.init7.net/ubuntu focal/restricted Sources [7032 B]
Get:14 https://mirror.init7.net/ubuntu focal/main Sources [838 kB]
Get:15 http://ddebs.ubuntu.com focal/universe amd64 Packages [4426 kB]
Get:16 https://mirror.init7.net/ubuntu focal/multiverse Sources [176 kB]
Get:17 https://mirror.init7.net/ubuntu focal/universe Sources [9792 kB]
Get:18 http://ddebs.ubuntu.com focal/multiverse amd64 Packages [67.4 kB]
Get:19 https://mirror.init7.net/ubuntu focal/main amd64 Packages [972 kB]
Get:20 https://mirror.init7.net/ubuntu focal/main Translation-en [500 kB]
Get:21 https://mirror.init7.net/ubuntu focal amd64 Contents (deb) [41.3 MB]
Get:22 https://mirror.init7.net/ubuntu focal/restricted amd64 Packages [21.4 kB]
Get:23 https://mirror.init7.net/ubuntu focal/restricted Translation-en [6136 B]
Get:24 https://mirror.init7.net/ubuntu focal/universe amd64 Packages [8651 kB]
Get:25 https://mirror.init7.net/ubuntu focal/universe Translation-en [5161 kB]
Get:26 https://mirror.init7.net/ubuntu focal/multiverse amd64 Packages [145 kB]
Get:27 https://mirror.init7.net/ubuntu focal/multiverse Translation-en [106 kB]
Fetched 73.3 MB in 6s (1072 kB/s)
Fetched 0 B in 0s (0 B/s)
Extracting downloaded debs...
DEBUG: http://ddebs.ubuntu.com/dists/focal-proposed/Contents-amd64.gz
DEBUG: http://ddebs.ubuntu.com/dists/focal/Contents-amd64.gz
Traceback (most recent call last):
  File "/usr/bin/apport-retrace", line 322, in <module>
    sandbox, cache, outdated_m...

Read more...

Revision history for this message
Esokrates (esokrarkose) wrote :
Download full text (5.8 KiB)

Commenting out the ddebs lines, I get:

apport-retrace -v -s -R -S system ./_usr_bin_gnome-shell-modified.1000.crash
Get:1 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal InRelease [20.8 kB]
Get:2 https://mirror.init7.net/ubuntu focal InRelease [255 kB]
Get:3 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal/main amd64 Packages [1604 B]
Get:4 https://mirror.init7.net/ubuntu focal-updates InRelease [79.7 kB]
Get:5 http://ppa.launchpad.net/saiarcot895/chromium-dev/ubuntu focal/main Translation-en [1084 B]
Get:6 https://mirror.init7.net/ubuntu focal-backports InRelease [79.7 kB]
Get:7 https://mirror.init7.net/ubuntu focal-security InRelease [79.7 kB]
Get:8 https://mirror.init7.net/ubuntu focal/restricted Sources [7032 B]
Get:9 https://mirror.init7.net/ubuntu focal/multiverse Sources [176 kB]
Get:10 https://mirror.init7.net/ubuntu focal/main Sources [838 kB]
Get:11 https://mirror.init7.net/ubuntu focal/universe Sources [9792 kB]
Get:12 https://mirror.init7.net/ubuntu focal/main amd64 Packages [972 kB]
Get:13 https://mirror.init7.net/ubuntu focal/main Translation-en [500 kB]
Get:14 https://mirror.init7.net/ubuntu focal amd64 Contents (deb) [41.3 MB]
Get:15 https://mirror.init7.net/ubuntu focal/restricted amd64 Packages [21.4 kB]
Get:16 https://mirror.init7.net/ubunt...

Read more...

Revision history for this message
Esokrates (esokrarkose) wrote :

I tried again with a fairly standard sources.list:

deb http://de.archive.ubuntu.com/ubuntu focal main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu eoan main restricted universe multiverse

deb http://de.archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu eoan-updates main restricted universe multiverse

deb http://de.archive.ubuntu.com/ubuntu focal-security main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu eoan-security main restricted universe multiverse

deb http://de.archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu eoan-backports main restricted universe multiverse

and with the ddebs in /etc/apt/sources.list.d/ddebs.list

I then arrived again at the situation as previously, see the output.log.

Revision history for this message
Esokrates (esokrarkose) wrote :

@Brian I have fixed the bug concerning the "SystemError: cannot determine default mirror", apport made the wrong assumption that all repo lines start with http://.

I was able to get the sandbox working by modifying /usr/lib/python3/dist-packages/apt/cache.py to not raise an exception in case of untrusted packages, but it is a serious bug that it cannot verify packages, I hope you can help there, I am not able to fix this by myself.

Revision history for this message
Esokrates (esokrarkose) wrote :

@Brian: I added additional debug print statements to find out on which packages the verification fails:

 <apt_pkg.AcquireItem object:Status: 0 Complete: 0 Local: 0 IsTrusted: 0 FileSize: 0 DestFile:'/home/user/FOSS/BUGS/cache/system/apt/var/cache/apt/archives//libx11-6-dbgsym_1.6.9-2_amd64.ddeb' DescURI: 'https://launchpad.net/ubuntu/+archive/primary/+files/libx11-6-dbgsym_1.6.9-2_amd64.ddeb' ID:0 ErrorText: ''>

However there are dbgsym packages where the verification is successful:

<apt_pkg.AcquireItem object:Status: 0 Complete: 0 Local: 0 IsTrusted: 1 FileSize: 120196 DestFile:'/home/user/FOSS/BUGS/cache/system/apt/var/cache/apt/archives/partial/libbsd0-dbgsym_0.10.0-1_amd64.ddeb' DescURI: 'http://ddebs.ubuntu.com/pool/main/libb/libbsd/libbsd0-dbgsym_0.10.0-1_amd64.ddeb' ID:0 ErrorText: ''>

So it seems when DescURI is 'https://launchpad.net/ubuntu/+archive/primary/+files/' it fails to verify which seems logical to me since it should fetch the package from the ddebs repo instead. However all of this happens behind the scenes, I fail to find out where and why the launchpad url gets invoked.

Revision history for this message
Esokrates (esokrarkose) wrote :

For anyone affected: Fix for https + workaround for the verification problem. Not though that the workaround undermines system security by allowing unverified packages!

Revision history for this message
Esokrates (esokrarkose) wrote :

The issues with untrusted packages does not happen anymore with the latest updates, so I'm closing this issue, the other issues are reported in https://bugs.launchpad.net/apport/+bug/1866996.

Changed in apport:
status: New → Incomplete
Revision history for this message
Esokrates (esokrarkose) wrote : Re: apport-retrace sandbox: Fails if mirror does not provide Contents-amd64.gz

@Brian, reopening this one here, I figured out this is another issue which I did not describe correctly before. I am going to update the description and provide all associated info in a tar.gz file.

summary: - apport-retrace fails to build sandbox
+ apport-retrace sandbox: Fails if mirror does not provide Contents-
+ amd64.gz
Changed in apport:
status: Incomplete → New
Revision history for this message
Esokrates (esokrarkose) wrote :
description: updated
Esokrates (esokrarkose)
description: updated
Esokrates (esokrarkose)
summary: - apport-retrace sandbox: Fails if mirror does not provide Contents-
+ apport-retrace sandbox fails if mirror does not provide Contents-
amd64.gz
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.