Rutabaga

407 Proxy Authentication errors from snap build on s390x and PPC64el

Bug #1969757 reported by Adam Collard
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Rutabaga
In Progress
High
Colin Watson

Bug Description

Some recent LP snap builds of MAAS on PPC64el and s390x failed with a HTTP 407 trying to fetch libpython3-dev from ftpmaster

https://launchpadlibrarian.net/597662141/buildlog_snap_ubuntu_focal_ppc64el_maas-master_BUILDING.txt.gz

has

E: Failed to fetch http://ftpmaster.internal/ubuntu/pool/main/p/python3-defaults/libpython3-dev_3.8.2-0ubuntu2_ppc64el.deb 407 Proxy Authentication Required [IP: 10.10.10.1 8222]

and https://launchpadlibrarian.net/597662144/buildlog_snap_ubuntu_focal_s390x_maas-master_BUILDING.txt.gz has:

E: Failed to fetch http://ftpmaster.internal/ubuntu/pool/main/p/python3-defaults/libpython3-dev_3.8.2-0ubuntu2_s390x.deb 407 Proxy Authentication Required [IP: 10.10.10.1 8222]

Jürgen Gmach (jugmac00)
Changed in launchpad:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Colin Watson (cjwatson) wrote :

There's not much obvious from logs, but my best guess is that this may be because of bad template substitution in a charm (that I fixed a little while ago in https://code.launchpad.net/~cjwatson/rutabaga/+git/rutabaga/+merge/418939, but hadn't got round to rolling out to production yet) that means that the builder proxy authenticator's database of valid tokens has been bloating without bound ever since it was installed, which is somewhat likely to cause random failures like this.

I've filed https://portal.admin.canonical.com/C149735 to get the fix rolled out. That should also give us better access to Squid logs, giving us a better chance of debugging future problems of this kind.

no longer affects: launchpad-buildd
affects: launchpad → rutabaga
Changed in rutabaga:
assignee: nobody → Colin Watson (cjwatson)
status: Triaged → In Progress
Revision history for this message
Adam Collard (adam-collard) wrote (last edit ):

Hit again on https://launchpadlibrarian.net/597887356/buildlog_snap_ubuntu_focal_arm64_maas-master_BUILDING.txt.gz (this time npm, so added confidence that it's not particular to the resource being fetched)

npm ERR! code E407
npm ERR! 407 Proxy Authentication Required - GET https://registry.npmjs.org/find-up

[22/Apr/2022:08:16:17 +0000] "CONNECT registry.npmjs.org:443 HTTP/1.1" 200 9604 "-" "-"

Revision history for this message
Colin Watson (cjwatson) wrote :

While reducing cruft in the DB seemed to help a bit, we're generally seeing `sqlite3.OperationalError: database is locked` at about the same time as these errors. I think we really need to finally get round to converting rutabaga to postgresql.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.