Failure to build snap that runs pull-lp-source

I don't think this is as architecture-independent as initially believed. The single amd64 failure looks like some kind of transient network breakage, while the non-amd64 failures are different and 100% consistent: judging from the traceback, pull-lp-source is successfully contacting launchpad.net, getting a redirect response, and then timing out. It makes sense that this would differ between x86 and non-x86 architectures due to the different network environments involved.

I'm suspicious that it appears that pull-lp-source isn't using the proxy, because it appears to get a redirect response without launchpad-buildd having logged the proxied request as it would normally do. By diffing build logs, I notice that successful and failing builds are using different versions of ubuntu-dev-tools (0.164 vs. 0.175~18.04.1) due to an SRU that happened in the meantime. I haven't yet managed to find an obvious cause in the diff, but that remains a possibility.

On bionic, python3-httplib2 doesn't support proxies, but python-httplib2 does. (This was fixed in 0.11.0 - https://github.com/httplib2/httplib2/pull/90 - so focal should work fine.) The ubuntu-dev-tools backport switched pull-lp-source to python3. Reverting that (and making sure that the Python 2 python-ubuntutools package has enough supporting code, since at least some of this seems to have been removed) should avoid this problem.

> On bionic, python3-httplib2 doesn't support proxies, but python-httplib2 does

This may have been introduced by the backport in bug 1864571

Also, the use of httplib2 has been removed (almost) entirely for a while in the package at:
https://launchpad.net/~ubuntu-support-team/+archive/ubuntu/ubuntu-dev-tools

that might work for you (though, the Bionic build in that ppa does no backporting at all, so it doesn't support py2 at all)

Hmm, it seems like python3-launchpadlib uses httplib2 as well, and since ubuntu-dev-tools uses that to communicate with launchpad, it would need to be fixed (or python3-httplib2 would need to be fixed) if ubuntu-dev-tools in Bionic stays python3-only.

Status changed to 'Confirmed' because the bug affects multiple users.

@ddstreet, that part isn't a practical issue in this environment because direct HTTPS connections to api.launchpad.net are allowed.

I've uploaded a fix for this. If someone can write up a test case I'd appreciate it, or I can do it tomorrow.

This is still missing a test case.

This still needs SRU information please.

Closing the launchpad-buildd task since this turned out not to be a Launchpad issue.

As it doesn't look like this is going to make any progress, I'm rejecting it from the queue to clean up.

That upload apparently wouldn't have worked anyway. I built a local package with:
https://launchpadlibrarian.net/489460611/ubuntu-dev-tools_0.175~18.04.2_source.changes
http://launchpadlibrarian.net/489460619/ubuntu-dev-tools_0.175~18.04.1_0.175~18.04.2.diff.gz

and:

$ pull-lp-source hello
Traceback (most recent call last):
  File "/usr/bin/pull-lp-source", line 11, in <module>
    PullPkg.main(distro='ubuntu', pull='source')
  File "/usr/lib/python2.7/dist-packages/ubuntutools/pullpkg.py", line 96, in main
    cls(*args, **kwargs).pull()
  File "/usr/lib/python2.7/dist-packages/ubuntutools/pullpkg.py", line 108, in __init__
    self._default_arch = kwargs.get('arch', host_architecture())
  File "/usr/lib/python2.7/dist-packages/ubuntutools/misc.py", line 96, in host_architecture
    encoding='utf-8').strip()
  File "/usr/lib/python2.7/subprocess.py", line 216, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'encoding'

> That upload apparently wouldn't have worked anyway

yeah, i'm not very happy that bug 1864571 was pushed through against my (and mapreri's) objections. This (and probably more) breakage is the result, and I don't particularly want to be the one to clean up the mess.

I have made improvements to pull-lp-source in hirsute which make things better.

launchpadlibrarian.net:443 now works directly from builders in bos02. Unlike launchpadlibrarian.net:80, there's no caching to avoid long-fat-pipe issues with the transatlantic hop, so 443 is likely to be noticeably slower; it won't be used for typical bulk transfer from the librarian in builds, but it at least minimally works.