ubuntutools.archive.UbuntuSourcePackage().pull() fails

Colin has a working patch for this.

more specifically, it says

Checksum for akonadi_1.1.2.orig.tar.gz does not match.
DownloadError: File akonadi_1.1.2.orig.tar.gz could not be found

Since the checksum actually is incorrect for this package/version, the MR will allow this to work if you add verify_signature=False:

>>> archive.UbuntuSourcePackage('akonadi', '1.1.2-0ubuntu1~jaunty1').pull()
Public key not found, could not verify signature
Checksum for akonadi_1.1.2.orig.tar.gz does not match.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/ubuntutools/archive.py", line 514, in pull
    raise DownloadError('File %s could not be found' % name)
ubuntutools.archive.DownloadError: File akonadi_1.1.2.orig.tar.gz could not be found
>>> archive.UbuntuSourcePackage('akonadi', '1.1.2-0ubuntu1~jaunty1', verify_signature=False).pull()
>>>

> Since the checksum actually is incorrect for this package/version...

I believe that's inaccurate. I suggest you wait to see Colin's patch. He was going to prepare an MP when he has time. The problem AIUI is something to do with ambiguity when multiple different orig tarballs are associated with a single upstream version string.

> The problem AIUI is something to do with ambiguity when multiple different orig tarballs are associated with a single upstream version string

ouch, that is *really* bad. But indeed, that's the case here. Oh well, it's jaunty, right?

updated MR fixes it:

>>> from ubuntutools import archive
>>> archive.UbuntuSourcePackage('akonadi', '1.1.2-0ubuntu1~jaunty1').pull()
Public key not found, could not verify signature
Checksum for akonadi_1.1.2.orig.tar.gz does not match.
>>>

> I suggest you wait to see Colin's patch

there should be a way to disable the checksum checking, whether we tie it into verify_signature or use a separate parameter, regardless of any other patch. updated MR creates --no-verify-checksums param to handle that.

@mapreri MR should be ready for your review and merge

It's not just jaunty. I've seen such packages uploaded last year too.
Though I must say it's so rare that I forgot such packages existed…

And a flag such as `--no-verify-checksums` sounds a very bad idea to have to me.

> I've seen such packages uploaded last year too

that's unfortunate and quite bad.

> And a flag such as `--no-verify-checksums` sounds a very bad idea to have to me.

ok i removed that patch; the MR still resolves this bug.

On Wed, 22 Jan 2020, 1:55 pm Dan Streetman, <email address hidden> wrote:

> > I've seen such packages uploaded last year too
>
> that's unfortunate and quite bad.
>

Well, it's a feature, although one that does create confusion in many
situations. So it's probably good that very few people know of it ^^

I'd still wait to see what Colin has to share.

ack, let's see what Colin has, unassigning myself.

> a flag such as `--no-verify-checksums` sounds a very bad idea

btw we don't verify checksums for any downloaded binaries, but it's probably a good idea to do so:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/377952

I've proposed my patch now. I have a backport for bionic as well, though I'll wait to propose that until the one against master is approved.

I'm not sure why you "strongly recommend" not to apply my patch; it uses the same url as your patch:

>>> from ubuntutools.archive import UbuntuSourcePackage
>>> import pprint
>>> pp = pprint.PrettyPrinter()
>>> pp.pprint(UbuntuSourcePackage('akonadi', '1.1.2-0ubuntu1~jaunty1').lp_spph.sourceFileUrls())
['https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/akonadi/1.1.2-0ubuntu1~jaunty1/akonadi_1.1.2.orig.tar.gz',
 'https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/akonadi/1.1.2-0ubuntu1~jaunty1/akonadi_1.1.2-0ubuntu1~jaunty1.diff.gz',
 'https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/akonadi/1.1.2-0ubuntu1~jaunty1/akonadi_1.1.2-0ubuntu1~jaunty1.dsc']

personally, I'd prefer to junk _lp_url() entirely and use proper lpapi methods to get the urls.

my latest PR now removes _lp_url() completely...I'm much happier removing the hardcoded url construction and using the lp api methods to get the proper urls.

@mapreri @cjwatson any decision on this? My ubuntu-dev-tools delta is growing again...

just busy due to pre/post FOSDEM + FOSDEM fringe + family event right after
that; TBH I haven't had the chance to look at this at all, but it has been
not even 2 weeks since Colin's MR…

On Wed, Feb 5, 2020 at 8:30 PM Dan Streetman <email address hidden> wrote:

> @mapreri @cjwatson any decision on this? My ubuntu-dev-tools delta is
> growing again...
>
> --
> You received this bug notification because you are subscribed to Ubuntu.
> https://bugs.launchpad.net/bugs/1860456
>
> Title:
> ubuntutools.archive.UbuntuSourcePackage().pull() fails
>
> Status in ubuntu-dev-tools package in Ubuntu:
> Confirmed
>
> Bug description:
> Steps to reproduce:
>
> import ubuntutools.archive
> ubuntutools.archive.UbuntuSourcePackage('akonadi',
> '1.1.2-0ubuntu1~jaunty1').pull()
>
> This fails with an ubuntutools.archive.DownloadError.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/ubuntu-dev-tools/+bug/1860456/+subscriptions
>
>

--
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`.
more about me: https://mapreri.org : :' :
Launchpad user: https://launchpad.net/~mapreri `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia `-

ack, no hurry, thanks.

> ack, no hurry, thanks.

Actually this blocks progress in fixing a class of import failures in git-ubuntu, so I would appreciate getting this landed sooner rather than later. I'd prefer not to have to patch the git-ubuntu snap's bundled ubuntu-dev-tools as that opens up a whole bunch of extra complications.

Can we just review and land Colin's patch please, as it's tight in scope and thus simple to review?

> Can we just review and land Colin's patch please...

I just realised that my words could be taken to mean that I'm being impatient at mapreri. I didn't mean that - only that if we just consider Colin's patch alone, then there's far less review work necessary. I'm hoping we can be less demanding of mapreri's time this way.

> Can we just review and land Colin's patch please...

I have no problem if Colin's patch is merged, although I should point out my patch completely removes the _lp_url() function.

@mapreri I moved @cjwatson's patch, along with the simpler patches for bug 1862286, bug 1862372, and bug 1863119, to the bottom of my patch queue, and included them all in this mr:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/378722

My work queue still has quite a few more patches on those, but there's no urgency to the rest of my patches.

Hi @mapreri,

I hope you don't mind, but I went ahead and pushed the fixes for this and the other 3 bugs mentioned in my last comment, and uploaded to focal. Of course, please feel free to revert anything I uploaded if you feel it's wrong. Note that for this bug I did use @cjwatson's commit.

I still have more commits in my working branch (e.g. to verify checksums for downloaded binary files) but I didn't push those, only what was needed to fix the bugs.

This bug was fixed in the package ubuntu-dev-tools - 0.175ubuntu1

---------------
ubuntu-dev-tools (0.175ubuntu1) focal; urgency=medium

  [ Dan Streetman ]
  * lpapicache: remove new requirement to pass arch to getBinaries()
    (LP: #1862286)
  * ubuntu-upload-permission: sort packagesets by name to fix py3
    (LP: #1862372)
  * submittodebian: open with 'b' before writing utf-8 encoded bytes
    (LP: #1863119)

  [ Colin Watson ]
  * Use +sourcefiles URLs where possible (LP: #1860456)

 -- Dan Streetman <email address hidden> Thu, 20 Feb 2020 17:20:23 -0500

Sure, having reviewed now what was merged I'd say I have nothing to complain about :)

Thank you for your work, and sorry for lagging behind!

Fix released in git-ubuntu version 1.0