Upload to PPA from bb-cloud-images jenkaas nodes taking over an hour per upload

Bug #1974036 reported by Philip Roche
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
txpkgupload
Triaged
High
Unassigned

Bug Description

As requested in internal Canonical RT https://rt.admin.canonical.com/Ticket/Display.html?id=149592 creating bug due to very slow uploads to PPA.

https://rt.admin.canonical.com/Ticket/Display.html?id=149592 is internal to Canonical but the relevant contents of the RT are:

We have a jenkins job https://jenkins.canonical.com/bb-cloud-images/job/18.04-Prepare-Secret-Sauce-Livecd-Rootfs/ which builds packages and uploads to private PPA https://launchpad.net/~cloudware/+archive/ubuntu/cpc-livecd-rootfs. We have recently noticed that uploads to the PPA take about an hour for each upload.

```
22:10:21 Uploading to ppa-secret-sauce (via sftp to ppa.launchpad.net):
22:10:21 livecd-rootfs_2.525.58~202204182110.upstream-7bef05d.cpc-cef6388.maas-aa52770.dsc: done.
22:10:22 livecd-rootfs_2.525.58~202204182110.upstream-7bef05d.cpc-cef6388.maas-aa52770.tar.xz: done.
22:10:22 livecd-rootfs_2.525.58~202204182110.upstream-7bef05d.cpc-cef6388.maas-aa52770_source.changes: Connection to ppa.launchpad.net closed by remote host.
23:10:22 done.
23:10:22 Successfully uploaded packages.
```

Do you have any ideas why this is taking so long or how we can debug? These are not large packages (fully build is only 256.5 KiB).

Nvidia driver PPA uploads are also taking the same about of time so it isn't bb-cloud-images specific.

See https://stable-cloud-images-jenkins.canonical.com/job/22.04-Minimal-Nvidia-Driver-Build-Part-1-nvidia-modules/default/20/console

```
13:29:53 Uploading to ppa (via sftp to ppa.launchpad.net):
13:29:53 nvidia-modules_0.68+22.04-5.15-470-82.dsc: done.
13:29:54 nvidia-modules_0.68+22.04-5.15-470-82.tar.xz: done.
13:29:54 nvidia-modules_0.68+22.04-5.15-470-82_source.changes: done.
14:29:54 Connection to ppa.launchpad.net closed by remote host.
14:29:54 Successfully uploaded packages.
```

We did not see these issues on https://jenkins.canonical.com/devel-cloud-images

See https://jenkins.canonical.com/devel-cloud-images/job/22.04-Minimal-Nvidia-Driver-Build-Part-1-nvidia-modules/default/6/console

```
15:51:58 Uploading to ppa (via ftp to ppa.launchpad.net):
15:51:58 Uploading nvidia-modules_0.68+22.04-5.15-470-80.dsc: done.
15:51:58 Uploading nvidia-modules_0.68+22.04-5.15-470-80.tar.xz: done.
15:51:58 Uploading nvidia-modules_0.68+22.04-5.15-470-80_source.buildinfo: done.
15:51:58 Uploading nvidia-modules_0.68+22.04-5.15-470-80_source.changes: done.
15:51:58 Successfully uploaded packages.
```

Could it be SFTP vs FTP upload related?

Note that some of the links above will not be accessible to non Canonical folk.

Revision history for this message
Colin Watson (cjwatson) wrote :

Please do try FTP if you can. It exercises sufficiently different code paths in txpkgupload that there's a good chance it'll avoid this particular bug.

Changed in txpkgupload:
importance: Undecided → High
status: New → Triaged
Revision history for this message
Thomas Bechtold (toabctl) wrote :

Looks like **not** using a dput profile and instead just doing

dput ppa:xxxxx *.changes

just works. our dput.cf profile looked like:

cat << DPUT_CONFIG_EOF > $HOME/.dput.cf
-[my-profile]
-method = sftp
-login = my-lp-username
-fqdn = ppa.launchpad.net
-incoming = ppa:cloudware/xxxxxx
-allow_unsigned_uploads = 0
-DPUT_CONFIG_EOF

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.