CPC AWS jammy builds fail with: "fuse3 : Breaks: fuse"

Sounds like something is trying to pull fuse on the image?

It's launchpad-buildd that installs fuse: https://git.launchpad.net/launchpad-buildd/tree/lpbuildd/target/build_livefs.py#n89

That installation by launchpad-buildd happens in the container that livecd-rootfs runs in, not in the chroot that livecd-rootfs is busy building. If you look more closely at the build log you can see it succeeding earlier on. We may indeed have to update launchpad-buildd to use fuse3, but it's not what's breaking the build here.

The more immediate problem is:

  Package: fuse
  Task: standard, server-minimal

  Package: fuse3
  Task: cloud-image, server, ubuntu-server-raspi

This in turn is because open-vm-tools 2:11.3.5-1 switched to fuse3, so fuse3 ends up in the expanded server seed (https://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.jammy/rdepends/ALL/fuse3), while loads of other things still depend on fuse and so fuse ends up in the expanded server-minimal seed (https://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.jammy/rdepends/ALL/fuse).

This is a distribution inconsistency problem, not a launchpad-buildd problem.

Hi Thomas,
hmm ... that might be fallout from the general move to fuse3.

See MIR bug
=> https://bugs.launchpad.net/ubuntu/+source/fuse3/+bug/1934510
And the links from there.

Ginggs did make an analysis back then if everything is ready to move:
=> https://lists.ubuntu.com/archives/ubuntu-devel/2021-July/041530.html

I think almost all in that list was ready to move
Your bug made me re-check things again from the POV of image builds.

In the seeds we have:
live-common:14:# * unionfs-fuse
desktop-minimal:88: * (gvfs-fuse) # let non-GNOME apps see GVFS via fuse

Not sure how that was missed in the past, probably the dependency checks were on reverse-depends, but not considering the seeds? Anyway, what we see now is a conflict between the seeds and some of the packaging changes that already happened (see the links from the link above - some paclages already switched - among others there is qemu and open-vm-tools).

The latter (open-vm-tools) is usually seeded for usage as vmware guest.
(Side-note: I see no useful use-case on a AWS image, and might be a separate improve-images discussion)

#1
Package: gvfs-fuse
Depends: fuse, gvfs (= 1.48.1-2ubuntu3), libc6 (>= 2.34), libfuse3-3 (>= 3.2.3), libglib2.0-0 (>= 2.65.1)

#2
Package: unionfs-fuse
Version: 1.0-1ubuntu2
Depends: libc6 (>= 2.7), libfuse2 (>= 2.8), fuse

Continuing on those in the next comment ...

The gvfs one is just an error in the revert of the delta we have to not use fuse3 yet, gvfs-fuse should depends on fuse3 now as it does in Debian, desktop will fix that one

#1 - Package: gvfs-fuse
This build for Desktop already switched to the version3 as you see in "libfuse3-3 (>= 3.2.3)".
But it still depends on the v2 user-tools. That mismatch needs to be resolved for 22.04 and it will need to switch to fuse3.
I'm adding a src:gvfs task for that to switch to fuse3 (assigning to Didier, please reassign in *Desktop accordingly)

#2
Package: unionfs-fuse
This wasn't yet tried with fuse3 AFAIC as it was missed on the initial analysis for bug 1934510.
I'll add a bug task for it here and ping bug 1934510 about it.
We will have to move this to fuse3 as well, and the others of the formerly identified packages.
I'll assign that to Ginggs as he has taken the initial archive wide check, maybe Foundations has this and more prepared to have all of them move at once.

If it turns out that not all are ready yet, then we need to tune back open-vm-tools "for now" until they are ready. Therefore I'll add an open-vm-tools tasks assigned to myself, but set to incomplete until we know from the others if everything could move now or if we have to wait.

P.S. There might be others like grub, s390x-tools, ... (see initial mail I linked above) which might trigger the same issue for this or other images, see bug 1934510 and the links from there.

Also see https://bugs.launchpad.net/ubuntu/+source/fuse3/+bug/1934510/comments/24

@Marco Trevisan - I subscribed you in case you want to steal the Desktop POV to this from Didier.

FYI - I uploaded a revert to the open-vm-tools change to un-block image builds for now.
But maybe this was a good wake-up call.
Please have a look at gvfs, union-fuse (here) and grub2, s390-tools, snapd, xdg-desktop-portal in bug 1934510.

I will prepare an upload of unionfs-fuse that switches to libfuse3 during the next days.

Hi Bernd,
I'm glad to heard that as we were not sure it would work well with fuse3.
As a reminder, the libs are co-installable it is "bin:fuse" vs "bin:fuse3" that are conflicting.
So when you change that remember to also change fuse->fuse3.

In the meantime it was found that unionfs-fuse isn't seeded (anymore) - it is only a comment in the Ubuntu seeds nowadays. So it will be great to move it to fuse3, but not resolve the issue discussed here. That must have been due to one of the other dependencies in the image.

Anyone looking at this - please coordinate with ginggs, trevinho, paride on #ubuntu-devel for the timing of related Ubuntu uploads if they are seeded (or even in general).

Hi Christian,

I will actually just prepare an upload to debian (via mentors) and then let that get pulled into ubuntu. I don't know what "unionfs-fuse isn't seeded (anymore)" means - automatic pulling from debian is not done anymore or something else?

This bug was fixed in the package open-vm-tools - 2:11.3.5-1ubuntu2

---------------
open-vm-tools (2:11.3.5-1ubuntu2) jammy; urgency=medium

  * d/control, d/rules: go back to fuse3 until all other
    packages are ready (LP: #1956949)

 -- Christian Ehrhardt <email address hidden> Mon, 10 Jan 2022 14:44:22 +0100

@Bernd "not being seeded" means that unionfs-fuse isn't in the set of packages that define what goes into the Ubuntu 'main' component. In summary this is done by choosing some "seed" packages and computing their (build-)dependency tree, see [1,2] if you want to know more.

Automatic syncing of unionfs-fuse from Debian won't happen at the moment because the package in Ubuntu has Ubuntu specific changes. By looking at the package changelog these are:

* No change rebuild to pick up -fPIE compiler default
* Merge from Debian unstable (LP: #1476699). Remaining changes:
  - debian/control: Add Multi-Arch: foreign.

Of course only the second one is a "real" change. If you can add

  Multi-Arch: foreign

to d/control in Debian then we can make unionfs-fuse auto-sync again to Ubuntu. See [3,4] for more details in Multi-Arch.

[1] https://wiki.ubuntu.com/SeedManagement
[2] https://wiki.ubuntu.com/Germinate
[3] https://wiki.debian.org/MultiArch/Hints
[4] https://wiki.ubuntu.com/MultiarchSpec

This bug was fixed in the package gvfs - 1.48.1-2ubuntu5

---------------
gvfs (1.48.1-2ubuntu5) jammy; urgency=medium

  * debian/patches/new-meson-build.patch:
    - don't use positional arguments in the 18n merge function, fix the
      build with the new meson version

 -- Sebastien Bacher <email address hidden> Wed, 12 Jan 2022 16:35:28 +0100

It was pointed out on IRC that ntfs-3g Pre-Depends on fuse which should probably be changed

Good catch, I filed a separate bug for ntfs-3g:

https://bugs.launchpad.net/ubuntu/+source/ntfs-3g/+bug/1957756

and mentioned it in the MIR bug LP: #1934510.

FYI according to
https://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.jammy/rdepends/ALL/fuse
https://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.jammy/rdepends/ALL/fuse3

This should no more be an issue now, so open-vm-tools will switch to fuse3 again in the next few days. This is a heads up in case it breaks again some way.

@aakef - while I said unionfs-fuse might not be a blocker I'd still recommend to upload the change switching to fuse3. On comment #12 it seemed you are close. Any chance to get this done or did unexpected blockers show up while trying?

The unionfs-fuse task here is assigned to Graham; @ginggs - did you make any progress on that yourself or together with Bernd?

FYI the re-upload of a fuse3 open-vm-tools happened yesterday and it migrated to jammy-release.
Any issues with the images?

@Paride Thanks for your help and explanations. Yeah I know what multi-arch does although I didn't know about 'Multi-Arch: foreign' - so thank you very much for the pointers.

@Christian I have uploaded the updated package to mentors on Sunday. It was prepared on Ubuntu, though, I guess it is better when to redo the steps through a debian chroot - I hope I come to that in the evening. https://mentors.debian.net/package/unionfs-fuse/