Installer crash when trying to unmount target

I guess this can be fixed by manually bind mounting /target/sys/fs/fuse/connections as we currently do for /target/sys/firmware/efi/efivars. But I don't understand why either of these is needed, unless /sys/fs/fuse/connections is not mounted before the install starts, because this sequence works for me:

root@venerated-bobcat:~/t# mkdir a a/b c
root@venerated-bobcat:~/t# mount -t fusectl fusectl a/b
root@venerated-bobcat:~/t# mount --bind a c
root@venerated-bobcat:~/t# umount c

This sequence does not:

root@venerated-bobcat:~/t# mkdir a a/b c
root@venerated-bobcat:~/t# mount --bind a c
root@venerated-bobcat:~/t# mount -t fusectl fusectl a/b
root@venerated-bobcat:~/t# umount c
umount: /root/t/c: target is busy.

But afaics /sys/fs/fuse/connections is mounted at startup (it's "wanted" by sysinit.target). So I don't know what's going on basically. But I can probably fix it anyway if it's urgent.

Is there any way I can poke around on a system where this happens?

I have checked and for this system /sys/fs/fuse/connections is not mounted at startup:

root@ubuntu-server:/# mount | grep fuse
root@ubuntu-server:/# ls /sys/fs/fuse/connections
ls: cannot access '/sys/fs/fuse/connections': No such file or directory

Some more observations. The fs-fuse mount unit fails:

root@ubuntu-server:/# systemctl status sys-fs-fuse-connections.mount
* sys-fs-fuse-connections.mount - FUSE Control File System
     Loaded: loaded (/lib/systemd/system/sys-fs-fuse-connections.mount; static; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Thu 2021-05-27 15:16:21 UTC; 1 months 16 days ago
      Where: /sys/fs/fuse/connections
       What: fusectl
       Docs: https://www.kernel.org/doc/Documentation/filesystems/fuse.txt
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

May 27 15:16:19 ubuntu-server systemd[1]: Condition check resulted in FUSE Control File System being skipped.
May 27 15:16:21 ubuntu-server systemd[1]: Condition check resulted in FUSE Control File System being skipped.
May 27 15:16:21 ubuntu-server systemd[1]: Condition check resulted in FUSE Control File System being skipped.
May 27 15:16:21 ubuntu-server systemd[1]: Condition check resulted in FUSE Control File System being skipped.

The condition that fails is apparently

ConditionPathExists=/sys/fs/fuse/connections

Which indeed does not exist. It seems to be created when a FUSE mount happens:

root@ubuntu-server:/# ls /sys/fs/fuse
ls: cannot access '/sys/fs/fuse': No such file or directory
root@ubuntu-server:/# mkdir xx
root@ubuntu-server:/# mount -t fusectl fusectl xx
root@ubuntu-server:/# ls xx
root@ubuntu-server:/# mount | grep fuse
fusectl on /xx type fusectl (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
root@ubuntu-server:/# umount xx
root@ubuntu-server:/# ls -d /sys/fs/fuse/connections
/sys/fs/fuse/connections

At that point I can restart successfully the mount unit with "systemctl restart sys-fs-fuse-connections.mount". And with that I am also able to finish successfully the installation.

To me, the ConditionPathExists=/sys/fs/fuse/connections is quite strange, and it seems like it is a dependency that is fulfilled when the mount unit is run. So the mount unit expects something has already mounted fusectl but its mission is to mount it too??

huh. does modprobe fuse make the mount happen?

Ok, the problem seems to be that the fuse module needs to be manually loaded. Running

# modprobe fuse

makes sys-fs-fuse-connections.mount re-start and it finishes successfully. The difference here is that in the Ubuntu generic kernel we have CONFIG_FUSE_FS=y, while in bluefield kernel it is a module.

We can force the loading of this kernel module either by configuration or from subiquity. Or maybe change order of things as in comment #2 so pre-loading the module is not needed.

On Tue, Jul 13, 2021 at 11:01 AM Michael Hudson-Doyle <
<email address hidden>> wrote:

> huh. does modprobe fuse make the mount happen?
>

Just answered to that before seeing this comment :D

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1934775
>
> Title:
> Installer crash when trying to unmount target
>
> Status in subiquity:
> New
>
> Bug description:
> Installing has crashed in the latest stages of the installation,
> apparently due to not being able to unmount /target/sys. The error I
> see in the log is:
>
> Running command ['umount', '/target/sys/firmware/efi/efivars'] with
> allowed return codes [0] (capture=False)
> Running command ['umount', '/target/sys'] with allowed return codes [0]
> (capture=False)
> umount: /target/sys: target is busy.
>
> Mount shows that there is something else still mounted in target/sys:
>
> sysfs on /target/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> fusectl on /target/sys/fs/fuse/connections type fusectl
> (rw,nosuid,nodev,noexec,relatime)
>
> This fails:
> # umount /target/sys
> But this is successful:
> # umount /target/sys/fs/fuse/connections
> # umount /target/sys
>
> See attached file with the crash. This happens with the revision in
> edge for arm64:
>
> edge arm64 21.06.01+git14.aa0dfb01 2543
> 2021-07-02T04:58:24.297995+00:00
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/subiquity/+bug/1934775/+subscriptions
>

I'm not sure the suggestion in #2 will work in fact, as I don't think /sys/fs/fuse/connections will even exist when setting up /target as a chroot.

This bug is fixed with commit 1f663e51 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=1f663e51

This bug is believed to be fixed in curtin in version 21.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.