Comment 0 for bug 2016398

Hi

BACKGROUND:
-----------
I have a set of scripts that debootstraps and builds vanilla Debian installs, where the only modifications are what packages are installed. Then that becomes the lower directory of an overlayfs mount, a tmpfs becomes the upperdir, and then that becomes the chroot where config changes are made with more scripts. This overlayfs is mounted in its own mount namespace as the script is unshare'd

Within these scripts, I make packages with a fork of checkinstall I made which uses the / as the lowerdir, as overlayfs allows that, a tmpfs as the upperdir, the install command is run with chroot, and then the contents are copied.

THE ISSUE:
----------
I noticed that it appears that my ramdisks are remaining in memory, I have isolated it out to the overlayfs mounts with the overlayfs as the lowerdir where chroot is being called on it. I tried entering the namespace myself and umounting it, and notice I get the error "Target is busy"

With the "Target Is Busy" issue, I tried to `lsof` and `fuser` and `lsfd` as much as I could, but every user mode tool shows absolutely NO files open at all

I was using Kubuntu 18.04, it was an old install and 32 bit, so I had no upgrade path to be using more recent versions. Now I am on 23.04, but I also think this is happening on a 22.10 laptop

DIAGNOSIS:
----------
I am able to isolate it out, and get this to replicate in the smallest script that I could that is now attached. the chroot is important. I am able to unmount the second overlayfs if chroot is never called. Nothing ever has to run IN the chroot, trying to call a binary that does not exist, and then trying to unmount it results in "target is busy" with no open files reportable by user mode tools

FURTHER TESTING:
----------------
Trying to find out the version of Linux this was introduced, I made a script that builds a very minimal ext4 file for a VM with a small debootstrapped system, builds a kernel, and runs the kernel with QEMU..
but when I started, after all that, I can't replicate it at all with a vanilla 6.2.0 kernel. I used the same config out of my /boot to the .config, only tweaking a few options like SYSTEM_TRUSTED_KEYS. Trying to unmount the second filesystem actually results in success

I then built a Kubuntu Lunar VM, and was able to replicate it on not just my system. When I built and install the vanilla kernel with bindeb-pkg, and then I boot the vanilla kernel, I can't replicate it, the second filesystem unmounts. When I reboot the VM, and start the default distribution kernel, I am able to replicate the issue, the second filesystem being unable to unmount, with "target is busy" with no files open according to standard usermode tools, when chroot is called on it

I tried to look at patches in
git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/lunar
and git log --oneline --grep="sauce" but none of them really look relevant. I could be VERY wrong in this matter though

SUMMARY:
--------
Long story short, (I am VERY sorry about the length:)

I make an overlayfs,
I make a second overlayfs with the first overlayfs as the second one's lowerdir,
I chroot into the second overlayfs, and then later I can't unmount the second one as it says "Target is busy", but the user mode tools don't show any files open at all. If I mount the second overlayfs, and then unmount it without ever chroot-ing into it I AM able to unmount it

The attached script should replicate the issue

This doesn't seem to happen on the vanilla Linux kernel, with the same as close as possible config that Ubuntu uses

Thanks

ProblemType: Bug
DistroRelease: Ubuntu 23.04
Package: linux-image-6.2.0-20-generic 6.2.0-20.20
ProcVersionSignature: Ubuntu 6.2.0-20.20-generic 6.2.6
Uname: Linux 6.2.0-20-generic x86_64
ApportVersion: 2.26.1-0ubuntu2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: nerdopolis 3007 F.... wireplumber
 /dev/snd/controlC0: nerdopolis 3007 F.... wireplumber
 /dev/snd/seq: nerdopolis 3003 F.... pipewire
CasperMD5CheckResult: pass
CurrentDesktop: KDE
Date: Sat Apr 15 19:08:48 2023
InstallationDate: Installed on 2023-04-09 (6 days ago)
InstallationMedia: Kubuntu 23.04 "Lunar Lobster" - Daily amd64 (20230408)
MachineType: ASRock B650M PG Riptide
ProcFB: 0 amdgpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.2.0-20-generic root=/dev/mapper/vgkubuntu-root ro quiet splash amdgpu.sg_display=0
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-6.2.0-20-generic N/A
 linux-backports-modules-6.2.0-20-generic N/A
 linux-firmware 20230323.gitbcdcfbcf-0ubuntu1
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/22/2023
dmi.bios.release: 5.26
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: 1.18
dmi.board.asset.tag: Default string
dmi.board.name: B650M PG Riptide
dmi.board.vendor: ASRock
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvr1.18:bd02/22/2023:br5.26:svnASRock:pnB650MPGRiptide:pvrDefaultstring:rvnASRock:rnB650MPGRiptide:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: B650M PG Riptide
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: ASRock