rm -r * fails to delete directories when using overlayfs in a user-namespace

Bug #1480411 reported by oleg
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Medium
Andy Whitcroft

Bug Description

rm -r * fails to delete directories when using overlayfs in a user-namespace.

If overlayfs is mounted as follows,

   mount -n -t overlay overlay -o lowerdir=lowerdir,upperdir=upperdir,workdir=workdir mntpt

and if lowerdir contains lowerdir/foo/bar.txt, then,

   rm -r mntpt/foo

fails with error message,

   "rm: cannot remove ‘mntpt/foo’: Operation not permitted"

strace shows,

    unlinkat(AT_FDCWD, "mntpt/foo", AT_REMOVEDIR) = -1 EPERM (Operation not permitted)

OS details: 64-bit ubuntu 14.04.2 with the linux-generic-lts-vivid kernel (3.19).

The bug does not occur on ubuntu 14.04.2 with the default linux-generic 3.13 kernel.

The bug occurs for any vanilla 3.18+ kernel to which the 1-line patch, "overlayfs: allow unprivileged mounts" is added. (The patch is at, http://kernel.ubuntu.com/git/ubuntu/ubuntu-vivid.git/commit/?id=78ec45495b15d27d4cc6d05cc454e30ec5b587ea)

The bug occurs regardless of whether the filesystem underlying overlayfs is tmpfs or ext4.

The bug also occurs when running ubuntu 15.04 (ubuntu vivid) in a virtual machine (qemu-system-x86_64). The host is ubuntu 14.04.2 with the default 3.13 kernel.

A script which reproduces the bug is attached and is also available at http://paste.ubuntu.com/11974137/

Hardware: intel core2 duo processor in a macbook-4.1

Revision history for this message
oleg (overlayfs) wrote :
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Andy Whitcroft (apw)
milestone: none → ubuntu-15.08
oleg (overlayfs)
description: updated
tags: added: bot-stop-nagging
oleg (overlayfs)
description: updated
Revision history for this message
Thomas Müller (muellerthomas977) wrote :

I can reproduce this problem on a fresh machine set-up at AWS (kernel Linux ip-172-31-6-51 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux).

Below the output of the script reproduction script linked above:

```
mounted overlayfs

ls -RF mntpt
 -------------
mntpt:
foo/

mntpt/foo:
bar.txt

rm -r mntpt/*
 -------------
rm: cannot remove ‘mntpt/foo’: Operation not permitted
exit code=1

ls -RF mntpt
 -------------
mntpt:
foo/

mntpt/foo:

unmounted overlayfs
cleaning up
```

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in lxc (Ubuntu):
status: New → Confirmed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

This is not a bug. Easiest way is to

lxc-usernsexec -- rm -rf $container_dir/
rm -rf $container_dir/

Changed in lxc (Ubuntu):
status: Confirmed → Invalid
Changed in linux (Ubuntu):
status: Triaged → Invalid
Revision history for this message
oleg (overlayfs) wrote :

I don't quite get the point underlying Serge's preceding comment, so I'll describe the problem I'm experiencing with lxc and see whether Serge classifies it as a bug.

On an ubuntu-vivid (amd64) host, create an unprivileged vivid container named 'vivid',
   lxc-create -n vivid -t download -- -d ubuntu -r vivid -a amd64
Start and attach to the container and create foo/bar.txt . Stop the container.

Then form a second container named vivid_overlay,
   lxc-clone -s -B overlayfs vivid vivid_overlay
Start and attach to vivid_overlay.
Delete bar.txt (this should succeed).
Attempt to delete foo. I get "Operation not permitted", even as root in the container.

The same issue arises if I replace the overlay container with an ephemeral container,
   lxc-start-ephemeral -o vivid -n vivid_ephemeral

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Ah, sorry. I thought you were trying to manually delete the whole container (instead of using lxc-destroy).

Changed in lxc (Ubuntu):
status: Invalid → New
Changed in linux (Ubuntu):
status: Invalid → New
status: New → Confirmed
Changed in lxc (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
importance: Medium → High
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

(confirmed here on my wily laptop - thanks for reporting this)

Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.08 → ubuntu-15.09
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.09 → ubuntu-15.10
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.10 → ubuntu-15.11
no longer affects: lxc (Ubuntu)
oleg (overlayfs)
Changed in lxc (Ubuntu):
status: New → Confirmed
Revision history for this message
oleg (overlayfs) wrote :

This bug does still affect lxc (as per comment #5) when using kernel 3.18 or above in conjunction with the latest daily-build of lxc, 1.1.5+master~20151110-0623-0ubuntu1~trusty (from the lxc-daily ppa, https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/daily).

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@oleg,

yes, but it is not an lxc bug, there's nothing lxc can do about it. Stéphane un-marked it from lxc to make the lxc bug view more usable so we can use it rather than ignore it :)

Revision history for this message
oleg (overlayfs) wrote :

@Serge,

Thanks for the clarification; I'll revert the status to un-marked.

no longer affects: lxc (Ubuntu)
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.11 → ubuntu-15.12
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.12 → ubuntu-16.01
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-16.01 → ubuntu-16.02
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.