Ubuntu
linux-raspi2 package

[uc18] docker overlayfs* seems broken

Bug #1868894 reported by Tony Espy
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
snapd
Fix Released
Medium
Jamie Strandboge
snapd 2.45
linux-raspi2 (Ubuntu)
Confirmed
Undecided
Unassigned
Bionic
Invalid
Undecided
Unassigned

Bug Description

A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image.

Here are the details from the UC18 image.

$ snap list
core 16-2.43.3 8691 stable canonical✓ core
core18 20200124 1673 stable canonical✓ base
docker 18.09.9 427 stable canonical✓ -
pi 18-1 27 18-pi canonical✓ gadget
pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel
snapd 2.43.3 6438 stable canonical✓ snapd

And here's the apparmor denial:

Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world
Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways.

Please let me know if there's anything else I can provide.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-raspi2 (Ubuntu):
status: New → Confirmed
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I can't comment on the interaction of AppArmor and overlay with the available information. I can say that we already have these rules:

const dockerSupportConnectedPlugAppArmorCore = `
# These accesses are necessary for Ubuntu Core 16 and 18, likely due to the
# version of apparmor or the kernel which doesn't resolve the upper layer of an
# overlayfs mount correctly the accesses show up as runc trying to read from
# /system-data/var/snap/docker/common/var-lib-docker/overlay2/$SHA/diff/
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/common/{,**/} rwl,
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/@{SNAP_REVISION}/{,**/} rwl,
`

The denial of 'apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0' doesn't match this though, because '.dockerenv' is a file, not a directory. If I were to guess, I'd guess that perhaps the snap is overlaying a file rather than a dir, but again, I don't know for sure.

It would be fine to adjust the policy to use this instead:

/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/common/{,**} rwl,
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/@{SNAP_REVISION}/{,**} rwl,

since the snap already has read/write access to these directories when /system-data is not prepended. I've taken a todo to send up a PR for this.

Changed in snapd:
status: New → Triaged
assignee: nobody → Jamie Strandboge (jdstrand)
Michael Vogt (mvo)
Changed in snapd:
importance: Undecided → Medium
Revision history for this message
Maciej Borzecki (maciek-borzecki) wrote :

Ian proposed a PR with the change suggested by Jamie: https://github.com/snapcore/snapd/pull/8426

Revision history for this message
Ian Johnson (anonymouse67) wrote :

I noticed that this same denial happens on UC20 on amd64 VM with kernel 5.4.0-20-generic, so perhaps this is just a new kernel behavior, and the linux-raspi2 kernel has newer patches than all the other UC18 kernels we have available.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Denial on UC20 VM:

Apr 06 13:11:35 ubuntu kernel: audit: type=1400 audit(1586178695.710:59): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/ba5b8287c3c94aade8e3254eadbee5d4a4d279b4645e4337af9dd877c97ca1f2-init/diff/etc/resolv.conf" pid=1197 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

kernel info:

Linux ubuntu 5.4.0-20-generic #24-Ubuntu SMP Mon Mar 23 20:55:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Revision history for this message
Juerg Haefliger (juergh) wrote :

This was fixed in snapd: https://github.com/snapcore/snapd/pull/8426

Changed in snapd:
status: Triaged → Fix Released
Changed in linux-raspi2 (Ubuntu Bionic):
status: New → Invalid
Ian Johnson (anonymouse67)
Changed in snapd:
milestone: none → 2.45
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.