lxc-start fails with 1.1.5-0ubuntu1

Bug #1516037 reported by Colin Watson
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Canonical System Image
Confirmed
Critical
Unassigned
apparmor (Ubuntu)
Confirmed
Undecided
Unassigned
lxc (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

After upgrading to lxc 1.1.5-0ubuntu1, lxc-start fails like this:

lxc-start: start.c: preserve_ns: 149 Permission denied - failed to open '/proc/7170/ns/mnt'
lxc-start: start.c: lxc_spawn: 993 failed to store namespace references
lxc-start: start.c: __lxc_start: 1192 failed to spawn 'trusty-lpdev'
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

This is with a trusty system container. precise system containers behave similarly. I don't have others to try. Downgrading liblxc1, lxc, lxc-templates, and python3-lxc to version 1.1.4-0ubuntu3 causes lxc-start to work again.

Revision history for this message
Colin Watson (cjwatson) wrote :
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Could you post the container configuration file?

Revision history for this message
Colin Watson (cjwatson) wrote :
Revision history for this message
Colin Watson (cjwatson) wrote :
Revision history for this message
Colin Watson (cjwatson) wrote :

But I should also note that the same thing happened with a freshly-created container, using "sudo lxc-create -n trusty-test -t ubuntu -- -r trusty". So I don't think my custom apparmor hack there can be at fault.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Hi Colin,

Could you:
 - Tell us what architecture you're running on (noticed the container config shows i386)
 - Confirm that both lxc and liblxc1 are at the same version
 - Tell us the exact kernel version you're running
 - Look for any apparmor denials in dmesg

Thanks!

Revision history for this message
Colin Watson (cjwatson) wrote :

My host is xenial amd64; the guest is i386 because a million lines of Python is rather more memory-efficient that way. :-)

Yes, lxc and liblxc1 are at the same version.

Kernel is linux-image-4.2.0-17-generic 4.2.0-17.21, though this also happened with 4.2.0-16.19 (I rebooted to see if that cleared it up).

There are no apparmor denials in dmesg.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

What does

ls -l /proc/sys/kernel/yama/
cat /proc/sys/kernel/yama/ptrace_scope

show?

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Nm, afaict yama should never prevent this.

Revision history for this message
Stéphane Graber (stgraber) wrote :

This bug was caused by a silent apparmor denial due to a local modification of /etc/apparmor.d/usr.bin.lxc-start (dropped attach_disconnected flag).

Changed in lxc (Ubuntu):
status: New → Invalid
Revision history for this message
Francis (francisd) wrote :

Mmm I'm having the same problem with lxc 1.1.5 on ubuntu 12.04 (using the ppa). I had to downgrade to 1.1.4 to make my containers bootable again.

I don't think I manually modified the file /etc/apparmor.d/usr.bin.lxc-start. It contains:

#include <tunables/global>

/usr/bin/lxc-start flags=(complain) {
  #include <abstractions/lxc/start-container>
}

I'm using LXC on two ubuntu 12.04 box and only one is failing with lxc-1.1.5.

Revision history for this message
Stéphane Graber (stgraber) wrote :

That most definitely is not our default profile, so you or some script you ran modified it.

You may want to install "debsums" and run "debsums -sa lxc" to get a list of configuration files which have been modified on your system.

Revision history for this message
Francis (francisd) wrote :

debsums: changed file /etc/default/lxc (from lxc package)
debsums: changed file /etc/apparmor.d/usr.bin.lxc-start (from lxc package)

So I restored the /etc/apparmor.d/usr.bin.lxc-start file to:

#include <tunables/global>

/usr/bin/lxc-start flags=(attach_disconnected) {
  #include <abstractions/lxc/start-container>
}

The only thing I remember I did is running aa-complain /usr/bin/lxc-start a while ago (I don't remember why) and this is what caused the file to be modified. I reverted aa-complain with aa-enforce /usr/bin/lxc-start, I restored the apparmor profil and redid the aa-complain and the file changed again to:

#include <tunables/global>

/usr/bin/lxc-start flags=(complain) {
  #include <abstractions/lxc/start-container>
}

Revision history for this message
Stéphane Graber (stgraber) wrote :

Adding an apparmor task as the apparmor tools really shouldn't mangle our profiles in a way that can't be reverted...

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apparmor (Ubuntu):
status: New → Confirmed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1516037] Re: lxc-start fails with 1.1.5-0ubuntu1

Ah, interesting. So this may be a bug in aa-complain, as it did not
retain (iiuc) the attach_disconnected flags.

Revision history for this message
Christian Boltz (cboltz) wrote :

Which apparmor version do you use?

I can't reproduce the problem using the latest version from the 2.9 branch and bzr trunk. (I tested with abstractions/base instead of abstractions/lxc/start-container in the profile, but that shouldn't matter.)

Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

It is also affecting devel-proposed phone images cf bug 1516971

Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

I confirm that the phone boots successfully after downgrading liblxc1, lxc and python3-lxc to version 1.1.4-0ubuntu3

Changed in canonical-devices-system-image:
importance: Undecided → Critical
status: New → Confirmed
Revision history for this message
Francis (francisd) wrote :

apparmor 2.7.102-0ubuntu3.10 running Ubuntu 12.04 LTS.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Oh, the phone problem is different I think, there the problem is that your kernel is pretty old and doesn't have the right /proc/self/ns files available. Looks like that's an actual bug with the new implementation. I'll report it upstream.

Jean-Baptiste, can you un-duplicate the phone bug and open a lxc task for it? It's a different issue from what was reported in this bug report.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Note that Jean-Baptiste's problem will only be seen on kernels higher than 3.2 and lower than 3.8. So it's not affecting any official distro kernel but it does affect those outdated android kernels.

Revision history for this message
Christian Boltz (cboltz) wrote :

AppArmor 2.7 is _very_ old - especially given the fact that the tools were rewritten in python for 2.9.

I just checked the perl code (which was used in 2.8.x and older) - it _sets_ the flags (instead of adding or removing them), so it's not surprising that attach_disconnected gets lost. (This is one of the fixes that went into the 2.9 during the rewrite to python.)

If someone is interested to fix this - the code is in Immunix/AppArmor.pm, sub complain() and sub enforce(), which both call setprofileflags().

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.