apparmor doesn't work on stacked file system (livecd) -- DHCP/cups/others fail to start

Can you do a "apt-get -f install"

This is the cause:

/usr/sbin/cupsd: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Operation not permitted

I never saw that, though. Kees, Mathias, is this from ASLR or from AppArmor?

Wow; I've never seen that before. I'm doubting this is from ASLR;
that's unchanged since Edgy. And the /etc/apparmor.d/abstractions/base
file correctly includes:

  /lib/tls/i686/cmov/lib*.so* mr,

So I'm at a loss for what that means. Code-diving shows this to be an
error from libc where it can't call mprotect relro on itself:

void internal_function
_dl_protect_relro (struct link_map *l)
{
  ElfW(Addr) start = ((l->l_addr + l->l_relro_addr)
              & ~(GLRO(dl_pagesize) - 1));
  ElfW(Addr) end = ((l->l_addr + l->l_relro_addr + l->l_relro_size)
            & ~(GLRO(dl_pagesize) - 1));

  if (start != end
      && __mprotect ((void *) start, end - start, PROT_READ) < 0)
    {
      static const char errstring[] = N_("\
cannot apply additional memory protection after relocation");
      _dl_signal_error (errno, l->l_name, NULL, errstring);
    }
}

I'm at loss here, too. cupsys' configure does support enabling relro, but it is not enabled by default, and AFAIK our compiler does not enable it by default either.

Can you reproduce this failure? It should not only happen during dist-upgrade, cupsd should not start at all.

Can you please try the following:

  sudo /etc/init.d/cupsys stop
  sudo cupsd -f

Does above command crash or continue to run? If it crashes, can you please attach /var/log/cups/error_log?

Jonathan gets that, too.

Ah, this only happens on the live system. Apparmor seems to stuble over the /rofs and /cow prefixes.

Preferably this should be fixed in apparmor itself, layered file systems like unionfs and squashfs with underlying different prefixes shouldn't break AppArmor rules.

A shy workaround in cups itself would be to install a casper hook to disable cups' apparmor protection on the live CD.

Printing does not work with the Gutsy Tribe 5 live CD fully updated.
'sudo cupsd -f' and 'sudo /etc/init.d/cupsys start' both crashes.
In folder /var/log/cups there is only one empty file cups-pdf_log, no error_log.

For both of these cases, if you look in /var/log/messages you can see that AppArmor is rejecting access to /rofs/lib/tls/i686/cmov/libc-2.6.1.so

A simple fix is to update the profiles to use
{/rofs,/cow,}/lib/tls/i686/cmov/*.so

AppArmor can block access to stacked filesystem paths depending on how the stacked fs is implemented.
The quickest solution at the moment is to modify the profiles, so that permissions are given for the given prefix.
I would probably setup a variable in global
@{LIB}={/ros,/cow,}/lib

and modify profiles to use the variable.
@{LIB}/tls/i686/cmov/*.so

The other possible solution I can suggest is further modification of the unionfs patch for unionfs 2.x under AppArmor 2.1 code base. It is possible to further patch unionfs so that it passes the lower filesystem requests so that AppArmor doesn't mediate them. This would get rid of the need to update profiles as suggested above.

I just played around with this a bit. It is not enough to add just two or three prefixes here and there to work around the problem, it needs dozens. So it would be utterly preferable to get this fixed in apparmor proper.

I'll create a casper hook with:

sed -i '/{/ s/{/flags=(complain) {/' /etc/apparmor.d/usr.sbin.cupsd

While I agree this is something needs to address with mount rules, I can't give an eta for when it will happen.

In the mean time is it feasible to use variables so the prefixes can be all added in one place?

Feasible yes, but too clumsy IMHO. Instead I'll just set the profiles to complain mode on the live CD, that's a good enough workaround IMHO.

casper (1.101) gutsy; urgency=low

  * Add scripts/casper-bottom/42disable_cups_apparmor: Disable AppArmor
    protection for cups on the live CD by switching the profiles to complain
    mode. This is necessary until AppArmor works properly on stacked file
    systems. (LP: #131976)

 -- Martin Pitt <email address hidden> Wed, 12 Sep 2007 16:44:02 +0200

This will be a problem for every profiles that would be shipped by packages on the livecd. Why not completly disabling apparmor on the live cd with "update-rc.d -f apparmor remove" ?

Mathias, good idea. Then we should also able to deliver the profile for dhclient.

casper (1.102) gutsy; urgency=low

  * Rename 42disable_cups_apparmor to 42disable_apparmor and remove AppArmor's
    rc.d startup links instead of just disabling the cups profile. AA profiles
    will not generally work on the live system, so disabling it completely
    allows us to ship more profiles without worrying any further.
    (LP: #131976)

 -- Martin Pitt <email address hidden> Fri, 14 Sep 2007 08:37:50 +0200

I was really happy to see that someone else had this problem and that its being worked on. Can I ask, what would be the workaround in the meantime to get apt-get/synaptic to work? Mathias' update-rc.d suggestion above didn't seem to solve the problem... It appears apt just starts AppArmor anyway when it tries to install a package. I'm getting this error on persistent LiveUSB Xubuntu 7.10 if that changes anything.

@aselya1 Welcome to the world of persistent USB where when things get rebroken noone notices.
'sudo aa-complain /usr/sbin/cupsd/' will fix your problem super fast.
On on unrelated note be sure to pin your upstart at edgy.

Thanks for the welcome, as bittersweet as it is. Just to confirm, 'sudo aa-complain cupsd' fixed my problem, so thanks for that as well.

"On on unrelated note be sure to pin your upstart at edgy."
I looked into this issue from this suggestion, but it doesn't seem that I'm having this problem. I couldn't tell you why or why not though.

Here's hoping persistency gets more unbroken (or at least less broken) as Ubuntu progresses.

Just wanted to thank Christopher Barth - this fix worked flawlessly.
Except that final slash was not needed for me.
This issue was a real pain for me...

Cheers to persistent USB community :)

sudo aa-complain /usr/sbin/cupsd/ "sort of" fixes the problem, but I suspect only partially.

The error:

There was a problem committing changes. Possibly there was a problem downloading some packages or the commit would break packages.

goes away, but if I look in the log, I still get:

Setting up cupsys (1.3.2-1ubuntu7.1) ...
Reloading AppArmor profiles : done.
 * Starting Common Unix Printing System: cupsd /usr/sbin/cupsd: error while loading shared libraries: /lib/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
invoke-rc.d: initscript cupsys, action "start" failed.
dpkg: error processing cupsys (--configure):
 subprocess post-installation script returned error exit status 127

and near the end:

Errors were encountered while processing:
 cupsys

and at the very end:

Reloading AppArmor profiles : done.
 * Starting Common Unix Printing System: cupsd
    . . . . . . . .
dpkg run finished!

Thus, I think there is still a problem, though not enough to bother me as far as I can tell right now.

Hmm, I think I got a similar problem:

===
Setting up mysql-server-5.0 (5.0.51a-1ubuntu1) ...
 * Stopping MySQL database server mysqld /usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
                                                                         [ OK ]
Reloading AppArmor profiles : done.
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
 * Starting MySQL database server mysqld /usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: /lib/tls/i686/cmov/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
/usr/sbin/mysqld: error while loading shar...

Vincent can you attach your /var/log/messages & if present /var/log/audit/audit.log

I think that I just ran into this problem when upgrading to 8.04 to 7.10 on my usb stick...

Preconfiguring packages ...
Setting up cupsys (1.3.2-1ubuntu7.7) ...
Reloading AppArmor profiles Skipping profile /etc/apparmor.d/usr.sbin.cupsd.dpkg-old
: Warning.
 * Starting Common Unix Printing System: cupsd /usr/sbin/cupsd: error while loading shared libraries: /lib/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
invoke-rc.d: initscript cupsys, action "start" failed.
dpkg: error processing cupsys (--configure):
 subprocess post-installation script returned error exit status 127
Errors were encountered while processing:
 cupsys
E: Sub-process /usr/bin/dpkg returned an error code (1)

setting to complain worked but I noticed that the bug was marked as fixed in casper, so I thought I'd chime in.

crakarjax [2008-05-18 1:54 -0000]:
> setting to complain worked but I noticed that the bug was marked as
> fixed in casper, so I thought I'd chime in.

Thanks. However, casper only affects the live CD system, and casper's
workaround is to disable AppArmor on it. The actual bug in AA is still
open.

This seems to have regressed in karmic recently (it still worked in alpha-5 at least). Now we ship quite a fair bunch of apparmor profiles, and none work on the live system:

[ 315.217585] type=1503 audit(1253718188.795:69): operation="open" pid=4505 parent=4504 profile="/usr/sbin/cupsd" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/rofs/usr/lib/libcups.so.2"
[ 420.625182] __ratelimit: 9 callbacks suppressed
[ 420.625187] type=1503 audit(1253718294.203:73): operation="open" pid=4611 parent=2801 profile="/sbin/dhclient3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/cow/etc/ld.so.cache"
[ 420.625242] type=1503 audit(1253718294.203:74): operation="open" pid=4611 parent=2801 profile="/sbin/dhclient3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/rofs/lib/libc-2.10.1.so"

to give some examples. In other words, you can't even get on the network due to those.

So we either need a workaround again (like telling casper to disable apparmor on the live system), or a workaround in some generic apparmor rules to allow /cow/ and /rofs/ (this can be set by casper as well), or a fix in apparmor to not expose the underlying file system.

Is it possible that this change

apparmor (2.3.1+1403-0ubuntu21) karmic; urgency=low

  * debian/apparmor.{init-bottom,functions,initramfs}: perform initial
    apparmor rule loading in initramfs.

 -- Kees Cook <email address hidden> Mon, 21 Sep 2009 14:16:26 -0700

somehow triggered this regression? I really doubt that a breakage this large (not being able to get online) would have gone unnoticed in alpha-6, and I tested both i386/amd64 alpha-6 myself (dhcp worked just fine, I didn't test cups). Now I get it with the current amd64 live system on real iron, and with the i386 one in kvm.

Tentatively assigning to Kees, please assing to someone appropriate (Johan?)

Yes, this is related to moving AppArmor init into the initramfs. I'll have it uploaded shortly.

This bug was fixed in the package apparmor - 2.3.1+1403-0ubuntu22

---------------
apparmor (2.3.1+1403-0ubuntu22) karmic; urgency=low

  * Do not run AppArmor on the LiveCD, again (LP: #131976).
  * More aggressively stay quiet when booting in quiet mode (LP: #435285).

 -- Kees Cook <email address hidden> Wed, 23 Sep 2009 15:40:22 -0700

Turning apparmor off on the LiveCD is ignoring the underlying problem that apparmor doesn't work on stacked file systems. I'm currently developing a system that uses aufs as the root file system, and there's no way to use apparmor with it. I assume this is a bug that would need to be fixed upstream. Does anyone else know where or how to file a bug report that will actually get this fixed, rather than just covered up?

I am still experiencing this problem:

# ifup eth1
dhclient3: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
Failed to bring up eth1.

# dmesg | tail
[ 226.453040] type=1503 audit(1265397277.944:30): operation="open" pid=2983 parent=2982 profile="/sbin/dhclient3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/rofs/etc/ld.so.cache"
[ 226.453138] type=1503 audit(1265397277.944:31): operation="open" pid=2983 parent=2982 profile="/sbin/dhclient3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/rofs/lib/tls/i686/cmov/libc-2.10.1.so"
[ 226.454022] type=1503 audit(1265397277.944:32): operation="open" pid=2983 parent=2982 profile="/sbin/dhclient3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/rofs/lib/libc-2.10.1.so"
[0] matterhorn[ltsp]:~# aa-complain /sbin/dhclient3
Setting /sbin/dhclient3 to complain mode.

[0] matterhorn[ltsp]:~# ifup eth1
Internet Systems Consortium DHCP Client V3.1.2
Copyright 2004-2008 Internet Systems Consortium.
[....]

Note that this is not on the LiveCD, but on a LTSP client. Root is a unionfs of /rofs (NFS) and /cow (tmpfs).

Nikolaus upstream does watch here and track bugs filed in launchpad. This particular issue arise due to the LSM not providing AppArmor sufficient information about access on stacked file systems. You can mostly work around the issue using the alias command

can you try adding the following commands to your /etc/apparmor.d/tunables/alias file

alias /rofs -> /,
alias /cow -> /,

Doesn't seem to help:

[0] matterhorn[ltsp]:~# /sbin/dhclient3
/sbin/dhclient3: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
[127] matterhorn[ltsp]:~# cat /etc/apparmor.d/tunables/alias
# https://bugs.launchpad.net/bugs/131976
alias file

alias /rofs -> /,
alias /cow -> /,

Nikolaus,

did you restart apparmor after making the change? The profiles will have to be reloaded.
> sudo /etc/init.d/apparmor restart

if that doesn't work can you attach your profiles

It still does not work:

[0] matterhorn[ltsp]:~# cat /etc/apparmor.d/tunables/alias
# https://bugs.launchpad.net/bugs/131976
alias file
alias /rofs -> /,
alias /cow -> /,

[0] matterhorn[ltsp]:~# /etc/init.d/apparmor restart
 * Reloading AppArmor profiles Skipping profile in /etc/apparmor.d/disable: usr.bin.firefox-3.5
                                                                         [ OK ]
[0] matterhorn[ltsp]:~# /sbin/dhclient3
/sbin/dhclient3: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
[127] matterhorn[ltsp]:~# dmesg | tail
[ 238.455429] type=1505 audit(1265816743.526:36): operation="profile_replace" pid=3252 name=/usr/lib/connman/scripts/dhclient-script
[ 250.748581] type=1505 audit(1265816755.818:37): operation="profile_replace" pid=3253 name=/usr/bin/evince
[ 250.750097] type=1505 audit(1265816755.821:38): operation="profile_replace" pid=3253 name=/usr/bin/evince-previewer
[ 250.751089] type=1505 audit(1265816755.821:39): operation="profile_replace" pid=3253 name=/usr/bin/evince-thumbnailer
[ 251.059593] type=1505 audit(1265816756.130:40): operation="profile_replace" pid=3255 name=/usr/lib/cups/backend/cups-pdf
[ 251.059841] type=1505 audit(1265816756.130:41): operation="profile_replace" pid=3255 name=/usr/sbin/cupsd
[ 251.148214] type=1505 audit(1265816756.220:42): operation="profile_replace" pid=3256 name=/usr/sbin/tcpdump
[ 259.679327] type=1503 audit(1265816764.748:43): operation="open" pid=3280 parent=3024 profile="/sbin/dhclient3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/rofs/etc/ld.so.cache"
[ 259.679422] type=1503 audit(1265816764.748:44): operation="open" pid=3280 parent=3024 profile="/sbin/dhclient3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/rofs/lib/tls/i686/cmov/libc-2.10.1.so"
[ 259.680201] type=1503 audit(1265816764.752:45): operation="open" pid=3280 parent=3024 profile="/sbin/dhclient3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/rofs/lib/libc-2.10.1.so"

Which profile do you want me to attach?

Nikolaus,

thanks, it looks like you have hit a bug in the alias command, I'll look into this and see if I can't get something for you to test in the next couple of days.

On Wed, Feb 10, 2010 at 03:46:42PM -0000, Nikolaus Rath wrote:
> It still does not work:
>
> [0] matterhorn[ltsp]:~# cat /etc/apparmor.d/tunables/alias
> # https://bugs.launchpad.net/bugs/131976
> alias file
> alias /rofs -> /,
> alias /cow -> /,

Err, the alias mappings should be the other round; i.e.

  alias / -> /rofs,
  alias / -> /cow,

except that you don't want (or can't have) two mappings for the same
path *and* you will be mapping everything under / to it, which may
break if there's anything that's not covered by the stacked file
system. You probably want something more akin to:

  alias /etc -> /rofs/etc,
  alias /lib -> /rofs/lib,
  alias /usr -> /rofs/usr,
  alias /var -> /cow/var,

(continue as needed)

--
Steve Beattie
<email address hidden>
http://NxNW.org/~steve/

Right I messed up the alias order but there is still a bug in alias processing, that needs to be fixed before this will work. Basically setting
  alias /etc -> /rofs/etc,

will result in rejections for accesses to /etc/ instead of /rofs/etc/

unassigned MOTU, invalidated wrong dhcp & cupsys tasks

Is it possible this problem is caused by a sequencing problem with the Live CD scripts? I run the Karmic LiveCD in persistent mode on a hard disk drive. Once I am logged in to Ubuntu, I can run 'sudo /etc/init.d/apparmor start' and see no errors in the terminal nor the log files. The kernel log shows all the aparmor profiles are loaded. Everything works normally.

In looking at the Karmic initrd.lz content, it appears that the '/init' script calls '/scripts/init-bottom/apparmor', which then tests for the existence of a file called '/scripts/casper-bottom/42disable-apparmor'. The latter file will be found when booting with the Live CD software, and finding that file will cause '/scripts/init-bottom/apparmor' to early exit instead of loading the apparmor profiles. The apparent reason for the early exit, in lieu of loading the profiles, is the assumed failure of apparmor to deal deal with the union file system created by the live CD software. BUT, when the early exit is taken, the '/scripts/casper' script that sets up the union files system has NOT YET RUN, as far as I can see, so of course apparmor would have a problem finding and loading profiles?

Is it possible that simply changing the apparmor profile load process to follow the creation of the union file system, would allow apparmor to execute properly? It seems like an odd coincidence that lots of software EXCEPT apparmor appears to run flawlessly on the 'stacked' file system, and that apparmor is happy to load profiles after booting the Live CD?

On Tue, Jun 22, 2010 at 02:45:13AM -0000, Sid MacT wrote:
> Is it possible that simply changing the apparmor profile load process to
> follow the creation of the union file system, would allow apparmor to
> execute properly? It seems like an odd coincidence that lots of software
> EXCEPT apparmor appears to run flawlessly on the 'stacked' file system,
> and that apparmor is happy to load profiles after booting the Live CD?

That is precisely the bug: AppArmor does not operate on a stacked
filesystem, so it must be disabled in the LiveCD. AIUI, when proper
union mount support is added to the upstream kernel, these problems will
start to disappear. :)

http://valerieaurora.org/union/

IF the problem is the stacked file system, then why am I able to, after booting, do:

sudo /etc/init.d/apparmor start

on my persistent 9.10 Live CD and have no errors?

Is it possible that this 'bug' with the stacked file system is just a problem caused by the Ubuntu initrd.lz trying to load apparmor profiles before the casper script has had a chance to fully create the stacked file system? (please see prior post #44 that addresses the sequence of script events during the Live CD boot process) I am just wondering if the problem is with the Ubuntu customization of the 'init' and 'casper' scripts within the Ubuntu initrd.lz, and not really an upstream problem at all.

If you look at "sudo aa-status", I would expect that DHCP and CUPS are not
actually being confined by AppArmor, so there aren't errors. If you
restart the network or CUPS after starting AppArmor in the LiveCD, I would
expect those services to start failing. If that's _not_ the case, then
this should be examined further.

Karmic has long since stopped to receive any updates. Marking the Karmic task for this ticket as "Won't Fix".