lxd-support interface doesn't appear to get properly connected/ready

Bug #1870201 reported by Chris Patterson
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
snapd
Triaged
High
Unassigned

Bug Description

This sequence happens many times on our spread tests and on one install of LXD, `waitready` fails because it appears that lxd-interface wasn't properly connected/ready. It is the first time I've encountered such an error, but thought it worth reporting.

+ apt-get install -y snapd

Reading package lists...

Building dependency tree...

Reading state information...

snapd is already the newest version (2.42.1+18.04).

snapd set to manually installed.

The following packages were automatically installed and are no longer required:

  dns-root-data dnsmasq-base ebtables grub-pc-bin liblxc-common liblxc1

  libnuma1 libuv1 lxcfs lxd-client uidmap xdelta3

Use 'apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 79 not upgraded.

+ '[' ubuntu-18.04-64 = ubuntu-16.04-64 ']'

+ '[' ubuntu-18.04-64 = ubuntu-18.04-64 ']'

+ apt-get remove --purge --yes lxd lxd-client

Reading package lists...

Building dependency tree...

Reading state information...

Package 'lxd' is not installed, so not removed

The following packages were automatically installed and are no longer required:

  dns-root-data dnsmasq-base ebtables grub-pc-bin liblxc-common liblxc1

  libnuma1 libuv1 lxcfs uidmap xdelta3

Use 'apt autoremove' to remove them.

The following packages will be REMOVED:

  lxd-client*

0 upgraded, 0 newly installed, 1 to remove and 79 not upgraded.

After this operation, 11.0 MB disk space will be freed.

(Reading database ... 145633 files and directories currently installed.)

Removing lxd-client (3.0.3-0ubuntu1~18.04.1) ...

Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

+ snap install lxd

2020-03-26T13:23:59Z INFO Waiting for restart...

2020-03-26T13:24:00Z INFO Waiting for restart...

lxd 3.23 from Canonical* installed

+ lxd waitready --timeout=30

cat: /proc/self/attr/current: Permission denied

/snap/lxd/13987/commands/lxd: 6: exec: aa-exec: Permission denied

Revision history for this message
Chris Patterson (cjp256) wrote :
Revision history for this message
Maciej Borzecki (maciek-borzecki) wrote :

I suspect a new core is being pulled, wonder if that increases the chances of this problem appearing?

Changed in snappy:
status: New → Triaged
importance: Undecided → High
affects: snappy → snapd
Revision history for this message
Callahan Kovacs (mr-cal) wrote :
Download full text (3.7 KiB)

We're still struggling with this bug in our spread tests. When running spread tests for *craft applications, we occasionally see the following failure:

+ snap install lxd
error: cannot perform the following tasks:
- Run configure hook of "lxd" snap if present (run hook "configure":
-----
cat: /proc/self/attr/current: Permission denied
/snap/lxd/23680/snap/hooks/configure: 5: exec: aa-exec: Permission denied
-----)

The failure is occurring in clean VMs with a stock Ubuntu image (18.04, 20.04, and 22.04). The failure rate for these steps is very low - less than 1% of the time. In our github workflow, we run ~350 spread tests, so the failure of the overall work flow is around 10-15% (although some weeks it has been closer to 50%).

The environment setup is brief. We apply the following steps to the image:

  apt-get install -y snapd
  snap install snapd
  snap wait system seed.loaded

  if [ "$SPREAD_SYSTEM" = "ubuntu-18.04-64" ] || [ "$SPREAD_SYSTEM" = "ubuntu-20.04-64" ]; then
      # Remove lxd and lxd-client deb packages as our implementation (pylxd) does not
      # nicely handle the snap and deb being installed at the same time.
      apt-get remove --purge --yes lxd lxd-client
  fi
  # Install and setup the lxd snap
  snap install lxd

------------------

Attempted solutions

I tried the 3 primary suggestions from this thread (https://forum.snapcraft.io/t/snapped-lxd-has-stopped-working-aa-exec-permission-denied/2356) with no luck:

1. Running snap interfaces (now snap connections)

This was suggested to check if the LXD snap is connected to lxd-support interface. This displays nothing, because the LXD snap is not installed yet.

2. Running snap install core20 and snap revert core20

This gives the error:
```
  + snap revert core20

  error: cannot revert "core20": no revision to revert to
```

3. Running snap install core20 and snap refresh core20

This still produces the same exec: aa-exec: Permission denied error.

I’ve captured the journal of a failure here (https://paste.ubuntu.com/p/XnyFRsvVfH/), with the failing section posted below:

```
oct142128-679423 audit[1763]: AVC apparmor="DENIED" operation="open" profile="snap.lxd.activate" name="/proc/1763/attr/current" pid=1763 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
oct142128-679423 audit[1764]: AVC apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 audit[1764]: AVC apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 kernel: kauditd_printk_skb: 4 callbacks suppressed
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:46): apparmor="DENIED" operation="open" profile="snap.lxd.activate" name="/proc/1763/attr/current" pid=1763 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:47): apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_m...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.