Deploying in an LXD container results in an inability to install kube-* snaps

Bug #1907153 reported by Chris Johnston
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Kubernetes Control Plane Charm
Triaged
High
Unassigned
Kubernetes Worker Charm
Triaged
High
Unassigned
snapd
Won't Fix
Undecided
Unassigned

Bug Description

Deploying kubernetes-master in a privileged container results in the kube-* snaps not being able to be installed:

sudo snap install kube-apiserver --channel=1.19/stable
error: cannot perform the following tasks:
- Run configure hook of "kube-apiserver" snap if present (run hook "configure": cannot perform operation: mount --bind /snap/core18/current/etc/apparmor /tmp/snap.rootfs_OYxyqR/etc/apparmor: Permission denied)

This appears to be due to a change that was recently released [1] which, in a privileged container, causes the container, which has a newer effective snap-confine profile than the host. Since the container does not have confinement, it reverts to using the hosts snap-confine profile from an older version of snapd (2.47.1) which does not contain the permissions to allow mounting /etc/apparmor inside the snap.

Upgrading the version of snapd on the host to 2.48+18.04 results in again being able to install the snaps in the container.

[1] https://github.com/snapcore/snapd/pull/9516

For @Canonical, entire discussion can be found at: https://chat.canonical.com/canonical/pl/brc4q5skzfbjinmyu4fd6nhcpy

Tags: sts
Revision history for this message
Chris Sanders (chris.sanders) wrote :

I'm marking this as invalid for the charm, this is a change in snapd that the charm has no control over.

Hopefully there's something snapd can do about graceful fallback in this case to avoid the regression in functionality.

Changed in charm-kubernetes-master:
status: New → Invalid
Revision history for this message
Ian Johnson (anonymouse67) wrote :

I'm setting this as WontFix because snapd can't support snaps being run inside privileged lxd containers, the host and container policies will clash with each other which is in fact exactly what happened here with this particular snap.

Changed in snapd:
status: New → Won't Fix
Changed in charm-kubernetes-master:
status: Invalid → Confirmed
Revision history for this message
Chris Sanders (chris.sanders) wrote :

After discussing with the snap and lxd teams, the issue is related to the profile which marks the apparmor profile as unconfined. The path forward to avoid this is to not override the lxc apparmor profile. That's what's causing the issues and not privileged setting.

tags: added: sts
Revision history for this message
George Kraft (cynerva) wrote :

Added kubernetes-worker because it likely has the same problem - both master and worker use the same LXD profile.

Changed in charm-kubernetes-master:
importance: Undecided → High
status: Confirmed → Triaged
Changed in charm-kubernetes-worker:
importance: Undecided → High
status: New → Triaged
Changed in charm-kubernetes-master:
assignee: nobody → Kevin W Monroe (kwmonroe)
Changed in charm-kubernetes-worker:
assignee: nobody → Kevin W Monroe (kwmonroe)
Changed in charm-kubernetes-master:
milestone: none → 1.22
Changed in charm-kubernetes-worker:
milestone: none → 1.22
Changed in charm-kubernetes-master:
status: Triaged → In Progress
Changed in charm-kubernetes-worker:
status: Triaged → In Progress
Revision history for this message
Kevin W Monroe (kwmonroe) wrote :
tags: added: review-needed
George Kraft (cynerva)
Changed in charm-kubernetes-master:
status: In Progress → Fix Committed
Changed in charm-kubernetes-worker:
status: In Progress → Fix Committed
tags: removed: review-needed
Revision history for this message
George Kraft (cynerva) wrote :
Changed in charm-kubernetes-worker:
status: Fix Committed → In Progress
George Kraft (cynerva)
Changed in charm-kubernetes-worker:
status: In Progress → Fix Committed
Revision history for this message
Kevin W Monroe (kwmonroe) wrote :

We had to revert this due to a regression for calico/vxlan when charms were deployed to lxd:

https://bugs.launchpad.net/charm-calico/+bug/1942099

PRs:
- https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/177
- https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/97

It's back on the list to fix for 1.22-ck1. The workaround for now is to ensure snapd versions are consistent between host and lxd VMs.

Changed in charm-kubernetes-master:
milestone: 1.22 → 1.22+ck1
Changed in charm-kubernetes-worker:
milestone: 1.22 → 1.22+ck1
Changed in charm-kubernetes-master:
status: Fix Committed → Triaged
Changed in charm-kubernetes-worker:
status: Fix Committed → Triaged
Changed in charm-kubernetes-master:
milestone: 1.22+ck1 → 1.23
Changed in charm-kubernetes-worker:
milestone: 1.22+ck1 → 1.23
Changed in charm-kubernetes-master:
assignee: Kevin W Monroe (kwmonroe) → nobody
milestone: 1.23 → 1.24
Changed in charm-kubernetes-worker:
assignee: Kevin W Monroe (kwmonroe) → nobody
milestone: 1.23 → 1.24
Changed in charm-kubernetes-master:
milestone: 1.24 → 1.24+ck1
Changed in charm-kubernetes-worker:
milestone: 1.24 → 1.24+ck1
Adam Dyess (addyess)
Changed in charm-kubernetes-master:
milestone: 1.24+ck1 → 1.25
Changed in charm-kubernetes-worker:
milestone: 1.24+ck1 → 1.25
Changed in charm-kubernetes-master:
milestone: 1.25 → none
Changed in charm-kubernetes-worker:
milestone: 1.25 → none
Revision history for this message
Kevin W Monroe (kwmonroe) wrote :

I've removed the milestone. We've verified the workaround here (ensure snapd versions across units), and that is less risky than a confined lxd profile considering calico+vxlan's importance in charmed k8s.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.