snapctl used from classic snaps with non-core bases on debian 9 is broken

The problem is caused by a very old snapd deb 2.21; even though snapd itself re-execs into 2.51, snapctl from the host (i.e. 2.21) is still used when executing snapctl either from the host or from classic snaps. The old snapctl tries to use /run/snapd.socket (and is denied the access), while nowadays snapd expects snapctl to talk via /run/snapd-snap.socket. This can be observed also with 'snap run --shell certbot', followed by manual invocation of snapctl as detailed in the github issue linked in the description.

The problem isn't new though, it must have been the case for a few years already, I suppose the affected snap didn't use snapctl before.

Thanks for finding that.

>The problem isn't new though, it must have been the case for a few years already, I suppose the affected snap didn't use snapctl before.

If it helps, snapctl has been used in the prepare-plug-plugin hook since June 2020. I recall successfully testing this on Stretch on a number of occasions between now and then, but I could certainly be mistaken about that.

Would it be acceptable to have a newer snapd for stretch in something like -backports?

> Would it be acceptable to have a newer snapd for stretch in something like -backports?

I don't think so. I think it's a reasonable expectation that the certbot snap, with its plugins, works out of the box with the snapd that shipped with Debian. Otherwise, you're effectively saying that you don't care about regressing snapd as shipped on non-Ubuntu distributions, and that snaps do not operate universally across distributions as advertised.

This also appears to be a recent regression. Apparently something changed in the core snap that caused the regression.

Note that the certbot snap is based on core20, so I'm puzzled as to why installing "core" triggers the issue.

> I think it's a reasonable expectation that the certbot snap, with its plugins, works out of the box with the snapd that shipped with Debian.

I tend to disagree with this line of thinking, no software is perfectly without bugs at a given point in time, by trying to upload a new version of snapd to backports we would be providing a fix for this bug, to me it sounds like you are saying that we can't fix bugs...

> Otherwise, you're effectively saying that you don't care about regressing snapd as shipped on non-Ubuntu distributions

I don't think that's what's being said at all, we do care, there was just a bug in that old version of snapd 2.21 that we have since fixed and so hence we would need to update snapd there.

One workaround the certbot snap could do to workaround this ancient version of snapd in Debian would be to always require the core or snapd snaps to be installed, and then to execute `/snap/{snapd,core}/current/usr/bin/snapctl` for it's snapctl things. You could ensure that the core/snapd snaps are installed via:

```
if ! snap list core 2>/dev/null && ! snap list snapd 2>/dev/null; then
    snap install snapd
    SNAPCTL=/snap/snapd/current/usr/bin/snapctl
else
    if snap list core; then
        SNAPCTL=/snap/core/current/usr/bin/snapctl
    else
        SNAPCTL=/snap/snapd/current/usr/bin/snapctl
    fi
fi

$SNAPCTL foo ...
```

> I tend to disagree with this line of thinking, no software is perfectly without bugs at a given point in time, by trying to upload a new version of snapd to backports we would be providing a fix for this bug, to me it sounds like you are saying that we can't fix bugs...

Not at all. You're framing this as a bug in snapd. This may be true, but the report points to the regression being triggered by a change in the core snap, and this suggests that the obvious resolution is to identify and immediately revert the change that triggered the issue. I'm surprised that an inadvertent regression that breaks users doesn't result in you immediately investigating and reverting the change that you made that triggered the bug. Instead you're leaving users broken while pushing the burden of dealing with this away from yourselves.

The problem with fixing a bug in backports is that users following your own instructions at https://snapcraft.io/docs/installing-snap-on-debian wouldn't receive the fix since backports is opt-in. That would leave things broken for Debian users by default.

> ...there was just a bug in that old version of snapd 2.21 that we have since fixed and so hence we would need to update snapd there.

I wonder if you could provide a minimal patch for snapd 2.21 please? Maybe it'd be possible to get such a fix into stretch-updates, although I'm not sure if that is still open.

> One workaround the certbot snap could do to workaround this ancient version of snapd in Debian would be to always require the core or snapd snaps to be installed, and then to execute `/snap/{snapd,core}/current/usr/bin/snapctl` for it's snapctl things.

Thank you for the suggestion! I'll leave this for the upstream certbot snap developers to investigate.

This came up in another context again, so I am going to try to leave some comments which hopefully make the situation more clear (because it is confusing and there are too many things which are named the same):

* snapctl comes from the snapd project
* snapd the project gets shipped onto a user's system through multiple possible places/packages
* snapd can come from the `core` snap, the `snapd` snap or from traditional linux packages like the debian package for Debian 9
* for most utilities, we have implemented re-exec logic so that when you run /usr/bin/snap for example, it will re-exec through the most recent version of code from wherever that is, i.e. if the deb is 2.21 and the snapd snap is 2.51, then /usr/bin/snap will exec (with the same args) /snap/snapd/current/usr/bin/snap
* We did not have re-exec logic implemented for snapctl as of 2.21
* Rather than implement re-exec logic for snapctl, we instead opted to have snapd when it sets up the mount namespace pick the most recent/correct version of snapctl from wherever that might be, and mount it in the mount namespace for the snap at /usr/bin/snapctl, so that strict snaps do not need to care about where snapctl is coming from they can always rely on /usr/bin/snapctl being available and being the right version.
* None of that description for snapctl mounting applies for classic snaps, which we did not consider at the time and in retrospect probably opting for re-exec logic directly inside snapctl would have been a better idea because it would have solved the problem for classic snaps as well
* This means that old snapctl's do not know any better to try and re-exec themselves into a newer version, and also that we do not get a chance to "fix" the version of snapctl for classic snaps since classic snaps do not enter into a mount namespace for us to mount[1]

This problem with snapctl may appear on the surface to be a "new" regression in snapd, but that is only because of a different set of facts:
* It used to be that there was only one base snap for snaps to use, the core snap. The core snap is/was unique in that it contained _both_ snapd the project and the rootfs that application snaps use/depend on. That means that snapctl is included in the core snap
* For bionic, we realized that it would make more sense to split snapd out into it's own snap, the snapd snap, and then to ship the rootfs for snaps to execute from into separate "base snaps" such as core18 and core20, etc.
* It is my understanding that wrapper scripts were setup in snapcraft to make the default $PATH include the the base snap even for classic snaps, this effectively meant that /snap/core/current/usr/bin ended up being higher in priority than /usr/bin, this was to facilitate classic snaps operating more correctly by using stuff from their base rather than from the host.
* Now, if a snap used to use `base: core` (or not specify a base at all in the snapcraft.yaml), and it transitioned to using `base: core18`, since core18 does not include snapd the project, it will appear as if snapctl changed, because the $PATH now has changed, where now /usr/bin/snapctl will show up before something else like /snap/snapd/current/usr/bin/...