ssh can't call a binary from a snap without the full path

This affects the mosh snap: https://github.com/elopio/mosh/blob/snapcraft/snapcraft.yaml

Comment from ogra in the mailing list:

bash actually checks the calling process and will not source the env if
it is invoked by sshd or rshd as non-login shell (it checks if
the SSH_CLIENT environment variable is set and uses a different code
path internally).

the follwing works:

$ ssh 192.168.2.91 env|grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/
games:/usr/local/games

$ ssh 192.168.2.91 bash -lc env|grep PATH
PATH=/home/ogra/bin:/home/ogra/.local/bin:/usr/local/sbin:/usr/local/bi
n:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

you could also do something like:
ssh 192.168.2.91 "source /etc/profile; hello"

i dont think we have a bug open for this and technically it is expected
behaviour (not different from any other ubuntu install), but given how
annoying it is i guess you should file one and we should find a proper
workaround or fix to ship in the image ...

Why is the PATH being set via login shells rather than e.g. init's environment variables or PAM environment mechanism? What led to setting the environment variables via shell startup scripts?

Thanks

 /etc/profile.d/apps-bin-path.sh does not fire on noninteractive logins.

$ ssh pi3 'echo $PATH'
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Workaround: make it fire, dammit:

--- .bashrc.orig 2017-01-04 19:55:33.105726685 +0000
+++ .bashrc 2017-01-04 19:54:11.505653043 +0000
@@ -5,7 +5,9 @@
 # If not running interactively, don't do anything
 case $- in
     *i*) ;;
- *) return;;
+ *) # Work around rsync snap PATH problem
+ . /etc/profile.d/apps-bin-path.sh
+ return;;
 esac

 # don't put duplicate lines or lines starting with space in the history.

@seth because it is dynamically shipped with the snapd package for classic installs so PATH gets automatically extended with install/removal of the snapd package.

for image builds i am now adding it to /etc/environment (which is traditionally the place for the global PATH variable in debian and Ubuntu) at build time, that should fix the issue for the images at least.

so with todays edge core snap it now looks like:

$ ssh 192.168.2.76 env |grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
$

Still hitting this with...

jenkins@juju-6f7422-solutions-qa-integration-1:~$ snap version
snap 2.27.6
snapd 2.27.6
series 16
ubuntu 16.04
kernel 4.4.0-83-generic
jenkins@juju-6f7422-solutions-qa-integration-1:~$ snap list
Name Version Rev Developer Notes
conjure-up 2.3.1 741 canonical classic
core 16-2.27.6 2898 canonical core
weebl-tools 1.1.8 50 autonomouse -

was it simply never released, or did we regress somehow, or am I missing something else?

this has been fixed long ago and you should have an /etc/profile.d/apps-bin-path.sh file that adds /snap/bin to your PATH

Yes it obviously works for login shells.

To make it work for non-login shells /etc/environment would need to include /snap/bin -- an issue for foundations to decide upon.

Thanks for the insight.

Adding a task against the Ubuntu snapd package for the non-interactive shell case (eg. ssh), since this still affects Ubuntu the distribution on this point.

Status changed to 'Confirmed' because the bug affects multiple users.

I'm very reluctant to reconfigure openssh for this; it feels like a layering violation, and a regression of all the work we did way back when to avoid having to change umpteen packages when we want to change the default command search path.

How about something like adding a pam_snap session module that would add itself to PATH in the PAM environment if needed, and plug that in via /usr/share/pam-configs/ ? You'd want to discuss that sort of thing with Steve Langasek, I expect, but it seems nicer than changing the configuration of individual PAM-using services.

Status changed to 'Confirmed' because the bug affects multiple users.

This is fairly problematic when core functionality that is orchestrated in the cloud moves from deb to snap. In our example, lxc (LXD command line tool) was assumed to be in the path when building the orchestration. Using the full path makes it less portable. It also makes managing multiple releases a pain since some hosts may run the snap, others installed by dpkg.

Is there any update on the intended direction?

I know that work was done to ensure /snap/bin was in the default path for systemd itself and that this should be inherited now by all units. 'systemd-path | grep search-binaries:' shows that it's there on my focal system. 'sudo ps axewwww | grep sshd' also shows that it's set in the PATH environment of the sshd parent processes. So I'm not sure what is still resetting the PATH for noninteractive ssh sessions.

It looks like the likely cause here is that sshd loads pam, which calls pam_env, which reads /etc/environment, which has a different PATH than the default one provided by systemd.

We should fix this to be aligned.

+1, tweaking /etc/environment makes it work. It seems that this has to be fixed in the libpam-modules postinst script then.

It's easy enough to unconditionally add /snap/bin to path, see attached, but I can't really see a clean way to only have it present in the $PATH when snapd is installed (without code changes to pam_env, anyway).

That won't be too different from /usr/local/sbin and /usr/local/bin being in the PATH but empty I think. I am not aware of any problem or annoyance caused by nonexisting directories being in the PATH.

Given Colin's comment 12, it doesn't look like there are any plans to change this in openssh packaging so I'll mark this Won't Fix.

The inconsistency between login and openssh session module ordering is still an issue. Reopening.

Here's a better patch that I've tested this time. I dropped the code around adding /usr/local/games to PATH as that was in trusty.

I've filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964580 about the pam ordering issue.

This bug was fixed in the package pam - 1.3.1-5ubuntu5

---------------
pam (1.3.1-5ubuntu5) groovy; urgency=medium

  * debian/libpam-modules.postinst: Add /snap/bin to $PATH in
    /etc/environment. (LP: #1659719)

 -- Michael Hudson-Doyle <email address hidden> Fri, 10 Jul 2020 08:35:49 +1200

Hello Leo, or anyone else affected,

Accepted pam into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pam/1.3.1-5ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted pam (1.3.1-5ubuntu4.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

kopanocore/8.7.0-7ubuntu1 (armhf)
cyrus-imapd/3.0.13-5 (armhf, arm64, s390x, ppc64el, amd64)
xpra/3.0.6+dfsg1-1build1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#pam

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hello Leo, or anyone else affected,

Accepted pam into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pam/1.1.8-3.6ubuntu2.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted pam (1.1.8-3.6ubuntu2.18.04.2) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
systemd/237-3ubuntu10.41 (ppc64el)
openssh/1:7.6p1-4ubuntu0.3 (arm64, amd64, i386, armhf, s390x, ppc64el)
vsftpd/3.0.3-9build1 (arm64)
lxc/3.0.3-0ubuntu1~18.04.1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#pam

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Verified on focal:

mwhudson@anduril:~$ ssh ubuntu@10.187.142.128 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.3.1-5ubuntu4
go: not found
mwhudson@anduril:~$ lxc exec test-pam-focal -- apt install -y libpam-modules/focal-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '1.3.1-5ubuntu4.1' (Ubuntu:20.04/focal-proposed [amd64]) for 'libpam-modules'
Selected version '1.3.1-5ubuntu4.1' (Ubuntu:20.04/focal-proposed [amd64]) for 'libpam-modules-bin' because of 'libpam-modules'
...
mwhudson@anduril:~$ ssh ubuntu@10.187.142.128 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.3.1-5ubuntu4.1
go is /snap/bin/go

The focal autopkgtest failures are in cyrus-imapd:

Mail::JMAPTalk version 0.15 required--this is only version 0.13 at ./Cassandane/Cyrus/JMAPCore.pm line 47.
BEGIN failed--compilation aborted at ./Cassandane/Cyrus/JMAPCore.pm line 47.

This seems very unlikely to be due to pam, and it happens with the version of pam anyway. The other failure is kopanocore/armhf which is known flaky: https://autopkgtest.ubuntu.com/packages/k/kopanocore/focal/armhf (I've retried it anyway).

And bionic:

mwhudson@anduril:~$ ssh ubuntu@10.187.142.63 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.1.8-3.6ubuntu2.18.04.1
go: not found
mwhudson@anduril:~$ lxc exec test-pam-bionic -- apt install -y libpam-modules/bionic-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '1.1.8-3.6ubuntu2.18.04.2' (Ubuntu:18.04/bionic-proposed [amd64]) for 'libpam-modules'
Selected version '1.1.8-3.6ubuntu2.18.04.2' (Ubuntu:18.04/bionic-proposed [amd64]) for 'libpam-modules-bin' because of 'libpam-modules'
...
mwhudson@anduril:~$ ssh ubuntu@10.187.142.63 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.1.8-3.6ubuntu2.18.04.2
go is /snap/bin/go

The autopkgtest failure, dovecot/armhf looks to be a flake. I've retried it anyway to see if it passes now.

This bug was fixed in the package pam - 1.1.8-3.6ubuntu2.18.04.2

---------------
pam (1.1.8-3.6ubuntu2.18.04.2) bionic; urgency=medium

  * debian/libpam-modules.postinst: Add /snap/bin to $PATH in
    /etc/environment. (LP: #1659719)

 -- Michael Hudson-Doyle <email address hidden> Wed, 22 Jul 2020 11:40:59 +1200

The verification of the Stable Release Update for pam has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

If kopanocore is known to be flaky on armhf please add a hint for it to https://code.launchpad.net/~ubuntu-sru/britney/hints-ubuntu-focal/.

This bug was fixed in the package pam - 1.3.1-5ubuntu4.1

---------------
pam (1.3.1-5ubuntu4.1) focal; urgency=medium

  * debian/libpam-modules.postinst: Add /snap/bin to $PATH in
    /etc/environment. (LP: #1659719)

 -- Michael Hudson-Doyle <email address hidden> Wed, 22 Jul 2020 11:30:44 +1200

The fix for this in the queue for Xenial is help up by the verification of bug 1666203.

An upload of pam to xenial-proposed has been rejected from the upload queue for the following reason: "needs sourceful fixes for the version previously in -proposed".

Hello Leo, or anyone else affected,

Accepted pam into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pam/1.1.8-3.2ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted pam (1.1.8-3.2ubuntu2.3) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/229-4ubuntu21.29 (amd64, i386)
kscreenlocker/5.5.5-0ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#pam

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

And verified on xenial:

(client-server-5)mwhudson@anduril:~/src/subiquity$ ssh ubuntu@10.187.142.193 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.1.8-3.2ubuntu2.1
go: not found
(client-server-5)mwhudson@anduril:~/src/subiquity$ lxc exec lp1659719-xenial -- apt install -y libpam-modules/xenial-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '1.1.8-3.2ubuntu2.3' (Ubuntu:16.04/xenial-proposed [amd64]) for 'libpam-modules'
Selected version '1.1.8-3.2ubuntu2.3' (Ubuntu:16.04/xenial-proposed [amd64]) for 'libpam-modules-bin' because of 'libpam-modules'
The following packages were automatically installed and are no longer required:
  libfreetype6 motd-news-config
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  libpam-modules-bin
Recommended packages:
  update-motd
The following packages will be upgraded:
  libpam-modules libpam-modules-bin
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 281 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libpam-modules-bin amd64 1.1.8-3.2ubuntu2.3 [36.9 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libpam-modules amd64 1.1.8-3.2ubuntu2.3 [244 kB]
Fetched 281 kB in 2s (123 kB/s)
Preconfiguring packages ...
(Reading database ... 25804 files and directories currently installed.)
Preparing to unpack .../libpam-modules-bin_1.1.8-3.2ubuntu2.3_amd64.deb ...
Unpacking libpam-modules-bin (1.1.8-3.2ubuntu2.3) over (1.1.8-3.2ubuntu2.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up libpam-modules-bin (1.1.8-3.2ubuntu2.3) ...
(Reading database ... 25804 files and directories currently installed.)
Preparing to unpack .../libpam-modules_1.1.8-3.2ubuntu2.3_amd64.deb ...
Unpacking libpam-modules:amd64 (1.1.8-3.2ubuntu2.3) over (1.1.8-3.2ubuntu2.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up libpam-modules:amd64 (1.1.8-3.2ubuntu2.3) ...
(client-server-5)mwhudson@anduril:~/src/subiquity$ ssh ubuntu@10.187.142.193 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.1.8-3.2ubuntu2.3
go is /snap/bin/go

This bug was fixed in the package pam - 1.1.8-3.2ubuntu2.3

---------------
pam (1.1.8-3.2ubuntu2.3) xenial; urgency=medium

  * Move patch fixing LP: #1666203 from debian/patches to
    debian/patches-applied so it actually gets applied.
  * debian/libpam-modules.postinst: Add /snap/bin to $PATH in
    /etc/environment. (LP: #1659719)

pam (1.1.8-3.2ubuntu2.2) xenial; urgency=medium

  * Fix: pam_tty_audit failed in pam_open_session (LP: #1666203)

 -- Michael Hudson-Doyle <email address hidden> Thu, 01 Oct 2020 10:03:21 +1300