gvfsd process does not have the KRB5CCNAME environment set

Status changed to 'Confirmed' because the bug affects multiple users.

Happens on Ubuntu 19.04 too.

As far as I know this problem comes from ubuntu 12.04 and still not resolved. Kerberos now works in ubuntu 18.04.
This is ubuntu specific bug. Nautilus in Centos 7.x and 8 works fine with kerberos

The problem persists in Ubuntu 20.04 as well.

I attempted to investigate this issue a little further, found that it is caused by a race condition between gvfsd and ibus-daemon, and filed a bug report upstream: https://gitlab.gnome.org/GNOME/gvfs/-/issues/481

My current workaround is hack, but it works:

Add this line to the [Service] section in /usr/lib/systemd/user/gvfs-daemon.service:

ExecStartPre=bash -c "for i in echo {1..20} ; do ps ax | grep -q \"^${USER}\b.*[i]bus-daemon\" || sleep 0.1 ; done"

Thanks for reporting the issue upstream

The upstream determined that gvfsd as a systemd user session is starting too early in Ubuntu, before the desktop enviroment variables are set. Specifically, KRB5CCNAME env var is missing at the time gvfsd is started, causing this bug. See the detailed report at ttps://gitlab.gnome.org/GNOME/gvfs/-/issues/481

CentOS 8 and Fedora 32 apparently fixed this problem by changing the default preset mechanism for user units, aligning them with the default preset for system units: https://fedoraproject.org/wiki/Changes/Systemd_presets_for_user_units

Indeed, there is no good reason to treat user services differently than system services with respect to the default presets. The default preset for user units should be changed to "disable *".

The upstream bug got reopened now, could you try disabling the tracker service and see if it resolves the issue for you?

 I do not have tracker-miner-fs.service at all. My instance of gvfsd is started by either ibus-daemon, or "systemd --user". Please see the controversy at https://gitlab.gnome.org/GNOME/gvfs/-/issues/481#note_948506

I think anything which causes gvfs to start "early enough" (before gnome-session has a chance to upload the environment to systemd) will trigger this problem.

So anything which starts from e.g. default.target, or maybe there are things like ibus which are started outside of systemd's control.

I did a little bit of investigation on the upstream bug, and I think what is happening is that we don't run pam_sss for systemd-user sessions - it's not in /etc/pam.d/systemd-user or included in there e.g. via common-session-noninteractive. That means that when the session starts `systemd --user`, in its own `systemd-user` PAM session, the env var is not instantiated there and so it's not available to stuff that starts really early. Early - but not early enough - in the startup process, gnome-session uploads environment variables into the systemd environment. Anything which starts after that will get the right environment. In other words this is a race condition.

Can someone experiencing this bug please undo all of the workarounds applied, and then try adding "session optional pam_sss.so" into /etc/pam.d/systemd-user just above the `pam_systemd.so` line? And then check that KRB5CCNAME is set in gvfsd's environment.

I don't have an environment to fully test this so I was just able to do it with a hack, but it worked that far for me.

@laney: I have disabled all my workarounds and placed "session optional pam_sss.so" just before "session optional pam_systemd.so" in /etc/pam.d/systemd-user on my Ubuntu 18.04.5 system. Checking
"journalctl --user" for gfvs-daemon entries:

Nov 02 22:45:31 vk2011 dbus-daemon[6128]: [session uid=1000 pid=6128] Activating via systemd: service name='org.gtk.vfs.Daemon' unit='gvfs-daemon.service' requested by ':1.1' (uid=1000 pid=6121 comm="/usr/bin/ibus-daemon --daemonize --xim --address u" label="unconfined")

And, KRB5CCNAME is not in the environment for gvfsd, unfortunately.

I think maybe sssd needs to learn how to set the environment in "session" mode. I did ask some people at Canonical who know about this project, hopefully they will have some advice soon. :)

Same issue here.

gvfsd is started by tracker-miner-fs. Disabling it made it work (note that i also made the /etc/pam/systemd-user pam_sss change), ie gvfsd now correctly has access to the kerberos token.

I'm wondering why gvfsd is started by systemd-user and not gnome-session. Changing that may be an acceptable workaround until a better solution is found.

On Tue, Nov 17, 2020 at 07:50:20AM -0000, Julien Blanc wrote:
> Same issue here.
>
> gvfsd is started by tracker-miner-fs. Disabling it made it work (note
> that i also made the /etc/pam/systemd-user pam_sss change), ie gvfsd now
> correctly has access to the kerberos token.
>
> I'm wondering why gvfsd is started by systemd-user and not gnome-
> session. Changing that may be an acceptable workaround until a better
> solution is found.

It's being D-Bus activated by things early in the startup of the
session. By disabling tracker or whatever you are stopping that D-Bus
activation.

Cheers,

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

> It's being D-Bus activated by things early in the startup of the
> session. By disabling tracker or whatever you are stopping that D-Bus
> activation.

Indeed, i've seen that. But i actually intended to say tracker, not gvfs.

I see little point in starting tracker before gnome-session. But i'm probably missing something.

Regards,

Julien

I see the same problem with Ubuntu 20.04 and 22.04, inside an Active Directory domain.
With slow machines (e.g. with rotational hard disks) it is always present; with fast machines (with SSDs) it is randomly present, maybe because it depends also on the time needed to contact the domain controller.

(I'm using a cinnamon desktop and I do not have ibus installed)
I have applied the following workaround:
copy /usr/lib/systemd/user/gvfs-daemon.service to /etc/systemd/user/gvfs-daemon.service
insert in the last file the following line, at the start of the [Service] section:

ExecStartPre=bash -c "for i in echo {1..20} ; do if [ $(env | grep KRB5CCNAME) == "" ]; then sleep 0.2 ; fi ; done"

In this way it is possible to browse the network with Nemo or Nautilus, without asking for user authentication.

When the workaround is not present I see this message in /var/log/syslog:

Mar 30 10:30:36 pc000327 gvfsd[2656]: got no contact to IPC$
Mar 30 10:30:39 pc000327 gvfsd[2672]: Kerberos auth with 'aduser@WORKGROUP' (WORKGROUP\aduser) to access '10.1.0.107' not possible

(Here kerberos is not aware of the real domain name and tries the generic WORKGROUP)

I report here also the relevant processes in the case of no workaround:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
...
root 977 0.0 0.4 78692 19508 ? Ss 12:52 0:00 /usr/sbin/winbindd --foreground --no-process-group
root 985 0.0 0.2 78596 11168 ? S 12:52 0:00 winbindd: domain child [PC000327]
root 989 0.1 0.7 105008 28716 ? Ss 12:52 0:00 /usr/sbin/smbd --foreground --no-process-group
root 990 0.0 0.4 79840 17180 ? S 12:52 0:00 winbindd: domain child [DOMAIN]
root 994 0.0 0.4 80464 16344 ? S 12:52 0:00 winbindd: idmap child
root 1002 0.0 0.5 97132 20524 ? S 12:52 0:00 /usr/lib/x86_64-linux-gnu/samba/samba-bgqd --ready-signal-fd=48 --parent-watch-fd=12 --debuglevel=0 -F
root 1014 1.2 2.5 153464 99180 ? S 12:52 0:01 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
root 1015 0.0 0.3 98056 14612 ? S 12:52 0:00 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --logger=files
root 1021 0.1 0.2 57304 8144 ? Ss 12:52 0:00 /lib/systemd/systemd-logind
root 1121 15.2 2.7 272756 108420 ? S 12:52 0:18 /usr/libexec/sssd/sssd_be --domain addomain.it --uid 0 --gid 0 --logger=files
root 1213 0.0 0.2 190492 11168 ? Sl 12:53 0:00 lightdm --session-child 12 19
root 1271 0.0 0.4 138968 16544 ? Ss 12:53 0:00 /usr/libexec/sssd/sssd_pac --logger=files --socket-activated
aduser 1279 0.5 0.2 17388 10136 ? Ss 12:53 0:00 /lib/systemd/systemd --user
aduser 1292 0.6 0.1 29736 5824 ? Ss 12:53 0:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
aduser 1294 0.0 0.1 241124 7680 ? Sl 12:53 0:00 /usr/bin/gnome-keyring-daemon --daemonize --login
aduser 1298 0.1 0.2 240844 8672 ? Ssl 12:53 0:00 /usr/libexec/gvfsd
aduser ...

Hi everyone, Fady, renbag,

I have been working on this bug on and off for a little while now, but I am stuck because I can't reproduce what you are all seeing. Having a reproducer will greatly speed up getting a fix created for this issue.

In my client gvfsd is always started via systemd --user, so I must be configuring something differently. Can you try out my reproducer and let me know what you are configuring differently?

Instructions to reproduce:

You will need a 20.04 server instance, and a 20.04 Desktop instance.

To set up the server:

1) Create a fresh 20.04 server instance
2) sudo apt update
3) sudo apt upgrade
4) sudo hostnamectl set-hostname samba-dc
5) sudo vim /etc/hosts
Add an entry with its IP address, e.g.:
192.168.122.199 samba-dc samba-dc.example.com
6) sudo apt install -y samba smbclient winbind libpam-winbind libnss-winbind krb5-kdc libpam-krb5
Note: skip config of kerberos KDC.
7) sudo rm /etc/krb5.conf
8) sudo rm /etc/samba/smb.conf
9) sudo samba-tool domain provision --server-role=dc --use-rfc2307 --dns-backend=SAMBA_INTERNAL --realm=samba-dc.EXAMPLE.COM --domain=SAMBA --adminpass=Password1
10) sudo cp /var/lib/samba/private/krb5.conf /etc/krb5.conf
11) sudo systemctl mask smbd nmbd winbind
12) sudo systemctl disable smbd nmbd winbind
13) sudo systemctl stop smbd nmbd winbind
14) sudo systemctl unmask samba-ad-dc
15) sudo systemctl start samba-ad-dc
16) sudo systemctl enable samba-ad-dc
17) sudo reboot
18) sudo systemctl stop systemd-resolved
19) sudo systemctl disable systemd-resolved
20) cat << EOF >> /etc/resolv.conf
nameserver 192.168.122.199
search SAMBA
EOF
21) sudo reboot
22) host -t SRV _ldap._tcp.samba-dc.example.com
_ldap._tcp.samba-dc.example.com has SRV record 0 100 389 samba-dc.samba-dc.example.com.
23) $ smbclient -L localhost -N
Anonymous login successful

 Sharename Type Comment
 --------- ---- -------
 sysvol Disk
 netlogon Disk
 IPC$ IPC IPC Service (Samba 4.13.17-Ubuntu)
SMB1 disabled -- no workgroup available
24) $ smbclient //localhost/netlogon -UAdministrator -c 'ls'
Enter SAMBA\Administrator's password:
  . D 0 Mon Feb 28 04:23:22 2022
  .. D 0 Mon Feb 28 04:23:27 2022

  9983232 blocks of size 1024. 7995324 blocks available
25) kinit administrator
Password for <email address hidden>:
Warning: Your password will expire in 41 days on Mon Apr 11 04:23:27 2022
26) klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: <email address hidden>

Valid starting Expires Service principal
02/28/22 04:32:47 02/28/22 14:32:47 <email address hidden>
 renew until 03/01/22 04:32:44
27)

Create a share:
28) sudo mkdir -p /srv/samba/Demo/
29) sudo vim /etc/samba/smb.conf
[Demo]
 path = /srv/samba/Demo/
 read only = no
30) sudo chmod 0770 /srv/samba/Demo/

Install a fresh 20.04.4 Desktop instance, and run the following:

31) sudo apt install realmd smbclient
32) sudo vim /etc/hosts
Add an entry with its IP address, e.g.:
192.168.122.199 samba-dc samba-dc.example.com
33) sudo realm j...

Hi Matthew,

I report the complete configuration of the machine in which I see the problem.
The machine is an Optiplex 745, with an Intel Core2 6320 CPU, 4 GB RAM and a rotational HD, which I use as a test box for Ubuntu 22.04.
It was joined to an AD domain with the "net ads join -U aduser" command and uses sssd for authentication and samba and winbind for sharing folders.

The minimun number of iterations needed for the
ExecStartPre=bash -c "for i in echo {1..20} ; do if [ $(env | grep KRB5CCNAME) == "" ]; then sleep 0.2 ; fi ; done"
command to work is 15, so it's a delay of about 3 s.

I normally do not see the bug in my personal workstation, which runs Ubuntu 20.04 and is a much faster machine (Ryzen 5 with nvme SSD).

From the logs I can see that gvfsd is correctly started by systemd --user also in all my cases; so I suspect that the problem is that, with the slow machine, the kerberos ticket needed by gvfsd is actually written to the hard disk with too much delay.

I tried this to check if the problem is really the slow writing of the kerberos ticket to the disk:

disable the workaround in /etc/systemd/user/gvfs-daemon.service
reboot the slow machine
connect to the slow machine with ssh as "aduser" (a kerberos ticket is acquired and written to /tmp/krb5cc_1136602666_6v25tM and gvfsd is started)
from the ssh session: killall gvfsd
login as aduser in the normal graphical console of the slow machine

After login, Nemo is able to browse the network without asking username and password.
Note that, after login, the /tmp/krb5cc_1136602666_6v25tM ticket is replaced by a new one.

Hi renbag,

Thanks for attaching your smb.conf and sssd.conf, I will try add them into my reproducer and see if I get closer to seeing the problem.

Maybe when you log in, smbd mounts the samba shares to /home/aduser/{Public},{Shared} before kerberos manages to acquire a new ticket and place it in /tmp, so gvfs doesn't get KRB5CCNAME set.

Maybe on your faster system, it can get the kerberos ticket before smbd starts mounting shares.

This is still a race condition where gvfs is starting too early though.

Let me re-adjust my reproducer, and I will let you know how I get on.

Thanks,
Matthew

Is this bug still being worked on? I'm running into the issue. Took me a couple of days before I found this bug here. I've applied the workaround from Val (vk1266) listed on 2020-05-28 and that works, problem is not visible anymore.

Can I be of assistance in anyway? Have an environment where I can repro the issue consistently.

I found a workaround for this: to define the KRB5CCNAME environment variable at /etc/environment.d/91kerberos.conf

In my case, I store the cache file at ~/kerberos, so I set the content of that file to:

    KRB5CCNAME=${HOME}/kerberos/krb5cc_${LOGNAME}

So, if my username is "username", this results in the environment variable set to

    /home/username/kerberos/krb5cc_username

After doing this, the tickets are preserved between reboots.

Can anybody test this to ensure that it fixes the problem, please?

If you try my line, be sure to create the folder ~/kerberos before, so maybe a better alternative would be the line

    KRB5CCNAME=${HOME}/.config/krb5cc_${LOGNAME}

I found something odd... Setting KRB5CCNAME in /etc/environment does work, but setting "default_ccache_name" in /etc/krb5.conf doesn't. In theory, when KRB5CCNAME isn't set, kerberos should use that value for the cache file. And although the command line tools do use it, it seems that gvfsd doesn't...

Hi Sergio,

I tried your workaround, but the KRB5CCNAME environment variable is not set, because I don't use krb5-user and libpam-krb5. In my case the authentication is made by sssd-krb5 and the kerberos ticket is stored in /tmp/krb5cc_...

So, I was investigating this issue for a while and after some debugging of journalctl --user and dbus, it's possible to see that the gvfs-daemon.service was being started too early due to another tracker: "tracker-extract-3.service", which has WantedBy=default.target. This default value of default.target is graphical.target, and that is also too early for gvfsd to be able to get the correct environment.
So, after disabling tracker-extract-3.service, changing its Wantedby to gnome-session.target and then reenabling it, the gvfsd service is now started with the right environment.

This bug was fixed in the package tracker-miners - 3.4.3-1ubuntu1

---------------
tracker-miners (3.4.3-1ubuntu1) lunar; urgency=medium

  [ Denison Barbosa ]
  * debian/patches/ubuntu-fix-tracker-extract-start-order.patch:
    Fix tracker-extract.service WantedBy target so that gvfsd has
    access to KRB5CCNAME. (LP: #1779890)

 -- Didier Roche <email address hidden> Tue, 21 Mar 2023 15:04:35 +0100

I never had tracker-extract or tracker-miner-fs installed, as also reported in comment #9.

Hey Renbag! I was looking at the process lists that you sent and you can see that you have the following line in the "ps auxf" output of the slow machines:
"aduser 1305 1.1 0.6 633728 26316 ? SNsl 12:53 0:00 \_ /usr/libexec/tracker-miner-fs-3"
This means that the tracker-miner service is likely installed on the machine. The tracker-extract service is a bit tricky to see, since it exits after it has done its job.

Could you re-check if you have the "tracker-miner-fs" and "tracker-extract" packages installed?
Also, if you could share the output from "journalctl --user -b", it would help to see the order in which the services were started in the system.

Ah OK, you are right Deninson. I forgot that the slow machine was a test box with a standard ubuntu installation, while my workstation is customized with a different desktop and without tracker-miners. I dot have anymore the slow machine, but on my workstation gvfs seems to be working well without workarounds. Thank you for your work on this bug!

Here is the debdiff to fix the issue on Jammy.

Here is the debdiff to fix the issue on Kinetic

The attachment "Debdiff for tracker-miners fix on Jammy" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Attached is a debdiff for Focal.

Attached is a v2 debdiff for Jammy. Very minor changes, corrected version, tidied up dep3 tags.

Attached is a v2 debdiff for kinetic. Again, minor changes, corrected version, dep3 tags.

Attached is a v2 debdiff for Focal, after Denison correctly pointed out that I was missing a patch.

Status changed to 'Confirmed' because the bug affects multiple users.

Hi folks,

thank you for the hard work on this bug! I've tested out the debdiff for Jammy, and unfortunately it seems we're still missing some pieces. A fresh install with the fixed version looks ok, but when upgrading from the previous version, I can still see the old link under default.target.wants/:

root@halves-jammy-tm:~# dpkg -l | grep tracker-ext
ii tracker-extract 3.3.0-1ubuntu1 amd64 metadata database, indexer and search tool - metadata extractors

root@halves-jammy-tm:~# ls -l /etc/systemd/user/default.target.wants/
total 0
lrwxrwxrwx 1 root root 47 May 1 14:32 tracker-extract-3.service -> /usr/lib/systemd/user/tracker-extract-3.service

We should include a proper cleanup step for the upgrade path for all debdiffs, as the other releases will probably present the same issue.

Attached is a V3 debdiff for Focal that fixes this issue

Attached is a V3 debdiff for Jammy that fixes this issue

Attached is a V3 debdiff for Kinetic that fixes this issue

Attached is a V3 debdiff for Lunar that fixes this issue

Attached is a V3 debdiff for Mantic that fixes this issue

This is in the general sponsorship queue, do you need help to get those uploads sponsored? Or is there someone with upload rights working with you on this? In case you already have someone to sponsor the uploads, please, unsubscribe ~ubuntu-sponsors.

After a quick look at the latest debdiff proposed by Matthew, I would suggest to use version 3.3.0-1ubuntu0.1 in Jammy. Right now, we have version 3.3.0-1 in the release pocket, and using version 3.3.0-1ubuntu1 might make some users think that this delta was introduced during the development cycle, and using 3.3.0-1ubuntu0.1 makes more explicit the fact that this was introduced after the release via a SRU. This is just my 2 cents, what was proposed would work out well, it is more a semantic thing that I like to follow when adding a delta via SRU to a package that used to be a sync from Debian.

Hi Lucas,

Thank you for taking a look at this! I was originally sponsoring this one for Matthew, but given it's now targeting a development release, I can't sponsor that one (as I'm currently an SRU developer). If a coredev could do the sponsoring for devel, I'd be happy to work with Matthew on the other series if necessary.

Attached is a v4 for Jammy that updates the version to ubuntu0.1 in both d/changelog and d/tracker-extract.postinst

Thanks for the clarification Matthew and Heitor - first I'll re-set the state of the devel release then.
As it isn't completely fixed there.

I can see why you do the purge forgetting the old state to enable them as if they were freshly installed.
I'm not sure how SRUable that will be afterwards as it is affecting behavior if someone has set something else than the default. But for now, going forward to mantic I agree that it will help to get an upgrading system out of the old wrong dependencies.

Given that this isn't meant to be enabled automatically (thanks for the explanation in the last patch) that should be ok'ish.

You do this change even to people which installed 3.4.3-1ubuntu2 as the first (which was correct). But I think adding more code to detect that rare case won't help many real cases.

Since this is juts for mantic for now, that is fine.
But double think about it on SRU time.

This is now the third revision of the change (3.4.3-1ubuntu1 + 3.4.3-1ubuntu2 + this) and by identifying that this also needs to go to -devel I wonder if you should not also use the chance to submit the outstanding Delta to Debian. That will help them directly and mid-term ease maintenance.
3.4.3-1ubuntu2 said "Cherry-pick patch ..." but didn't mention from where - will this be in next upstream and that way come to Debian?
If so, you might still help by reminding them to use the purge mechanism to get rid of the old on-disk config.

Some things just do not fit together.
You write
"... to update from default.target.wants to gnome-session.target.wants. ..."
But while 3.4.3-1ubuntu1 does:
+ [Install]
+-WantedBy=default.target
++WantedBy=gnome-session.target

Then later 3.4.3-1ubuntu2 does
+-[Install]
+-WantedBy=default.target
++# This is a helper service that is controlled by "tracker-miner-fs-3.service".
++# It's not supposed to be enabled or started manually.

So the changelog and intent doesn't match.
Is it just the changelog that got confused or is there more that needs to be clarified and corrected?

Then the most recent applied patch says:
...
Since this is a helper service that is meant to be controlled by
"tracker-miner-fs", the install section shouldn't exist, as it allows
the service to enabled, meaning that its execution would be
controlled by systemd.

But I can't see where/how:

root@m:~# grep -Hn -- tracker-extract $(dpkg -L tracker-miner-fs| xargs) 2>/dev/null | grep -v copyright

Would you mind clarifying that interaction on your update.
Because without I'm afraid more will be confused :-/

If it would be an MP I'd set it to needs-information, but this is a debdiff in a bug.
I hope my questions and confusion was clear so that you can iterate and clarify.

I'll remove ubuntu-sponsors which I'll ask you to add back once this is ready for review again.

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

This bug was fixed in the package tracker-miners - 3.4.3-2

---------------
tracker-miners (3.4.3-2) unstable; urgency=medium

  [ Jeremy Bícha ]
  * debian/gbp.conf: Branch for trixie
  * Cherry-pick 2 patches to improve tracker systemd services
    - Fix tracker-extract.service WantedBy target so that gvfsd has
      access to KRB5CCNAME. Thanks Denison Barbosa (LP: #1779890)
    - Don't start the daemons for non-root system users like gdm
      and lightdm (LP: #2007755) (Closes: #981089)

  [ Daniel van Vugt ]
  * Add tracker-extract-Add-missing-break-to-remove-a-single.patch
    to fix SIGSEGV in decorator_item_cache_remove (LP: #2012638)

 -- Jeremy Bícha <email address hidden> Mon, 21 Aug 2023 08:32:05 -0400

Status changed to 'Confirmed' because the bug affects multiple users.