network file systems in FSTAB no longer mount at boot with NetworkManager

Opening a task on systemd as well for this.

When called early in boot, mount.nfs may fail because of any number of problems with the network. In this case, it appears it's being called by systemd before /etc/resolv.conf has been set up, resulting in a temporary DNS failure. There is no way for mount.nfs to work when invoked at this point in the boot.

So either systemd needs to more finely control the timing of the mount call so that it definitely happens only after the network is up (including /etc/resolv.conf configuration), or it needs to handle retrying the mount in the case of a temporary failure. (The latter is what mountall did in Ubuntu 14.10 and earlier.)

Status changed to 'Confirmed' because the bug affects multiple users.

I'm seeing this behavior as well on a fresh install of Xubuntu 15.10.

http://askubuntu.com/questions/689515/mount-network-share-automatically-on-startup

This bug has not been tagged before as 'wily' and unfortunately I have created a duplicate bug report of this one, not being able to find it earlier.

A few more observations from my original bug report:

It looks like the network is not up when the mounting is done however systemd dependencies as well as unit files generated from /etc/fstab seem to be OK

$ tail -2 /etc/fstab
# Media Library
disk.lan:/volume1/Library /media/Library nfs rw,async,auto,nfsvers=3 0 0

$ systemd-analyze dot media-Library.mount
digraph systemd {
        "remote-fs.target"->"media-Library.mount" [color="green"];
        "remote-fs.target"->"media-Library.mount" [color="black"];
        "media-Library.mount"->"network-online.target" [color="green"];
        "media-Library.mount"->"remote-fs-pre.target" [color="green"];
        "media-Library.mount"->"systemd-journald.socket" [color="green"];
        "media-Library.mount"->"system.slice" [color="green"];
        "media-Library.mount"->"network.target" [color="green"];
        "media-Library.mount"->"-.mount" [color="green"];
        "media-Library.mount"->"-.mount" [color="black"];
        "media-Library.mount"->"system.slice" [color="grey66"];
        "media-Library.mount"->"network-online.target" [color="grey66"];
        "media-Library.mount"->"umount.target" [color="red"];
        "umount.target"->"media-Library.mount" [color="green"];
        "umount.target"->"media-Library.mount" [color="red"];
}
   Color legend: black = Requires
                 dark blue = Requisite
                 dark grey = Wants
                 red = Conflicts
                 green = After

Despite the correct dependency between "media-Library.mount" and the "network-online.target" looking at the journal log reveals that the network is not yet started when the "network-online.target" is reached.

$ journalctl --boot _COMM=systemd _COMM=NetworkManager
Dez 04 15:16:55 localhost systemd[1]: Starting Hostname Service...
Dez 04 15:16:55 localhost systemd[1]: Started Hostname Service.
Dez 04 15:16:55 localhost NetworkManager[693]: <info> NetworkManager (version 1.0.4) is starting...
Dez 04 15:16:55 localhost NetworkManager[693]: <info> Read config: /etc/NetworkManager/NetworkManager.conf
Dez 04 15:16:55 localhost systemd[1]: Started Network Manager.
Dez 04 15:16:55 localhost NetworkManager[693]: <info> VPN: loaded org.freedesktop.NetworkManager.pptp
Dez 04 15:16:55 localhost systemd[1]: Reached target Network.
Dez 04 15:16:55 localhost NetworkManager[693]: <info> DNS: loaded plugin dnsmasq
Dez 04 15:16:55 localhost systemd[1]: Reached target Network is Online.
Dez 04 15:16:55 localhost systemd[1]: Mounting /media/Library...
Dez 04 15:16:55 localhost systemd[1]: Starting /etc/rc.local Compatibility...
Dez 04 15:16:55 localhost NetworkManager[693]: <info> init!
Dez 04 15:16:55 localhost NetworkManager[693]: <info> update_system_hostname
Dez 04 15:16:55 localhost systemd[1]: Started /etc/rc.local Compatibility.
Dez 04 15:16:55 localhost NetworkManager[693]: <info> interface-parser: parsing file /etc/network/interfaces
Dez 04 15:16:55 localhost systemd[1]: Started Authenticate and Authorize Users to Run Privileged Tasks.
Dez 04 15:16:55 localhos...

I've been able to work around the problem by adding 'x-systemd.automount,' to the list of options on the fstab line.

@mtd0, adding that to my fstab worked as well.

Workaround seems good. Adding 'x-systemd.automount' to fstab mounts NFS shares during boot.

The x-systemd.automout option works for me as well running 15.10.

Interestingly, it was commented out in my fstab. I believe I tried that option on 15.04, and it didn't work then.

Confirming on Lubuntu 15.10

I looked at what Fedora was doing. The only difference seems to be network-online waits for Networkmanager to say it's online.
sudo systemctl add-wants network-online.target NetworkManager-wait-online.service

This fixes the issue in my testing. Unfortunately it also makes NetworkManager-wait-online.service the biggest blame for bootup time. Any other ideas?

To be clear, Fedora does not have this problem in 23 or Rawhide.

This is not an issue in Debian stretch. Same systemd version, newer network manager version.
Debian does not have anything defined (on fs) in network-online.target. I tried removing all of the network-online.target.wants but that hasn't helped (the only meaningful difference I've found so far)

That's not quite correct. The mount initially fails but then it appears to try again. In any case it does appear end up mounted in Debian. This also fails on an install without Network-manager on Ubuntu. Will retry over many restarts on Debian just to confirm.

Currently, my testing has not found a specific package that is the cause (ignore network-manager comment above)
Start with Ubuntu-desktop - it won't mount reliably. Remove packages like unity, still fails...
Start with Ubuntu-server - it works every time. Install ubuntu-desktop meta package, it still works every time!

Oops - It is related to network-manager.. or more precisely it works fine if you specify to use dhcp in /etc/network/interfaces. If you leave it blank (and network-manager actually does the work) then you have the failures.

So is that a regression from bug 1430280, perhaps this got inadvertently dropped in a merge?

It's disabled explicitly in the rules.
Also in network-manager.postinst it seems that causes it to be disabled as well in my test.

Unfortunately my patch only makes it enabled on a fresh install of network-manager (or after a purge). It (the upgrade) tries to make have the services respect if the user changed them starting. Do we care for network-manager's services? If you don't want them you should have the package uninstalled?

The attachment "network-manager_1.0.4-0ubuntu8.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

This should do it properly now on upgrade.. Just readd the blurb that says it enables it in postinst that actually disabled it..

Hey Bryan,

thanks for investigating and the patch! Indeed it seems the enabling in debian/rules got reverted in some merge. The patch is good, except that I changed postinst to

- if dpkg --compare-versions "$2" lt-nl "0.9.10.0-4ubuntu10"; then
+ if dpkg --compare-versions "$2" lt-nl "1.0.4-0ubuntu8"; then

i. e. I just bumped the number, but I kept the "purge". We want to reset deb-systemd-helper's brain about the automatic/manual enablement state of NetworkManager-wait-online.service, the actual "enable" call will be automatically generated by dh_installinit/dh_systemd_enable. Sorry, that's a bit complicated..

I committed this updated patch to bzr and uploaded.

This bug was fixed in the package network-manager - 1.0.4-0ubuntu8

---------------
network-manager (1.0.4-0ubuntu8) xenial; urgency=medium

  * Fix NetworkManager-wait-online.service so it is enabled so NFS
    and other remote FS work with name resolution. (LP: #1515446)

 -- Bryan Quigley <email address hidden> Thu, 28 Jan 2016 10:45:31 +0100

Thanks for uploading to dev!

Here is a debdiff for an SRU to wily

Debdiff from comment #24 is for wily and should fix this.

Hello lnxusr, or anyone else affected,

Accepted network-manager into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.0.4-0ubuntu5.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Thanks Brian. I just installed from the proposed repository, removed the '.x-systemd.automount' from fstab and rebooted. All seems to be working well. I may compile and package it up for local install this weekend, I don't want the other proposed packages always in my upgradable listing.

Either way, seems to be working great. Thank you.

I can confirm that the bug is fixed in network-manager 1.0.4-0ubuntu5.3. Thanks!

This bug was fixed in the package network-manager - 1.0.4-0ubuntu5.3

---------------
network-manager (1.0.4-0ubuntu5.3) wily; urgency=medium

  * Fix NetworkManager-wait-online.service so it is enabled so NFS
    and other remote FS work with name resolution. (LP: #1515446)

 -- Bryan Quigley <email address hidden> Thu, 03 Mar 2016 16:50:46 +0100

The verification of the Stable Release Update for network-manager has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug is back in 17.04. Network (NFS4) shares are no longer automatically mounted on bootup.

I can confirm that. I updated 16.04 (works) -> 16.10 (works) -> 17.04 (doesn't work any more).
calling "mount -a" mounts the directories without problems.

As workaround you can put "mount -a" into /etc/rc.local just before the "exit 0". This at least works here to get my shares. How can I help to solve this?

I see the problem here as well in 17.04.

The solution with x-systemd.automount mount option in fstab seems to help.

mount -a doesn't work in my case (17.04) but x-systemd.automount helps