computer freezes when sshfs blocks waiting for connection

I have had similar experiences. I suspect the culprit is the nm-applet and/or NetworkManager itself, which goes into a thread-lock.

I reported this so long ago, and yet nothing has been done about it. The issue appears only when repeatedly falling of a wireless network and reconnecting to it.

And not always can one just kill nautilus. Sometimes hard reboot is necessary.

I also have this problem (and a crappy ISP). Most of the times I just go to console mode, unmount the drive (fusermount -u -z /media/name_of_drive) and then remount when connection is available again. However, if I have one nautilus window open the whole system freezes.

This happens to me on Edgy, Hardy, as well as Linux Mint Elysa and Dariana.

Happens to me on Hardy.

steps to recreate:
1. mount a samba share (smbmount)
2. open location in nautilus, then close window. (not even really necessary)
3. pull out the LAN wire (simulate network failure)
4. goto Places>Home Folder
window doesn't open up; sometimes the panels also freeze; sometimes need to reboot.

frequency: happens always for the above steps.
severity: I'd say CRITICAL since it freezes the machine, and costs all your work.. user loses confidence in stability.

Any update on this?

I jaunty I get the same behavior.

It frequently happens for me if I :

1) Mount sshfs filesystems
2) cd to them
3) turn on a VPN

...
then I have other sshfs mounts that I can connect to after the VPN is connected. They seem okay, but the other mount points completely freeze up. Also, any subsequent bash sessions (in or out of the affected filesystem) are frozen.

Eventually, I am only able to reboot/restart X to deal with it.

I get something similar:
1.) Mount sshfs filesystems from Virtual Machine
2.) Turn off the virutal machine - forget to unmount first
3.) Nautilus crashes when I start it

I get this same behaviour when I mount a samba share and then the samba shared computer gets turned off. Nautilus becomes completely unresponsive. Any windows that were open are frozen and any new windows will not open until the samba server is turned back on.

 Thank you for taking the time to report this bug and helping to make Ubuntu better. Please try to obtain a Strace following the instructions at https://wiki.ubuntu.com/DebuggingProcedures and upload the stacktrace (as an attachment) to the bug report. This will greatly help us in tracking down your problem.

I have this problem in Ubuntu 10.4...

1. sshfs mount a directory: $ sudo sshfs example.com example
2. open up nautilus, gedit, system monitor, etc...
3. network failure
4. kill gedit... nautilus and system monitor can't be killed, and need a hard reboot to fix

I will attempt to post a stack trace... (this is my first time commenting on a bug, so this is new to me)

-smlefo

@smlefo: You can not mount sshfs using the method you have given. I would suggest taking a look at the documentation. SSHFS is designed to be used by the user, not by root. Any mount using 'sudo' is not being mounted by the user and will fail.

I think this is combination of 2 bugs:

 - Nautilus freezes because of network failure
 - Panels are frozen because of nautilus hangs

Really annoying and happens often when network connection is not well.

@Charlie - oops, yeah, I mount as a user... ignore the sudo in the first command. I have to sudo to umount... not to mount.

-smlefo

I'm usually connected to one of my web servers via sshfs+fuse and I've been having this exact issue with Nautilus ever since I started using Ubuntu. Then 6.06, now 10.04.

My internet provider in their infinite wisdom insists on disconnecting their customers once every 24 hours. Of course the router is set to reconnect immediately, but at that point Nautilus has already gone kaputt.

After half an hour or so, it starts responding again and I can reconnect to the server drive (only Nautilus freezes, not the whole machine, which means I usually can continue working locally). Sometimes I'm not so patient and simply reboot.

Exact error description has already been given by wolfie2x, Robert and smlefo. And I echo Robert's sentiments regarding severity, I'd consider this to be a critical bug as well, as it seems it's not limited to sshfs but also to samba and possibly others.

You can't mark this bug as "incomplete" because someone can't get a stack trace from a system which is frozen. Also, strace doesn't generate stack traces. It generates system call traces, and any intelligent developer can tell you that everything's hung in read().

This bug is presumably related to the fact that missing NFS shares also hang large parts of the gnome environment (e.g. all keybindings are queued instead of executed) and inhibit suspend.

This is a serious issue, and since it causes a freezing machine, should be treated as such, not incomplete/expired.

@Shevek:
This is being treated as a serious issue. Unfortunately, from the supplied information, no one seems to be able to resolve the issue. Are you able to give a good resolution to this? If so, please help us with it.

Perhaps we can obtain some further information from ssh-fuse. Logs with debugging output can be useful for diagnosing the problem. Try running sshfs with the following options:

sshfs -odebug,sshfs_debug,loglevel=debug ...

and attach the plain text log files. Thanks for helping.

I experience exactly the same behavior in Ubuntu 9.10 (Karmic Koala) and in Ubuntu 10.04 (Lucid Lynx). Both client system use 64-bit versions of Ubuntu. I can also confirm that kill command doesn't terminate Nautilus process (even with SIGKILL).

A portion of syslog, related to this bug.

---

Steps to reproduce:
1) Mount a remote file system. The command I used was "sshfs User@Server:/ /media/Server -o port=10000,allow_other,default_permissions".
2) Open the remote file system in Nautilus. Open a text file from the remote file system in Gedit.
3) Shut down the ssh server.
4) Try do something with the remote file system in Nautilus (e.g. to open a file). Try to do something with a file opened in Gedit (e.g. save it).
5) Both Nautilus and Gedit will hang. (Force Quit may or may not close the frozen window but the process keeps running in any case.)

Thanks for that information and attachment. Now could you please try starting sshfs with the following and attach the sshfs log also?

sshfs User@Server:/ /media/Server -o debug,sshfs_debug,loglevel=debug port=10000,allow_other,default_permissions

Alright. I redirected log to a file with "> ~/sshfs.log 2>&1". Is this ok?

Full log is attached, but as far as I can tell, there wasn't many messages after connection was lost:

---

unique: 2986, opcode: GETATTR (3), nodeid: 1, insize: 56
getattr /
[00572] LSTAT
unique: 2987, opcode: GETATTR (3), nodeid: 1, insize: 56
getattr /
[00573] LSTAT
unique: 2988, opcode: INTERRUPT (36), nodeid: 0, insize: 48
INTERRUPT: 2987
unique: 2989, opcode: INTERRUPT (36), nodeid: 0, insize: 48
Timeout, server not responding.
remote host has disconnected

---

However sshfs process keeps running after that.

While the deeper issues are being worked (thanks folks!) I wonder if it would be a good idea to automatically unmount remotely mounted filesystems when the network goes down, and then remount when it comes up. If so, best practices for doing so would be welcome.

Providing a hint here on how to run network-manager from the command line might help. I notice nmcli and cnetworkmanager mentioned in web pages, but they may not be in ubuntu yet?

FWIW, on my netbook, after a visit to a different wifi network, a few things got hung: emacs (after I tried to open a file on a remote filesystem), evince (no idea why - won't even restart - something in the history perhaps??) and lsof (even with the -b option - huh??)

 Thanks for reporting this bug and any supporting documentation. Since this bug has enough information provided for a developer to begin work, I'm going to mark it as confirmed and let them handle it from here. Thanks for taking the time to make Ubuntu better!

My problem with evince (Comment #21) was indeed caused by it trying to get information on previously opened files, which it found in its cache at ~/.local/share/gvfs-metadata/home. That problem cleared up when I used the unmounting method recommended early on in this report:

 fusermount -u -z ...name_of_mountpoint...

and then did a `killall evince evinced` and restarted it. Previously I had used umount -l ...mountpoint....

My original emacs process remained hung though - not sure how to get it to respond. So I had to kill and restart it also.

@Neal McBurnett & any others concerned: I attached a script with some information to do automatic unmounting of SSHFS shares when the network goes down (and the opposite, too) to bug #388419.

Note that with that work-around I experience bug #610048, and I still have to do manual unmounting to avoid that. I don’t know if it’s specific to my machine or if it’d happen to others, I’d like to hear from you either way if you try it.

I would like to add that this bug is excruciating. The way it freezes your terminal is very unpleasant. SSHFS is a great tool, but this bug is a killer.

I also consider this bug as really severe.

A sshfs mount in my home broke because I lacked internet connection for some minutes. Now:

- Open a terminal. ls. Terminal freezes.
- Open a terminal. cd [TAB]. Terminal freezes.
- Open ~ in Nautilus. Window appears contentless, then freezes.
- Right click -> Unmount on the sshfs mount icon on the Desktop background. Desktop background freezes.

This bug is open for almost three years.

Does anyone have an idea what to do about this?
Can this issue be addressed in sshfs or do shells, Nautilus and so on have to handle such problems?
What do these programs do if other non-sshfs shares hang?

I have two "hotfixes"/workarounds:

Once you encounter the problem:
fusermount -u -z mountpoint (directly, don't use TAB!)

echo ServerAliveInterval 15 >> ~/.ssh/config

Then after 15 seconds your system should come back to life.

Another "hotfix" that worked for me is

umount /mount/point -f

I sudo-ed it, don't know if that's necessary for all.

Also, like above, don't autocomplete path with TAB. This "fix" made all my hung apps and windows complete their task queues and work as normal.

Confirmed for me as well that these work arounds at least prevent rebooting.
Still it seems something in fuse should recognize a dropped connection and
have a means to deal with it.

John
On Nov 26, 2010 8:46 AM, "åsmundG" <email address hidden> wrote:
> Another "hotfix" that worked for me is
>
> umount /mount/point -f
>
> I sudo-ed it, don't know if that's necessary for all.
>
> Also, like above, don't autocomplete path with TAB. This "fix" made all
> my hung apps and windows complete their task queues and work as normal.
>
> --
> computer freezes when sshfs blocks waiting for connection
> https://bugs.launchpad.net/bugs/159031
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “sshfs-fuse” package in Ubuntu: Confirmed
>
> Bug description:
> This is about Gutsy. I have set-up remote drives mounted over sshfs, which
in turns runs over a WiFi connection.
>
> Every now and then, the router reboots itself (crappy ISP, irrelevant),
and the network connection naturally stops. When that happens, sshfs will
block on any access requests for the remote drive.
>
> The nasty part is that (a) Nautilus freezes (Compiz turns it gray), and
(b) sometimes this freezes the whole desktop. I've just had a case when I
could see the Network Manager dialog asking for the network password(*),
with the Nautilus window grayed out above it. I couldn't access the NM
dialog, so there was no way to re-connect and thus unfreeze Nautilus. (The
system wasn't completely frozen, though: the mouse cursor worked, and the
System Monitor applet on the panel updated itself correctly, though nothing
responded to mouse clicks. And I could switch to a text terminal.)
>
> I don't see how Nautilus could cause this system freeze, but as soon as I
killed it (from a console), everything unfroze, I could re-connect to the
network and re-mount the remote directories.
>
> To unsubscribe from this bug, go to:
>
https://bugs.launchpad.net/ubuntu/+source/sshfs-fuse/+bug/159031/+subscribe

I agree this is a severe bug, should affect quite a few. Trying to minimize the bug and learning to live with it in different ways:

I'm mounting with this command:
sshfs <email address hidden>:/users/username /mount/dir -o idmap=user -o allow_other -o reconnect

The -reconnect part is important, as it minimizes lockups, and instead a popus asks for password again, to reconnect. This happens whenever I've not used the connection like the last hour or so.

My emacs session froze when sshfs disconnected. After reading everything related to this bug and other similar bugs I have found a workaround but not a solution. My system disconnected overnight but did not freeze after logging in with these lines
sudo fusermount -uz /mnt/remote
sudo sshfs -o ServerAliveInterval=15 xx@xx: /mnt/remote
In the morning the remote server did not recieve the changes to the file although they were apparently saved. Mount returned fuse but not sshfs in its table and so these lines were used to reconnect without problems
sudo fusermount -uz /mnt/remote
sudo sshfs -o ServerAliveInterval=15 xx@xx: /mnt/remote -o nonempty
Note that ServerAliveInterval 15 is also in the config file but while it keeps a remote login alive (not overnight however) it does not effect the sshfs system.

I just wanted to add another work-around if you find yourself with a hung sshfs mount.

I had one, did a forced, lazy unmount, but a number of processes were still blocked (terminals, gvim, nautilus). The solution was to kill -9 the sshfs process (which was still running). This immediately un-froze everything.

sshfs makes any remote SSH-accessible file accessible like a regular local file, with the same ancient C file I/O API blocking functions 'fopen(...)'/'fread(...)'/'fwrite(...)' in '/usr/include/stdio.h'.

But fundamentally these remain remote files, subject to network latency and timeout issues, just like web pages, and should not be accessed with blocking functions from UI threads.

The network issues were not a consideration in the era of local storage in which these legacy apps like vi/emacs/etc were originally developed, so they still operate on an old assumption that all files are local files, and try to access them in the same thread, rather than with callbacks like modern web browsers.

So I think this is not a problem with sshfs itself, but rather an architectural issue with a lot of software still assuming that all files are local files.

The consideration of using some kind of modern approach to work with files is good until you ask to rewrite all legacy application. That is not feasible approach as ubuntu should be rewritten as well. I have the Ubuntu 10.04.2 with gnome desktop and the taskbars freeze, gedit freeze, nautilus freeze when I disconnect (and/or reconnect) to the VPN.
kill -9 all sshfs processes can not help immediately. I need to wait ~5 min for taskbars are restarted but nautilus and gedit still do not work.That means reboot. I believe that it is just a bug of sshfs implementation that is not able to report network problem as a file problem gracefully. That is a great architecture gain if the top level application need no knowledge if the file is local or network. As for me the bug has no workaround yet. The process list is in attachment.

@Marat BN
Normally I would agree, except NFS is ages old and does not freeze applications/the entire operating system when the Network File System goes away. The same applications that freeze when sshfs mounts go away do not freeze when NFS mounts go away.

Part of the spec of NFS is that the file system's availability is untrusted, perhaps sshfs should also make that part of the spec?

Also I'll add that this bug exhibits itself to this very day in the following scenarios:
1) Intermittent wireless
2) Flaky ISP
3) Remote end implements a timeout
4) Remote end has intermittent connectivity problems

Again, NFS handles all of these gracefully and does not freeze the system. However, using NFS over the internet is a pretty poor choice -- let alone the problem of getting someone to run NFS with authentication for you on the remote end.

I don't have the C skills to code a solution, however I'm willing to test fixes because this has been a serious problem for me since 6.06.

Gutsy? I have the same problem in Ubuntu 11.04. There's also an ugly workaround.

Problem:

When the remote computer mounted with sshfs loses the connection, the local computer (Ubuntu 11.04) freezes all related processes (but not the entire computer) indefinitely. That means usually Nautilus freezes first, and the Terminal Emulator freezes second when you try to tab-complete the mountpoint for the umount command line.

Workaround:

If you go to a different TTY (ctrl+alt+F1) in stead of a Terminal Emulator, tab-complete will have no problem. You can
umount --force /mount/point
It will fail, but repeat a bunch of times and it will unmount.

Switch back you your X session and all your resources are released and functional.

SSHFS 2.3 should fix this issue, from
http://fuse.git.sourceforge.net/git/gitweb.cgi?p=fuse/sshfs;a=blob_plain;f=ChangeLog :

2011-01-25 Miklos Szeredi <email address hidden>

 * Fix cleanup when ssh connection is terminated. This prevents
 sshfs hanging when the server is rebooted, for example.

Commit 6d5e12e1b6abc2f07af547478f2497e06df988b1 fixed it.

Can 2.3 be packaged for Oneiric?

It is too late to upgrade the package in Oneiric, however, perhaps we can get the new version into Debian in time to have it in the next release.

---
Ubuntu Bug Squad volunteer triager
http://wiki.ubuntu.com/BugSquad

I have filed Bug 840626 to request this be packaged for the next release.

I've upgraded to sshfs 2.3 using Lekensteyn's PPA ( https://launchpad.net/~lekensteyn/+archive/sshfs ) but my system still hangs when uploading several large files over a sshfs connection over a wi-fi, just like in the bug's description. Is it fixed for everyone else? I'm running Natty, by the way.

The hotfix by åsmundG (agarfors) in #28 worked for me on maverick - thanks! It may be the same thing as #36, which has a clearer explanation of how to run the umount command, but the "--force" option to umount isn't in the man page (still true in natty).

I haven't tried the patched sshfs yet.

Looks like a hung sshfs also breaks hibernating:

[58492.144636] PM: Marking nosave pages: 000000000009d000 - 0000000000100000
[58492.144640] PM: Marking nosave pages: 00000000da99f000 - 00000000dafff000
[58492.144661] PM: Marking nosave pages: 00000000db000000 - 0000000100000000
[58492.145172] PM: Basic memory bitmaps created
[58492.145174] PM: Syncing filesystems ... done.
[58492.156416] Freezing user space processes ...
[58512.124270] Freezing of tasks failed after 20.00 seconds (1 tasks refusing to freeze, wq_busy=0):
[58512.124315] synapse D 0000000000000000 0 15217 1623 0x00800004
[58512.124318] ffff880157397ac8 0000000000000082 ffff880157397fd8 ffff880157396000
[58512.124321] 0000000000013d00 ffff880195479a98 ffff880157397fd8 0000000000013d00
[58512.124323] ffff8801dc118000 ffff8801954796e0 ffff880184204000 ffff880184204000
[58512.124326] Call Trace:
[58512.124332] [<ffffffff8125b70d>] request_wait_answer+0xbd/0x230
[58512.124337] [<ffffffff81087fb0>] ? autoremove_wake_function+0x0/0x40
[58512.124339] [<ffffffff8125b8fd>] fuse_request_send+0x7d/0xc0
[58512.124341] [<ffffffff8125f4e3>] fuse_dentry_revalidate+0x153/0x2a0
[58512.124345] [<ffffffff8116f0b7>] do_revalidate+0x17/0x60
[58512.124347] [<ffffffff81171606>] do_lookup+0x236/0x2e0
[58512.124350] [<ffffffff8118165e>] ? vfsmount_lock_local_unlock+0x1e/0x30
[58512.124352] [<ffffffff8118165e>] ? vfsmount_lock_local_unlock+0x1e/0x30
[58512.124354] [<ffffffff81171816>] link_path_walk+0x166/0xc40
[58512.124356] [<ffffffff8108bcdc>] ? hrtimer_try_to_cancel+0x4c/0xe0
[58512.124359] [<ffffffff811725eb>] do_path_lookup+0x5b/0x160
[58512.124361] [<ffffffff81172957>] user_path_at+0x57/0xa0
[58512.124363] [<ffffffff815c326e>] ? _raw_spin_lock+0xe/0x20
[58512.124366] [<ffffffff8109abc3>] ? futex_wake+0x103/0x120
[58512.124369] [<ffffffff81169649>] vfs_fstatat+0x39/0x70
[58512.124371] [<ffffffff8109cebb>] ? do_futex+0xbb/0x210
[58512.124374] [<ffffffff81055e11>] ? finish_task_switch+0x41/0xe0
[58512.124376] [<ffffffff8116969e>] vfs_lstat+0x1e/0x20
[58512.124378] [<ffffffff8116993a>] sys_newlstat+0x1a/0x40
[58512.124381] [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b
[58512.124400]
[58512.124401] Restarting tasks ... done.
[58512.125483] PM: Basic memory bitmaps freed
[58512.125492] video LNXVIDEO:00: Restoring backlight state

I think this bug is not in sshfs, I just installed the latest version (2.4) and the problem persists.

Neither does it seem to be in fuse.
Again, I installed the latest version (2.9.0) which also required linux-utils to be updated (installed version 2.21.2).
After installing both I recompiled/installed sshfs 2.4.
No success.

I believe the problem with this is that the connection is reset when there's no traffic which causes sshfs to hang. This is exactly what happens if you leave a ssh terminal open and walk away. Since sshfs simply uses ssh, it will behave the same way. There's many ways to workaound this.

* configure ssh with a ServerAliveInterval value. When there's no traffic to the ssh server, the ssh client will check if the server is still alive every once in a while. This will keepp traffic alive which means that whatever is in between will not terminate the connection. You will want this value to be small but not so small that it sends packagesevery second. For me, 120 (2 minutes) solves the problems but others may need to set it at 30. Add the following to /etc/ssh/ssh_config or to ~/.ssh/config file

ServerAliveInterval 120

* mount with the ServerAliveInterval option. Same as before but rather than configuring ssh, you pass this option when using sshfs. Basically, mount with the following command:

sshfs user@host:dir mountpoint -o ServerAliveInterval=120

* use the sshfs specific reconnect option (I haven't tested this option)

sshfs user@host:dir mountpoint -o reconnect

For more information see "man ssh_config" for all ssh options and "man sshfs" for the sshfs specific options.

I do believe that this is poor design of sshfs but in their opinion the problem is that the user doesn't have its ssh client configured properly.

Does this affect Ubuntu 12.04 LTS?

Yes. This bug still persists on 12.04 LTS.

>* mount with the ServerAliveInterval option.

This workaround only works if the underlying connection can be preserved during the entire session. If the server is rebooted or if the connection itself is unreliable (Wi-Fi, WAN, etc.) we're back to square one.

https://bugs.launchpad.net/ubuntu/+source/sshfs-fuse/+bug/159031

Hi all,

I've been having this problem for quite some time. Whenever I accidentally touch an sshfs mounted directory which is unavailable, I have to run "killall sshfs" to get things working again. This often happens when I mount a directory on my home network and touch that directory while at work.

Right now my workaround is to use the "ServerAliveInterval" option without the "reconnect" option. This means that when the ssh connection is lost, sshfs exits (and unmounts the locally mounted directory). I then need to run sshfs again when I want to use the mount again (I could also use autofs, I assume).

In my opinion the main problem is that sshfs leaves the local directory in an invalid state while it is "trying to reconnect". No matter what happenes, it seems clear that at any given time the mounted directory should either be mounted or unmounted -- meaning available or unavailable. If the underlying ssh connection has been lost, it makes no sense to keep the directory mounted (and locking up system calls). It's like if you pull a usb disk out of your computer. The media is unavailable. It should be unmounted, no matter what. This is more or less the behaviour I get without the reconnect option.

I would suggest that the reconnect option should simply act as an "outer loop" around a non-reconnecting version of sshfs. The non-reconnecting version should close the ssh connection and unmount the local directory on disconnect. The reconnect loop should then both connect and mount when a connection to the remote host is available again.

Obviously it is also a problem that many applications don't properly handle IO errors, and some even have IO operations in their main loops. But sshfs, fuse, and ssh can't do much to fix this.

Best,
Kevin

Xubuntu 12.04 LTS
SSHFS version 2.3
FUSE library version: 2.8.6
fusermount version: 2.8.6
using FUSE kernel interface version 7.12

The same is true of NFS and SMBFS - the whole back-end is a disaster.

Also affects 14.04 LTS. When the connection times out, I can't open a nautilus window, and any programs accessing data over SSHFS (e.g. rhythmbox in my case) freeze up.

Any updates on this issue? This bug has been around for a while...

Bug reported in 2007, still problem in 2014 on 14.04. Major usability issue.

The debian bug linked to above mentions not being able to suspend, which seems like a slightly different problem than the original bug report and not what I am experiencing.

Nautilus / files locks up, won't open if it is killed, etc. Is there anything missing from this bug report which would be needed to act on it? If nothing else, Kevin's workaround above (#49) should be set as the default configuration for Ubuntu, assuming it works. Simply trying to reconnect will not help when the remote computer is no longer available on the network.

Same here. Whenever my VPN connection drops, the complete desktop freezes.
Workaround of #36 works for me, but is pretty time consuming and annoying...

+1

still present in Ubuntu 16.04 LTS

Still present in Gentoo Linux, up-to-date system (rolling release).

+1 Xubuntu 16.04 LTS

This also happens to me, using dolphin 16.04.3, sshfs 2.5-1ubuntu1 and Ubuntu 16.04.4 LTS.

It is unfortunate that this bug couldn't be fixed yet, 10 years after it was first reported.

I attached a strace log file created by running
$ strace -Ff -tt dolphin 2>&1
and a debug file created by running
$ sshfs <serverdir> <mount point> -o password_stdin,debug,sshfs_debug,loglevel=debug,idmap=user
as requested by Charlie Kravetz on comments #8 and #16. Hopefully they can be useful.

For those who arrive here looking for workaround, I would suggesting reading comment #27, #28, #30, #31, #32, #45 and #49.

I disconnected from Wifi while an sshfs was active, and the /mnt directory was inaccessible afterwords. "ls /mnt" hung. "sudo umount /mnt/mysshfs" hung. "sudo umount -f /mnt/mysshfs" said "target was busy". "sudo umount -l /mnt/mysshfs" worked. Though that's not a nice solution because the file is still busy somewhere, but it worked for me.

Still present in 20.04.