FoxtrotGPS

g_thread_new() fails with "creating thread 'gps thread': Error creating thread: Resource temporarily unavailable".

Bug #1876744 reported by Russell Grokett on 2020-05-04

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	FoxtrotGPS	Fix Committed	Undecided	Unassigned

Bug Description

I ran into a bug causing foxtrotgps to crash due to out-of-threads error on Debian Buster when I happened to disconnect the GPS and let it run overnight (it can take hours before failure!). I see numerous GLib-CRITICAL messages concerning the gps_functions.c connecting to gpsd (not unexpected!) but then eventually, a GLib-ERROR g_thread_new() fails with "creating thread 'gps thread': Error creating thread: Resource temporarily unavailable".

This appears only in v1.2.2 using the new GLIB g_thread_new().

When I reverted to v1.2.1 with g_thread_create() I do not see this issue.

This appears to affect several (all?) threads being used as I was able to recreate with other threaded operations, like map tile download thread.

My best guess is that g_thread_exit() or g_thread_unref() isn't being called when the thread is no longer needed and every call to g_thread_new() is creating yet another thread.

Strangely, the GLIB docs say that the thread should free when the function returns, but that doesn't appear to me to be the case. (Maybe they mean it is free to be reused, but is not released and unrefed?)

I haven't worked with GLIB before, but I assume that wherever g_free() is being called, it may need g_thread_exit() or g_thread_unref()?

Reading GLIB info from:
https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-new

----- EXAMPLE ERROR MESSAGE -------

(foxtrotgps:18515): GLib-CRITICAL **: 11:22:28.267: Source ID 29030 was not found when attempting to remove it
connection to gpsd SUCCEEDED

(foxtrotgps:18515): GLib-CRITICAL **: 11:23:28.355: Source ID 29157 was not found when attempting to remove it

(foxtrotgps:18515): GLib-ERROR **: 11:23:28.355: creating thread 'gps thread': Error creating thread: Resource temporarily unavailable
Trace/breakpoint trap

Another...

(foxtrotgps:10300): GLib-ERROR **: 10:22:46.899: creating thread 'download thread': Error creating thread: Resource temporarily unavailable
Trace/breakpoint trap

gps_functions.c:
#if GLIB_CHECK_VERSION(2,34,0)
g_thread_new("gps thread", &get_gps_thread, (gpointer)NULL);
#else
g_thread_create(&get_gps_thread, NULL, FALSE, NULL);
#endif
}

Currently, I am testing the following change:

g_thread_unref(g_thread_new("gps thread", &get_gps_thread, (gpointer)NULL));

But I don't know if this is an appropriate fix.

Ideas? Thanks greatly! And Thanks for a great program... for Raspberry Pi's. :-)

Related branches

lp://staging/foxtrotgps

Revision history for this message

Paul Wise (Debian) (pabs) wrote on 2020-05-05: Re: [Bug 1876744] [NEW] g_thread_new() fails with "creating thread 'gps thread': Error creating thread: Resource temporarily unavailable".

I can't reproduce this on the x86 hardware that I have.

It sounds like you are running FoxtrotGPS on a Raspberry Pi?

Which operating system are you running?

Do you know what the thread limits are on your OS? Run this:

$ cat /proc/sys/kernel/threads-max
95400

As a workaround you can also increase the amount of threads:

sudo sh -c 'echo 1000000 > /proc/sys/kernel/threads-max'

Are there any particular circumstances where it happens? Like just
after downloading a lot of tiles or something like that?

Please try to figure out which of the threads is being created but not
destroyed. I assume it isn't the gps thread, that is probably just the
final one that causes the limit to be reached. On my system I see this:

   $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status
   /proc/457081/task/457081/status:Name: foxtrotgps
   /proc/457081/task/457082/status:Name: gmain
   /proc/457081/task/457083/status:Name: dconf worker
   /proc/457081/task/457084/status:Name: gdbus
   /proc/457081/task/466976/status:Name: download thread
   /proc/457081/task/466977/status:Name: download thread
   /proc/457081/task/466978/status:Name: download thread
   /proc/457081/task/466982/status:Name: download thread
   /proc/457081/task/466984/status:Name: download thread
   /proc/457081/task/466985/status:Name: download thread

If you are able to bisect this issue to find out which commit causes
it, that would help in resolving the problem. I'm assuming it was
caused by the commit changing to the new GLib thread APIs.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Revision history for this message

Russell Grokett (rgrokett) wrote on 2020-05-05:

Download full text (5.4 KiB)

Thanks for the super fast response back!
I suspect the high thread-max you have is helping.

$ cat /proc/sys/kernel/threads-max15137
OS is Raspbian Buster on RPi 4.

pi@raspix86TEST:~/foxtrotgps-1.2.2 $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status/proc/31606/task/31606/status:Name: foxtrotgps/proc/31606/task/31607/status:Name: gmain/proc/31606/task/31608/status:Name: gdbus/proc/31606/task/31609/status:Name: dconf worker
Using the g_thread_unref() below seems to have eliminated the issue as I have not been able to reproduce since adding. I will try backing out the change and see if it reoccurs again and capture the threads.
g_thread_unref(g_thread_new("gps thread", &get_gps_thread,
(gpointer)NULL));

g_thread_unref(g_thread_new("download thread", &dl_thread, (gpointer *)tile_data));

Thanks!Russell

On Monday, May 4, 2020, 08:50:36 PM EDT, Paul Wise (Debian) <email address hidden> wrote:

I can't reproduce this on the x86 hardware that I have.

It sounds like you are running FoxtrotGPS on a Raspberry Pi?

Which operating system are you running?

Do you know what the thread limits are on your OS? Run this:

$ cat /proc/sys/kernel/threads-max
95400

As a workaround you can also increase the amount of threads:

sudo sh -c 'echo 1000000 > /proc/sys/kernel/threads-max'

Are there any particular circumstances where it happens? Like just
after downloading a lot of tiles or something like that?

  $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status
  /proc/457081/task/457081/status:Name:    foxtrotgps
  /proc/457081/task/457082/status:Name:    gmain
  /proc/457081/task/457083/status:Name:    dconf worker
  /proc/457081/task/457084/status:Name:    gdbus
  /proc/457081/task/466976/status:Name:    download thread
  /proc/457081/task/466977/status:Name:    download thread
  /proc/457081/task/466978/status:Name:    download thread
  /proc/457081/task/466982/status:Name:    download thread
  /proc/457081/task/466984/status:Name:    download thread
  /proc/457081/task/466985/status:Name:    download thread

If you are able to bisect this issue to find out which commit causes
it, that would help in resolving the problem. I'm assuming it was
caused by the commit changing to the new GLib thread APIs.

--
bye,
pabs

https://wiki.debian.org/PaulWise

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1876744

Title:
g_thread_new() fails with "creating thread 'gps thread': Error
creating thread: Resource temporarily unavailable".

Status in FoxtrotGPS:
New

Bug description:
  I ran into a bug causing foxtrotgps to crash due to out-of-threads
  error on Debian Buster when I happened to disconnect the GPS and let
  it run overnight (it can take hours before failure!). I see numerous
  GLib-CRITICAL messages concerning the gps_functions.c connecting to
  gpsd (not unexpected!) but then eventually, a GLib-ERROR
  g_thread_new...

Thanks for the super fast response back!  
I suspect the high thread-max you have is helping.

$ cat  /proc/sys/kernel/threads-max15137
OS is Raspbian Buster on RPi 4.

pi@raspix86TEST:~/foxtrotgps-1.2.2 $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status/proc/31606/task/31606/status:Name:     foxtrotgps/proc/31606/task/31607/status:Name:     gmain/proc/31606/task/31608/status:Name:     gdbus/proc/31606/task/31609/status:Name:     dconf worker
Using the g_thread_unref() below seems to have eliminated the issue as I have not been able to reproduce since adding.   I will try backing out the change and see if it reoccurs again and capture the threads. 
g_thread_unref(g_thread_new("gps thread", &get_gps_thread,
  (gpointer)NULL));

g_thread_unref(g_thread_new("download thread", &dl_thread, (gpointer *)tile_data));

Thanks!Russell

On Monday, May 4, 2020, 08:50:36 PM EDT, Paul Wise (Debian) <1876744@bugs.launchpad.net> wrote:  
 
 I can't reproduce this on the x86 hardware that I have.

It sounds like you are running FoxtrotGPS on a Raspberry Pi?

Which operating system are you running?

Do you know what the thread limits are on your OS? Run this:

$ cat  /proc/sys/kernel/threads-max
  95400

As a workaround you can also increase the amount of threads:

sudo sh -c 'echo 1000000 > /proc/sys/kernel/threads-max'

Are there any particular circumstances where it happens? Like just
after downloading a lot of tiles or something like that?

$ grep Name /proc/$(pgrep foxtrotgps)/task/*/status
  /proc/457081/task/457081/status:Name:    foxtrotgps
  /proc/457081/task/457082/status:Name:    gmain
  /proc/457081/task/457083/status:Name:    dconf worker
  /proc/457081/task/457084/status:Name:    gdbus
  /proc/457081/task/466976/status:Name:    download thread
  /proc/457081/task/466977/status:Name:    download thread
  /proc/457081/task/466978/status:Name:    download thread
  /proc/457081/task/466982/status:Name:    download thread
  /proc/457081/task/466984/status:Name:    download thread
  /proc/457081/task/466985/status:Name:    download thread

If you are able to bisect this issue to find out which commit causes
it, that would help in resolving the problem. I'm assuming it was
caused by the commit changing to the new GLib thread APIs.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

-- 
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1876744

Title:
  g_thread_new() fails with "creating thread 'gps thread': Error
  creating thread: Resource temporarily unavailable".

Status in FoxtrotGPS:
  New

Bug description:
  I ran into a bug causing foxtrotgps to crash due to out-of-threads
  error on Debian Buster when I happened to disconnect the GPS and let
  it run overnight (it can take hours before failure!). I see numerous
  GLib-CRITICAL messages concerning the gps_functions.c connecting to
  gpsd (not unexpected!) but then eventually, a GLib-ERROR
  g_thread_new() fails with "creating thread 'gps thread': Error
  creating thread: Resource temporarily unavailable".

This appears only in v1.2.2 using the new GLIB g_thread_new().

When I reverted to v1.2.1 with g_thread_create() I do not see this
  issue.

This appears to affect several (all?) threads being used as I was able to recreate with other threaded operations, like map tile download thread.
    
  My best guess is that g_thread_exit() or g_thread_unref() isn't being called when the thread is no longer needed and  every call to g_thread_new() is creating yet another thread.

Strangely, the GLIB docs say that the thread should free when the
  function returns, but that doesn't appear to me to be the case. (Maybe
  they mean it is free to be reused, but is not released and unrefed?)

I haven't worked with GLIB before, but I assume that wherever g_free()
  is being called, it may need g_thread_exit() or g_thread_unref()?

Reading GLIB info from:
  https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-new

----- EXAMPLE ERROR MESSAGE -------

(foxtrotgps:18515): GLib-CRITICAL **: 11:22:28.267: Source ID 29030 was not found when attempting to remove it
  connection to gpsd SUCCEEDED

(foxtrotgps:18515): GLib-CRITICAL **: 11:23:28.355: Source ID 29157
  was not found when attempting to remove it

(foxtrotgps:18515): GLib-ERROR **: 11:23:28.355: creating thread 'gps thread': Error creating thread: Resource temporarily unavailable
  Trace/breakpoint trap

Another...

(foxtrotgps:10300): GLib-ERROR **: 10:22:46.899: creating thread 'download thread': Error creating thread: Resource temporarily unavailable
  Trace/breakpoint trap

gps_functions.c: 
  #if GLIB_CHECK_VERSION(2,34,0)
          g_thread_new("gps thread", &get_gps_thread, (gpointer)NULL);
  #else
          g_thread_create(&get_gps_thread, NULL, FALSE, NULL);
  #endif
  }

Currently, I am testing the following change:

g_thread_unref(g_thread_new("gps thread", &get_gps_thread,
  (gpointer)NULL));

But I don't know if this is an appropriate fix.

Ideas?  Thanks greatly! And Thanks for a great program... for
  Raspberry Pi's. :-)

To manage notifications about this bug go to:
https://bugs.launchpad.net/foxtrotgps/+bug/1876744/+subscriptions

Revision history for this message

Paul Wise (Debian) (pabs) wrote on 2020-05-05:

Can you try increasing thread-max?

Your thread list doesn't look particularly long, hmm.

Looking at the GLib threads documentation it seems that you do need to
g_thread_unref if you aren't using g_thread_join indeed.

https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-new
https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-unref
https://codesearch.debian.net/search?q=g_thread_unref.*g_thread_new&literal=0

We have a number of calls to g_thread_new, so all of them will have to
get g_thread_unref added. If you would like to contribute a patch that
does that, that would be great, otherwise I can try to schedule some
time to work on it.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Revision history for this message

Jesse Gardner (jgardner7289) wrote on 2021-08-16 (last edit on 2021-08-16):

I am experiencing this bug on two x86_64 machines. On both, when downloading tiles at max depth, a crash occurs after about 15 minutes. The output before the crash is this several times (for different tiles):

TILE DL PROBLEM: getaddrinfo() thread failed to start
https://tile.openstreetmap.org/12/1279/1588.png|/home/jesse/Maps/OSM/12/1279/1588.png|/home/jesse/Maps/OSM/12/1279/

Followed by this:

(foxtrotgps:68150): GLib-ERROR **: 03:37:17.434: creating thread 'download thread': Error creating thread: Resource temporarily unavailable
Trace/breakpoint trap (core dumped)

The two machines are running Ubuntu 20.04, using foxtrotgps 1.2.2 from the official repos, and Manjaro, using foxtrotgps compiled from bzr. The thread-max for theses machines are 513321 and 62553, respectively.

On the manjaro machine, while downloading tiles, I ran this command given above every 2s until crash: grep Name /proc/$(pgrep foxtrotgps)/task/*/status

At any given time, there were 1 each of threads named foxtrotgps, gmain, dconf worker, gdbus, and a handful of download threads. About 10 seconds before crash, another (second) thread named foxtrotgps appeared. Not sure what that means, but hopefully it is helpful.

Edit: I was able to determine that this bug was introduced by commit 315. Compiling from 315 produced the same behavior, but compiling from 314 produced a build that was able to download tiles at max depth and stay running for over an hour, where previously it crashed consistently around the 15 minute mark.

Revision history for this message

Paul Wise (Debian) (pabs) wrote on 2021-08-17:

Jesse Gardner created, tested and submitted a patch that fixes this problem and I have now merged the patch into trunk.

Changed in foxtrotgps:
status:	New → Fix Committed

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.