MAAS

Potential DB threadpool exhaustion issue in bootresources.py

Bug #1950648 reported by Jerzy Husakowski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Triaged
Medium
Jacopo Rota
MAAS 3.4.x
3.3
Triaged
Medium
Unassigned
MAAS 3.3.x

Bug Description

deferToDatabase() is meant to be used to run all DB-accessing code, and there's a threadpool (9 threads), each with own DB connection, to allow parallel queries. The code that is "deferredToDatabase" shouldn't call deferToDatabase() again, as it will grab another thread from the pool, and can deadlock in the worst case if there are no free threads left.

The following call sequence contains a deferToDatabase() call inside another deferToDatabase() call:
https://git.launchpad.net/maas/tree/src/maasserver/bootresources.py#n1455 calls _import_resources_in_thread(), which
https://git.launchpad.net/maas/tree/src/maasserver/bootresources.py#n1430 defers to database _import_resources(), which
https://git.launchpad.net/maas/tree/src/maasserver/bootresources.py#n1294 defers to database again.

Alberto Donato (ack)
Changed in maas:
status: New → Confirmed
status: Confirmed → Triaged
milestone: none → next
importance: Undecided → High
Revision history for this message
Jerzy Husakowski (jhusakowski) wrote :

Let's introduce a test case to confirm whether the assumption that this will lead to a deadlock is correct, and re-triage this issue accordingly.

Changed in maas:
importance: High → Medium
milestone: next → 3.3.0
Adam Collard (adam-collard)
Changed in maas:
milestone: 3.3.0 → 3.4.0
Jacopo Rota (r00ta)
Changed in maas:
assignee: nobody → Jacopo Rota (r00ta)
Revision history for this message
Jacopo Rota (r00ta) wrote :

I confirm this is a bug. Some ways to reproduce:

1) change `max_threads_for_database_pool = 1` in /src/maasserver/utils/threads.py
2) run `sudo maas init region+rack --database-uri maas-test-db:///`
3) createadmin
4) login

The import image page does not work

You could also
1) change `max_threads_for_database_pool = 1` in /src/maasserver/utils/threads.py
2) add some log debug statements in the deferred functions
3) login and make a POST call to `/MAAS/api/2.0/boot-resources/op-import` to trigger the deadlock

check the logs: the second deferred function is never called.

Alberto Donato (ack)
Changed in maas:
milestone: 3.4.0 → 3.4.x
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.