module level init of db transation contexts cause failure under mod_wsgi on module reload
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
sean mooney | ||
Rocky |
Fix Committed
|
Medium
|
Lee Yarwood |
Bug Description
Description
===========
This is related to a downstream bug first reported here:https:/
triplo currently deploys the placemetn api under mod_wsgi
when deployed under mod_wsgi if the wsgi application exits with an error it is reloaded
back into the same python interperter instance. As a result of this behavior module level
variable have a longer lifetime then the lifetime of the application. when run under uwsgi
when the application is reloaded it is reloaded in a new python interperter meaning
the lifetime of the module level variables is scoped to the life time of the application.
As a result of the life time semantics of mod_wsgi the placment api and nova-api
must assume that the wsgi applications init can be invoked multiple times on failure.
The current use of the sqlalcamey enginefacade transaction_context is not reentrant
resulting in a type error being raised on subsequet calls to configure when the wsgi
application is reloaded.
Expected result
===============
it should be possible to reload the nova and placement api wsgi application
under mod_wsgi on failure
Actual result
=============
46087 [Wed Oct 10 15:10:49.433284 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Target WSGI script '/var/www/
46088 [Wed Oct 10 15:10:49.433305 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Exception occurred processing WSGI script '/var/www/
46089 [Wed Oct 10 15:10:49.433320 2018] [:error] [pid 14] [remote 172.25.0.10:208] Traceback (most recent call last):
46090 [Wed Oct 10 15:10:49.433333 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/var/www/
46091 [Wed Oct 10 15:10:49.433354 2018] [:error] [pid 14] [remote 172.25.0.10:208] application = init_application()
46092 [Wed Oct 10 15:10:49.433361 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/
46093 [Wed Oct 10 15:10:49.433386 2018] [:error] [pid 14] [remote 172.25.0.10:208] db_api.
46094 [Wed Oct 10 15:10:49.433392 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/
46095 [Wed Oct 10 15:10:49.433403 2018] [:error] [pid 14] [remote 172.25.0.10:208] **_get_
46096 [Wed Oct 10 15:10:49.433408 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/
46097 [Wed Oct 10 15:10:49.433420 2018] [:error] [pid 14] [remote 172.25.0.10:208] self._factory.
46098 [Wed Oct 10 15:10:49.433425 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/
46099 [Wed Oct 10 15:10:49.433435 2018] [:error] [pid 14] [remote 172.25.0.10:208] return wrapped(*args, **kwargs)
46100 [Wed Oct 10 15:10:49.433440 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/
46101 [Wed Oct 10 15:10:49.433449 2018] [:error] [pid 14] [remote 172.25.0.10:208] self._configure
46102 [Wed Oct 10 15:10:49.433453 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/
46103 [Wed Oct 10 15:10:49.433462 2018] [:error] [pid 14] [remote 172.25.0.10:208] raise TypeError("this TransactionFactory is already started")
46104 [Wed Oct 10 15:10:49.433473 2018] [:error] [pid 14] [remote 172.25.0.10:208] TypeError: this TransactionFactory is already started
Changed in nova: | |
assignee: | nobody → sean mooney (sean-k-mooney) |
status: | New → In Progress |
importance: | Undecided → Medium |
tags: | added: api placement |
summary: |
- module level init of db transation contexts cause failure under - mod_uwsgi on module reload + module level init of db transation contexts cause failure under mod_wsgi + on module reload |
Reviewed: https:/ /review. openstack. org/610034 /git.openstack. org/cgit/ openstack/ nova/commit/ ?id=00d08a3288f 61268f8a823991f adf0d83c36c04d
Committed: https:/
Submitter: Zuul
Branch: master
commit 00d08a3288f6126 8f8a823991fadf0 d83c36c04d
Author: Sean Mooney <email address hidden>
Date: Fri Oct 12 14:26:14 2018 +0100
Harden placement init under wsgi
- This change tries to address an edge case discovered
when running placement under mod_wsgi where
if the placement wsgi application is re-initialize the db_api
configure method attempts to reconfigure a started transaction
factory.
- This since oslo.db transaction factories do not support ration at runtime this result in an exception being
reconfigu
raised preventing reloading of the Placement API without
restarting apache to force mod_wsgi to recreate the
python interpreter.
- This change introduces a run once decorator to allow annotating
functions that should only be executed once for the lifetime fo
an interpreter.
- This change applies the run_once decorator to the db_api configure onFactory on application reload.
method, to suppress the attempt to reconfigure the current
Transacti
Co-Authored-By: Balazs Gibizer <email address hidden> 124e713ac31111a 8ea6fa2f1ba
Closes-Bug: #1799246
Related-Bug: #1784155
Change-Id: I704196711d30c1