module level init of db transation contexts cause failure under mod_wsgi on module reload

Bug #1799246 reported by sean mooney
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
sean mooney
Rocky
Fix Committed
Medium
Lee Yarwood

Bug Description

Description
===========
This is related to a downstream bug first reported here:https://bugzilla.redhat.com/show_bug.cgi?id=1630069 where in it was discovers that due to how
triplo currently deploys the placemetn api under mod_wsgi

when deployed under mod_wsgi if the wsgi application exits with an error it is reloaded
back into the same python interperter instance. As a result of this behavior module level
variable have a longer lifetime then the lifetime of the application. when run under uwsgi
when the application is reloaded it is reloaded in a new python interperter meaning
the lifetime of the module level variables is scoped to the life time of the application.

As a result of the life time semantics of mod_wsgi the placment api and nova-api
must assume that the wsgi applications init can be invoked multiple times on failure.
The current use of the sqlalcamey enginefacade transaction_context is not reentrant
resulting in a type error being raised on subsequet calls to configure when the wsgi
application is reloaded.

Expected result
===============
it should be possible to reload the nova and placement api wsgi application
under mod_wsgi on failure

Actual result
=============

46087 [Wed Oct 10 15:10:49.433284 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Target WSGI script '/var/www/cgi-bin/nova/nova-placement-api' cannot be loaded as Python module.
46088 [Wed Oct 10 15:10:49.433305 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Exception occurred processing WSGI script '/var/www/cgi-bin/nova/nova-placement-api'.
46089 [Wed Oct 10 15:10:49.433320 2018] [:error] [pid 14] [remote 172.25.0.10:208] Traceback (most recent call last):
46090 [Wed Oct 10 15:10:49.433333 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/var/www/cgi-bin/nova/nova-placement-api", line 54, in <module>
46091 [Wed Oct 10 15:10:49.433354 2018] [:error] [pid 14] [remote 172.25.0.10:208] application = init_application()
46092 [Wed Oct 10 15:10:49.433361 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi.py", line 108, in init_application
46093 [Wed Oct 10 15:10:49.433386 2018] [:error] [pid 14] [remote 172.25.0.10:208] db_api.configure(conf.CONF)
46094 [Wed Oct 10 15:10:49.433392 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/db_api.py", line 35, in configure
46095 [Wed Oct 10 15:10:49.433403 2018] [:error] [pid 14] [remote 172.25.0.10:208] **_get_db_conf(conf.placement_database))
46096 [Wed Oct 10 15:10:49.433408 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 788, in configure
46097 [Wed Oct 10 15:10:49.433420 2018] [:error] [pid 14] [remote 172.25.0.10:208] self._factory.configure(**kw)
46098 [Wed Oct 10 15:10:49.433425 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
46099 [Wed Oct 10 15:10:49.433435 2018] [:error] [pid 14] [remote 172.25.0.10:208] return wrapped(*args, **kwargs)
46100 [Wed Oct 10 15:10:49.433440 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 312, in configure
46101 [Wed Oct 10 15:10:49.433449 2018] [:error] [pid 14] [remote 172.25.0.10:208] self._configure(False, kw)
46102 [Wed Oct 10 15:10:49.433453 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 317, in _configure
46103 [Wed Oct 10 15:10:49.433462 2018] [:error] [pid 14] [remote 172.25.0.10:208] raise TypeError("this TransactionFactory is already started")
46104 [Wed Oct 10 15:10:49.433473 2018] [:error] [pid 14] [remote 172.25.0.10:208] TypeError: this TransactionFactory is already started

Tags: api placement
Changed in nova:
assignee: nobody → sean mooney (sean-k-mooney)
status: New → In Progress
importance: Undecided → Medium
tags: added: api placement
summary: - module level init of db transation contexts cause failure under
- mod_uwsgi on module reload
+ module level init of db transation contexts cause failure under mod_wsgi
+ on module reload
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/610034
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=00d08a3288f61268f8a823991fadf0d83c36c04d
Submitter: Zuul
Branch: master

commit 00d08a3288f61268f8a823991fadf0d83c36c04d
Author: Sean Mooney <email address hidden>
Date: Fri Oct 12 14:26:14 2018 +0100

    Harden placement init under wsgi

    - This change tries to address an edge case discovered
      when running placement under mod_wsgi where
      if the placement wsgi application is re-initialize the db_api
      configure method attempts to reconfigure a started transaction
      factory.

    - This since oslo.db transaction factories do not support
      reconfiguration at runtime this result in an exception being
      raised preventing reloading of the Placement API without
      restarting apache to force mod_wsgi to recreate the
      python interpreter.

    - This change introduces a run once decorator to allow annotating
      functions that should only be executed once for the lifetime fo
      an interpreter.

    - This change applies the run_once decorator to the db_api configure
      method, to suppress the attempt to reconfigure the current
      TransactionFactory on application reload.

    Co-Authored-By: Balazs Gibizer <email address hidden>
    Closes-Bug: #1799246
    Related-Bug: #1784155
    Change-Id: I704196711d30c1124e713ac31111a8ea6fa2f1ba

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/617297

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/617297
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=74ed2fe69f15d86252b1c7779c916cb93ffc252d
Submitter: Zuul
Branch: stable/rocky

commit 74ed2fe69f15d86252b1c7779c916cb93ffc252d
Author: Sean Mooney <email address hidden>
Date: Fri Oct 12 14:26:14 2018 +0100

    Harden placement init under wsgi

    - This change tries to address an edge case discovered
      when running placement under mod_wsgi where
      if the placement wsgi application is re-initialize the db_api
      configure method attempts to reconfigure a started transaction
      factory.

    - This since oslo.db transaction factories do not support
      reconfiguration at runtime this result in an exception being
      raised preventing reloading of the Placement API without
      restarting apache to force mod_wsgi to recreate the
      python interpreter.

    - This change introduces a run once decorator to allow annotating
      functions that should only be executed once for the lifetime fo
      an interpreter.

    - This change applies the run_once decorator to the db_api configure
      method, to suppress the attempt to reconfigure the current
      TransactionFactory on application reload.

    NOTE(lyarwood): Simple positional conflict due to
    Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 and
    I81d13418d75b46fbdb9f6d44889a207528c8d6de not being present in
    stable/rocky.

    Conflicts:
            nova/utils.py

    Co-Authored-By: Balazs Gibizer <email address hidden>
    Closes-Bug: #1799246
    Related-Bug: #1784155
    Change-Id: I704196711d30c1124e713ac31111a8ea6fa2f1ba
    (cherry picked from commit 00d08a3288f61268f8a823991fadf0d83c36c04d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.1.0

This issue was fixed in the openstack/nova 18.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.