neutron

DB2 error occurs when neutron server enables multiple api workers

Series kilo
Bug #1458718

Bug #1458718 reported by Gong Zhang on 2015-05-26

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Undecided	Gong Zhang	neutron 7.0.0 "liberty"
	Kilo	New	Undecided	Unassigned

Bug Description

When neutron server enables multiple api workers, it will use os.fork to start multiple neutron server process. During this period, some DB2 error will occur as below, which shows we are trying to close a closed connection. It seems like that pooled connection is shared by processes.

2015-04-29 22:27:39.330 567 ERROR sqlalchemy.pool.QueuePool [-] Exception closing connection <ibm_db_dbi.Connection object at 0x469a190>
2015-04-29 22:27:39.330 567 TRACE sqlalchemy.pool.QueuePool Traceback (most recent call last):
2015-04-29 22:27:39.330 567 TRACE sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 250, in _close_connection
2015-04-29 22:27:39.330 567 TRACE sqlalchemy.pool.QueuePool self._dialect.do_close(connection)
2015-04-29 22:27:39.330 567 TRACE sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 412, in do_close
2015-04-29 22:27:39.330 567 TRACE sqlalchemy.pool.QueuePool dbapi_connection.close()
2015-04-29 22:27:39.330 567 TRACE sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/ibm_db_dbi.py", line 688, in close
2015-04-29 22:27:39.330 567 TRACE sqlalchemy.pool.QueuePool raise _get_exception(inst)
2015-04-29 22:27:39.330 567 TRACE sqlalchemy.pool.QueuePool OperationalError: ibm_db_dbi::OperationalError: [IBM][CLI Driver] CLI0106E Connection is closed. SQLSTATE=08003 SQLCODE=-99999

Currently neutron is using dispose() in child process to release the connnection and create new one. Actually we should dispose the pool before os.fork in father process.

Reference to sqlalchemy doc(http://docs.sqlalchemy.org/en/latest/core/connections.html#basic-usage)

See original description

Tags:

Gong Zhang (zhanggbj) on 2015-05-26

Changed in neutron:
assignee:	nobody → Zhang Gong (zhanggbj)

Gong Zhang (zhanggbj) on 2015-05-26

summary:	- DB2 error occurs when neutron server enables multiple api works + DB2 error occurs when neutron server enables multiple api workers
Changed in neutron:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-27: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/185898

Gong Zhang (zhanggbj) on 2015-05-27

description:

updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-06-02: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/185898
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=88e499d1c10eaae59546d9f16c9c9c262766de84
Submitter: Jenkins
Branch: master

commit 88e499d1c10eaae59546d9f16c9c9c262766de84
Author: Gong Zhang <email address hidden>
Date: Wed May 27 17:10:17 2015 +0800

Move pool dispose() before os.fork

    Currently pool dispose() is done after os.fork, but this will
    produce shared DB connections in child processes which may lead
    to DB errors.

    Move pool dispose() before os.fork. This will remove all existing
    connections in the parent process and child processes will create
    their own new ones.

Change-Id: Ie36417a64f0eb39b53dad61517f834aec37bacfb
Closes-Bug: 1458718

Changed in neutron:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-06-05: Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/188658

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-06-05: Change abandoned on neutron (stable/kilo)

Change abandoned by Zhang Gong (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/188658
Reason: Wrong Commit

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-06-09: Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/188658
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2884595c96f331facb11510057d68d5a01bb4747
Submitter: Jenkins
Branch: stable/kilo

commit 2884595c96f331facb11510057d68d5a01bb4747
Author: Gong Zhang <email address hidden>
Date: Wed May 27 17:10:17 2015 +0800

Move pool dispose() before os.fork

    Currently pool dispose() is done after os.fork, but this will
    produce shared DB connections in child processes which may lead
    to DB errors.

    Move pool dispose() before os.fork. This will remove all existing
    connections in the parent process and child processes will create
    their own new ones.

(cherry-picked from 88e499d1c10eaae59546d9f16c9c9c262766de84)

Change-Id: Ie36417a64f0eb39b53dad61517f834aec37bacfb
Closes-Bug: 1458718

tags:

added: in-stable-kilo

Thierry Carrez (ttx) on 2015-06-24

Changed in neutron:
milestone:	none → liberty-1
status:	Fix Committed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-06-29: Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/196701

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-06-30: Change abandoned on neutron (feature/pecan)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/196701
Reason: This is lacking the functional fix [1], so I'll propose a new merge commit which includes that one.

[1] https://review.openstack.org/#/c/196711/