Cache database and message queue connection objects
Recently in the gate we have seen a trace on some work-in-progress
patches:
OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
and at least one operator has reported that the number of database
connections increased significantly going from Mitaka to Newton.
It was suspected that the increase was caused by creating new oslo.db
transaction context managers on-the-fly when switching database
connections for cells. Comparing the dstat --tcp output of runs of the
gate-tempest-dsvm-neutron-full-ubuntu-xenial job with and without
caching of the database connections showed a difference of 445 active
TCP connections and 1495 active TCP connections, respectively [1].
This adds caching of the oslo.db transaction context managers and the
oslo.messaging transports to avoid creating a large number of objects
that are not being garbage-collected as expected.
NOTE(melwitt): Conflicts caused by the fact that no other global cache
resets exist in nova.test.py in Newton and the get_context function
doesn't exist in Newton.
Change-Id: I17e0eb836dd87aac5859f506e7d771d42753d31a
(cherry picked from commit f4159d17552603b90912dba6fe5c604e8d0b8aa7)
Reviewed: https:/ /review. openstack. org/466424 /git.openstack. org/cgit/ openstack/ nova/commit/ ?id=d6a628da62f 810310ab1bdc2e0 4222d8010e7b62
Committed: https:/
Submitter: Jenkins
Branch: stable/newton
commit d6a628da62f8103 10ab1bdc2e04222 d8010e7b62
Author: melanie witt <email address hidden>
Date: Tue May 16 10:25:42 2017 +0000
Cache database and message queue connection objects
Recently in the gate we have seen a trace on some work-in-progress
patches:
Operation alError: (pymysql. err.Operational Error)
(1040, u'Too many connections')
and at least one operator has reported that the number of database
connections increased significantly going from Mitaka to Newton.
It was suspected that the increase was caused by creating new oslo.db tempest- dsvm-neutron- full-ubuntu- xenial job with and without
transaction context managers on-the-fly when switching database
connections for cells. Comparing the dstat --tcp output of runs of the
gate-
caching of the database connections showed a difference of 445 active
TCP connections and 1495 active TCP connections, respectively [1].
This adds caching of the oslo.db transaction context managers and the
oslo.messaging transports to avoid creating a large number of objects
that are not being garbage-collected as expected.
Closes-Bug: #1691545
[1] https:/ /docs.google. com/spreadsheet s/d/1DIfFfX3kaA _SRoCM- aO7BN4IBEShChXL ztOBFeKryt4/ edit?usp= sharing
Conflicts: tests/unit/ test_context. py
nova/test.py
nova/
NOTE(melwitt): Conflicts caused by the fact that no other global cache
resets exist in nova.test.py in Newton and the get_context function
doesn't exist in Newton.
Change-Id: I17e0eb836dd87a ac5859f506e7d77 1d42753d31a 90912dba6fe5c60 4e8d0b8aa7)
(cherry picked from commit f4159d17552603b