Heat Stack Causes MariaDB Too Many Connections

Bug #1465422 reported by Daneyon Hansen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
High
Daneyon Hansen

Bug Description

OpenStack works just fine until creating a Heat stack. Creating the stack appears too creates too many connections to the DB, causing the DB to reach the max_connection count. Therefore, the DB blocks all connections from the source IP for a specific period of time. Since all OpenStack services are running on the same host, all services are blocked.

From the Heat Engine logs:

heatengine_1 | six.reraise(type(newraise), newraise, exc_info[2])
heatengine_1 | File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 109, in _wrap_pool_connect
heatengine_1 | return fn()
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 338, in connect
heatengine_1 | return _ConnectionFairy._checkout(self)
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 641, in _checkout
heatengine_1 | fairy = _ConnectionRecord.checkout(pool)
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 440, in checkout
heatengine_1 | rec = pool._do_get()
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 961, in _do_get
heatengine_1 | return self._create_connection()
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 285, in _create_connection
heatengine_1 | return _ConnectionRecord(self)
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 411, in __init__
heatengine_1 | self.connection = self.__connect()
heatengine_1 | File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in __connect
heatengine_1 | connection = self.__pool._creator()
heatengine_1 | File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 94, in connect
heatengine_1 | raise original_exception
heatengine_1 | OperationalError: (OperationalError) (1040, 'Too many connections') None None

Increasing the max_connection count fixes this problem.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.openstack.org/192011

Changed in kolla:
status: New → In Progress
Changed in kolla:
status: In Progress → Fix Committed
Changed in kolla:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.