if keystone starts first, spam of errors

Bug #1379057 reported by Steven Dake
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Steven Dake

Bug Description

If the keystone container starts prior to the mariadb container, the error logs contain all kinds of spam. Before keystone is started, mariadb health should be checked to determine that is up and operational.

[fedora@my-n6xa3qimt6on-0-yhalyw3gjl6p-kube-node-r2v7264y2be6 ~]$ sudo docker logs 64cb16680cab

2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/catalog/core.py", line 117, in list_services
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return self.driver.list_services(hints or driver_hints.Hints())
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/sql/core.py", line 224, in wrapper
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return f(self, hints, *args, **kwargs)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/catalog/backends/sql.py", line 162, in list_services
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi session = sql.get_session()
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/sql/core.py", line 192, in get_session
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return _get_engine_facade().get_session(expire_on_commit=expire_on_commit)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/sql/core.py", line 176, in _get_engine_facade
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi CONF.database.connection, CONF)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/openstack/common/db/sqlalchemy/session.py", line 904, in from_config
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi **dict(conf.database.items()))
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/openstack/common/db/sqlalchemy/session.py", line 847, in __init__
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi retry_interval=kwargs.get('retry_interval', 10))
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/openstack/common/db/sqlalchemy/session.py", line 653, in create_engine
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi _mysql_set_mode_callback(engine, mysql_sql_mode)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/openstack/common/db/sqlalchemy/session.py", line 577, in _mysql_set_mode_callback
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi _mysql_check_effective_sql_mode(engine)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/openstack/common/db/sqlalchemy/session.py", line 556, in _mysql_check_effective_sql_mode
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi realmode = _mysql_get_effective_sql_mode(engine)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/openstack/common/db/sqlalchemy/session.py", line 548, in _mysql_get_effective_sql_mode
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi row = engine.execute("SHOW VARIABLES LIKE 'sql_mode'").fetchone()
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1613, in execute
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi connection = self.contextual_connect(close_with_result=True)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi self.pool.connect(),
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 326, in connect
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return _ConnectionFairy(self).checkout()
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in __init__
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi rec = self._connection_record = pool._do_get()
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 770, in _do_get
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return self._create_connection()
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 279, in _create_connection
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return _ConnectionRecord(self)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 372, in __init__
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi self.connection = self.__connect()
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 433, in __connect
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi connection = self.__pool._creator()
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return dialect.connect(*cargs, **cparams)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return self.dbapi.connect(*cargs, **cparams)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi return Connection(*args, **kwargs)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi super(Connection, self).__init__(*args, **kwargs2)
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi OperationalError: (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") None None
2014-10-08 22:48:28.077 39 TRACE keystone.common.wsgi
An unexpected error prevented the server from fulfilling your request. (HTTP 500)

Steven Dake (sdake)
Changed in kolla:
importance: Undecided → Critical
status: New → Triaged
Steven Dake (sdake)
Changed in kolla:
assignee: nobody → Steven Dake (sdake)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla (master)

Change abandoned by Steven Dake (<email address hidden>) on branch: master
Review: https://review.openstack.org/127093
Reason: larsks indicates he has something to resolve this issue in his queued patches

Steven Dake (sdake)
Changed in kolla:
status: Triaged → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/128075
Committed: https://git.openstack.org/cgit/stackforge/kolla/commit/?id=a19e1cdd00f0b3394f9146eab8cb7f7fe2b6e0f2
Submitter: Jenkins
Branch: master

commit a19e1cdd00f0b3394f9146eab8cb7f7fe2b6e0f2
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Oct 10 11:33:43 2014 -0400

    keystone: exit on errors; cleanup

    Cause keystone start script to exit on any errors (permitting kubernetes
    to reschedule the pod).

    Closes-bug: 1379057
    Change-Id: Ide2eac6495250c8cba91cf9b7409dcb313f6f74b

Steven Dake (sdake)
Changed in kolla:
milestone: none → liberty-1
Sam Yaple (s8m)
Changed in kolla:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.