Juniper Openstack

Openstack Helm : Launching a VM failed due to mysql DBReferenceErro

Series trunk
Bug #1749614

Bug #1749614 reported by venu kolli on 2018-02-14

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R5.0	New	High	Madhava Jayamani	Juniper Openstack r5.0.3
Trunk	New	High	Madhava Jayamani	Juniper Openstack r5.1.0

Bug Description

Launching a VM failed due to mysql DBReferenceErro

Contrail build : 5.0.0-175

Provisioned by contrail-helm charts

# Role definition of the hosts.
env.roledefs = {
    'all': [host1, host2, host3, host4,host5],
    'cfgm': [host1, host2, host3],
    'openstack': [host1],
    'control': [host1, host2, host3],
    'compute': [host4,host5],
    'collector': [host1, host2, host3],
    'webui': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

VM with instanceid d57ac8a7-b4cf-48f6-9194-cb8bf99ced0d failed

t default] 192.168.231.204,10.84.29.32 \"POST /v2/abafb177bc5047b9998a872d4ad701f4/os-server-external-events HTTP/1.1\" status: 200 len: 444 time: 0.1740110\n","stream":"stderr","time":"2018-02-14T19:50:59.24646567Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions [req-6ddb3f12-a9f8-41e6-aaa0-6f85eeda8428 d0afc688d8144eab8c61c8becca55096 bdac9c8d03d7415e87a39ab36aa32b1d - default default] Unexpected exception in API method\n","stream":"stderr","time":"2018-02-14T19:51:00.218987612Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions Traceback (most recent call last):\n","stream":"stderr","time":"2018-02-14T19:51:00.219027353Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/api/openstack/extensions.py\", line 338, in wrapped\n","stream":"stderr","time":"2018-02-14T19:51:00.219034919Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return f(*args, **kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.21904109Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/api/validation/__init__.py\", line 73, in wrapper\n","stream":"stderr","time":"2018-02-14T19:51:00.21904703Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return func(*args, **kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.219052915Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/api/validation/__init__.py\", line 73, in wrapper\n","stream":"stderr","time":"2018-02-14T19:51:00.219057752Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return func(*args, **kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.219063027Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/api/validation/__init__.py\", line 73, in wrapper\n","stream":"stderr","time":"2018-02-14T19:51:00.219067893Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return func(*args, **kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.219073192Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/api/validation/__init__.py\", line 73, in wrapper\n","stream":"stderr","time":"2018-02-14T19:51:00.219078035Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return func(*args, **kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.219083349Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/api/validation/__init__.py\", line 73, in wrapper\n","stream":"stderr","time":"2018-02-14T19:51:00.219088136Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return func(*args, **kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.219093498Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py\", line 631, in create\n","stream":"stderr","time":"2018-02-14T19:51:00.219100032Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions **create_kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.219105523Z"}

{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/api.py\", line 1159, in _create_instance\n","stream":"stderr","time":"2018-02-14T19:51:00.219142374Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions key_pair)\n","stream":"stderr","time":"2018-02-14T19:51:00.219148041Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/api.py\", line 1022, in _provision_instances\n","stream":"stderr","time":"2018-02-14T19:51:00.219152629Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions quotas.rollback()\n","stream":"stderr","time":"2018-02-14T19:51:00.219157587Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py\", line 220, in __exit__\n","stream":"stderr","time":"2018-02-14T19:51:00.219162187Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions self.force_reraise()\n","stream":"stderr","time":"2018-02-14T19:51:00.219167038Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise\n","stream":"stderr","time":"2018-02-14T19:51:00.219171416Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)\n","stream":"stderr","time":"2018-02-14T19:51:00.219176358Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/api.py\", line 976, in _provision_instances\n","stream":"stderr","time":"2018-02-14T19:51:00.219180855Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions self._create_block_device_mapping(block_device_mapping)\n","stream":"stderr","time":"2018-02-14T19:51:00.219186412Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/api.py\", line 1258, in _create_block_device_mapping\n","stream":"stderr","time":"2018-02-14T19:51:00.219191078Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions bdm.update_or_create()\n","stream":"stderr","time":"2018-02-14T19:51:00.219196201Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py\", line 226, in wrapper\n","stream":"stderr","time":"2018-02-14T19:51:00.219200515Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.219205349Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/objects/block_device.py\", line 166, in update_or_create\n","stream":"stderr","time":"2018-02-14T19:51:00.219209687Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions self._create(self._context, update_or_create=True)\n","stream":"stderr","time":"2018-02-14T19:51:00.219214498Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/objects/block_device.py\", line 145, in _create\n","stream":"stderr","time":"2018-02-14T19:51:00.219218917Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions context, updates, legacy=False)\n","stream":"stderr","time":"2018-02-14T19:51:00.219223689Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/db/api.py\", line 1293, in block_device_mapping_update_or_create\n","stream":"stderr","time":"2018-02-14T19:51:00.219228117Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return IMPL.block_device_mapping_update_or_create(context, values, legacy)\n","stream":"stderr","time":"2018-02-14T19:51:00.219237434Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py\", line 240, in wrapped\n","stream":"stderr","time":"2018-02-14T19:51:00.219242651Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs)\n","stream":"stderr","time":"2018-02-14T19:51:00.219247523Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py\", line 4163, in block_device_mapping_update_or_create\n","stream":"stderr","time":"2018-02-14T19:51:00.219251854Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions result.save(context.session)\n","stream":"stderr","time":"2018-02-14T19:51:00.219257608Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py\", line 48, in save\n","stream":"stderr","time":"2018-02-14T19:51:00.219262291Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions session.flush()\n","stream":"stderr","time":"2018-02-14T19:51:00.219267632Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2019, in flush\n","stream":"stderr","time":"2018-02-14T19:51:00.219272094Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions self._flush(objects)\n","stream":"stderr","time":"2018-02-14T19:51:00.219276881Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2137, in _flush\n","stream":"stderr","time":"2018-02-14T19:51:00.219281238Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions transaction.rollback(_capture_exception=True)\n","stream":"stderr","time":"2018-02-14T19:51:00.219285937Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py\", line 60, in __exit__\n","stream":"stderr","time":"2018-02-14T19:51:00.219290285Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions compat.reraise(exc_type, exc_value, exc_tb)\n","stream":"stderr","time":"2018-02-14T19:51:00.219294979Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2101, in _flush\n","stream":"stderr","time":"2018-02-14T19:51:00.219299269Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions flush_context.execute()\n","stream":"stderr","time":"2018-02-14T19:51:00.219303899Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py\", line 373, in execute\n","stream":"stderr","time":"2018-02-14T19:51:00.219308101Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions rec.execute(self)\n","stream":"stderr","time":"2018-02-14T19:51:00.21931284Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py\", line 532, in execute\n","stream":"stderr","time":"2018-02-14T19:51:00.2193172Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions uow\n","stream":"stderr","time":"2018-02-14T19:51:00.219322166Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py\", line 174, in save_obj\n","stream":"stderr","time":"2018-02-14T19:51:00.219327068Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions mapper, table, insert)\n","stream":"stderr","time":"2018-02-14T19:51:00.219336165Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py\", line 800, in _emit_insert_statements\n","stream":"stderr","time":"2018-02-14T19:51:00.219341197Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions execute(statement, params)\n","stream":"stderr","time":"2018-02-14T19:51:00.21934615Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 914, in execute\n","stream":"stderr","time":"2018-02-14T19:51:00.219350645Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return meth(self, multiparams, params)\n","stream":"stderr","time":"2018-02-14T19:51:00.219355531Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py\", line 323, in _execute_on_connection\n","stream":"stderr","time":"2018-02-14T19:51:00.219359994Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions return connection._execute_clauseelement(self, multiparams, params)\n","stream":"stderr","time":"2018-02-14T19:51:00.219364837Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1010, in _execute_clauseelement\n","stream":"stderr","time":"2018-02-14T19:51:00.219369377Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions compiled_sql, distilled_params\n","stream":"stderr","time":"2018-02-14T19:51:00.219374204Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1146, in _execute_context\n","stream":"stderr","time":"2018-02-14T19:51:00.219378641Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions context)\n","stream":"stderr","time":"2018-02-14T19:51:00.219383417Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1337, in _handle_dbapi_exception\n","stream":"stderr","time":"2018-02-14T19:51:00.219387657Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions util.raise_from_cause(newraise, exc_info)\n","stream":"stderr","time":"2018-02-14T19:51:00.219392516Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py\", line 202, in raise_from_cause\n","stream":"stderr","time":"2018-02-14T19:51:00.219397124Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions reraise(type(exception), exception, tb=exc_tb, cause=cause)\n","stream":"stderr","time":"2018-02-14T19:51:00.219402527Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1010, in _execute_clauseelement\n","stream":"stderr","time":"2018-02-14T19:51:00.219369377Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions compiled_sql, distilled_params\n","stream":"stderr","time":"2018-02-14T19:51:00.219374204Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1146, in _execute_context\n","stream":"stderr","time":"2018-02-14T19:51:00.219378641Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions context)\n","stream":"stderr","time":"2018-02-14T19:51:00.219383417Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1337, in _handle_dbapi_exception\n","stream":"stderr","time":"2018-02-14T19:51:00.219387657Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions util.raise_from_cause(newraise, exc_info)\n","stream":"stderr","time":"2018-02-14T19:51:00.219392516Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py\", line 202, in raise_from_cause\n","stream":"stderr","time":"2018-02-14T19:51:00.219397124Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions reraise(type(exception), exception, tb=exc_tb, cause=cause)\n","stream":"stderr","time":"2018-02-14T19:51:00.219402527Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1139, in _execute_context\n","stream":"stderr","time":"2018-02-14T19:51:00.219407127Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions context)\n","stream":"stderr","time":"2018-02-14T19:51:00.219412013Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py\", line 450, in do_execute\n","stream":"stderr","time":"2018-02-14T19:51:00.219416538Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions cursor.execute(statement, parameters)\n","stream":"stderr","time":"2018-02-14T19:51:00.219421283Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/cursors.py\", line 167, in execute\n","stream":"stderr","time":"2018-02-14T19:51:00.219429478Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions result = self._query(query)\n","stream":"stderr","time":"2018-02-14T19:51:00.219434797Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/cursors.py\", line 323, in _query\n","stream":"stderr","time":"2018-02-14T19:51:00.219439375Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions conn.query(q)\n","stream":"stderr","time":"2018-02-14T19:51:00.21944429Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py\", line 836, in query\n","stream":"stderr","time":"2018-02-14T19:51:00.219448805Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n","stream":"stderr","time":"2018-02-14T19:51:00.219453652Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py\", line 1020, in _read_query_result\n","stream":"stderr","time":"2018-02-14T19:51:00.219458239Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions result.read()\n","stream":"stderr","time":"2018-02-14T19:51:00.219465439Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py\", line 1303, in read\n","stream":"stderr","time":"2018-02-14T19:51:00.219470051Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions first_packet = self.connection._read_packet()\n","stream":"stderr","time":"2018-02-14T19:51:00.219474966Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py\", line 982, in _read_packet\n","stream":"stderr","time":"2018-02-14T19:51:00.219480008Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions packet.check_error()\n","stream":"stderr","time":"2018-02-14T19:51:00.219484942Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py\", line 394, in check_error\n","stream":"stderr","time":"2018-02-14T19:51:00.219489568Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions err.raise_mysql_exception(self._data)\n","stream":"stderr","time":"2018-02-14T19:51:00.219494419Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/err.py\", line 120, in raise_mysql_exception\n","stream":"stderr","time":"2018-02-14T19:51:00.219498777Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions _check_mysql_exception(errinfo)\n","stream":"stderr","time":"2018-02-14T19:51:00.219503528Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/err.py\", line 112, in _check_mysql_exception\n","stream":"stderr","time":"2018-02-14T19:51:00.219507912Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions raise errorclass(errno, errorvalue)\n","stream":"stderr","time":"2018-02-14T19:51:00.219512768Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions DBReferenceError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`nova`.`block_device_mapping`, CONSTRAINT `block_device_mapping_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))') [SQL: u'INSERT INTO block_device_mapping (created_at, updated_at, deleted_at, deleted, instance_uuid, source_type, destination_type, guest_format, device_type, disk_bus, boot_index, device_name, delete_on_termination, snapshot_id, volume_id, volume_size, image_id, no_device, connection_info, tag) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(instance_uuid)s, %(source_type)s, %(destination_type)s, %(guest_format)s, %(device_type)s, %(disk_bus)s, %(boot_index)s, %(device_name)s, %(delete_on_termination)s, %(snapshot_id)s, %(volume_id)s, %(volume_size)s, %(image_id)s, %(no_device)s, %(connection_info)s, %(tag)s)'] [parameters: {'instance_uuid': 'd57ac8a7-b4cf-48f6-9194-cb8bf99ced0d', 'guest_format': None, 'boot_index': 0, 'delete_on_termination': 1, 'volume_id': None, 'no_device': 0, 'connection_info': None, 'created_at': datetime.datetime(2018, 2, 14, 19, 50, 59, 876680), 'snapshot_id': None, 'deleted': 0, 'updated_at': None, 'device_name': None, 'disk_bus': None, 'image_id': u'2d5b23c6-a1d7-43d9-902c-f665e909e174', 'source_type': u'image', 'tag': None, 'device_type': u'disk', 'destination_type': u'local', 'deleted_at': None, 'volume_size': None}]\n","stream":"stderr","time":"2018-02-14T19:51:00.219518196Z"}
{"log":"2018-02-14 19:51:00.207 1 ERROR nova.api.openstack.extensions \n","stream":"stderr","time":"2018-02-14T19:51:00.219533154Z"}
{"log":"2018-02-14 19:51:00.218 1 INFO nova.api.openstack.wsgi [req-6ddb3f12-a9f8-41e6-aaa0-6f85eeda8428 d0afc688d8144eab8c61c8becca55096 bdac9c8d03d7415e87a39ab36aa32b1d - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n","stream":"stderr","time":"2018-02-14T19:51:00.21953796Z"}
{"log":"\u003cclass 'oslo_db.exception.DBReferenceError'\u003e\n","stream":"stderr","time":"2018-02-14T19:51:00.219543459Z"}
{"log":"2018-02-14 19:51:00.220 1 INFO nova.osapi_compute.wsgi.server [req-6ddb3f12-a9f8-41e6-aaa0-6f85eeda8428 d0afc688d8144eab8c61c8becca55096 bdac9c8d03d7415e87a39ab36aa32b1d - default default] 172.17.0.2,10.84.29.32 \"POST /v2/bdac9c8d03d7415e87a39ab36aa32b1d/servers HTTP/1.1\" status: 500 len: 434 time: 1.2977979\n","stream":"stderr","time":"2018-02-14T19:51:00.220997545Z"}
{"log":"2018-02-14 22:59:17.599 1 INFO nova.osapi_compute.wsgi.server [req-f4f40dd1-26c2-47db-9f3d-7490e1d571bc 92389ff9b8584cbaa8dc93af9345995f be36578b7f714c5db08a32a0bd54a85b - default default] 10.84.29.32,10.84.29.32 \"GET /v2/be36578b7f714c5db08a32a0bd54a85b HTTP/1.1\" status: 404 len: 326 time: 0.1411641\n","stream":"stderr","time":"2018-02-14T22:59:17.599840153Z"}

Tags:

Jeba Paulaiyan (jebap) on 2018-02-22

tags:

added: helm sanityblocker

Revision history for this message

mnayakbomman@juniper.net (mnayakbomman) wrote on 2018-03-14:

Cannot recreate this issue at this time, please give steps to recreate the issue

Revision history for this message

venu kolli (vkolli) wrote on 2018-03-20:

You should be able to reproduce it when u have 3 mysql db cluster

Revision history for this message

mnayakbomman@juniper.net (mnayakbomman) wrote on 2018-03-20:

This is a known issue on openstack-helm side, can only be fixed once the fix is available on openstack-helm upstream repo

Answer from openstack-helm core reviewer (Alan)

"There is a known issue with > 1 – we are working on a sqlproxy solution to ensure all writes are sent to one container (reads can go to all). This isn’t really a containerization defect but simply an issue with multi-master mariadb members all receiving write requests. Today the service distributes them evenly to all containers, and in production environments (e.g. AIC 3.x) only one of the 3 members is actually used to avoid this."

Workaround is to decrease the replicas to 1

Vineet Gupta (vineetrf) on 2018-03-26

tags:

added: releasenote

Jeba Paulaiyan (jebap) on 2018-04-25

information type:

Proprietary → Public

Jeba Paulaiyan (jebap) on 2018-05-04

tags:

added: sanity
removed: sanityblocker

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.