Nova API logs:
2016-09-06 03:15:02.287 11428 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212 2016-09-06 03:15:02.289 11428 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 2016-09-06 03:15:02.235 11858 DEBUG oslo_db.sqlalchemy.engines [req-a55cfeec-cf17-41ba-bfa0-ed8b990b0675 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256 2016-09-06 03:15:02.324 11429 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212 2016-09-06 03:15:02.342 11429 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 2016-09-06 03:15:02.452 12084 INFO oslo_service.service [-] Child 11428 exited with status 1 2016-09-06 03:15:02.490 12084 DEBUG oslo_service.service [-] Started child 11915 _start_child /usr/lib/python2.7/dist-packages/oslo_service/service.py:475 2016-09-06 03:15:02.550 12084 INFO oslo_service.service [-] Child 11429 exited with status 1 2016-09-06 03:15:02.667 12084 DEBUG oslo_service.service [-] Started child 11923 _start_child /usr/lib/python2.7/dist-packages/oslo_service/service.py:475 2016-09-06 03:15:03.306 11915 DEBUG oslo_db.sqlalchemy.engines [req-306febd7-3f59-49ef-ab24-127303c76c48 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256 2016-09-06 03:15:03.325 11923 DEBUG oslo_db.sqlalchemy.engines [req-fc808806-e531-4dc2-9512-6ce6dc34c81d - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service [req-c9831327-9d76-46d7-b24e-e572f2bbd0bd - - - - -] Error starting thread. 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service Traceback (most recent call last): 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 680, in run_service 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service service.start() 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 434, in start 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service self.binary) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service result = fn(cls, context, *args, **kwargs) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 230, in get_by_host_and_binary 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service host, binary) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 131, in service_get_by_host_and_binary 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service return IMPL.service_get_by_host_and_binary(context, host, binary) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 315, in wrapped 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service return f(context, *args, **kwargs) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 560, in service_get_by_host_and_binary 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service filter_by(binary=binary).\ 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2634, in first 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service ret = list(self[0:1]) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__ 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service return list(res) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__ 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service return self._execute_and_instances(context) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service result = conn.execute(querycontext.statement, self._params) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service return meth(self, multiparams, params) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service return connection._execute_clauseelement(self, multiparams, params) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service compiled_sql, distilled_params 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service context) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service util.raise_from_cause(newraise, exc_info) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service reraise(type(exception), exception, tb=exc_tb, cause=cause) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service context) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service cursor.execute(statement, parameters) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service self.errorhandler(self, exc, value) 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service raise errorvalue 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service OperationalError: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'services.last_seen_up' in 'field list'") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up, services.forced_down AS services_forced_down, services.version AS services_version \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.`binary` = %s \n LIMIT %s'] [parameters: (0, '10.109.6.9', 'nova-osapi_compute', 1)] 2016-09-06 03:15:04.361 11507 ERROR oslo_service.service 2016-09-06 03:15:04.536 11507 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212 2016-09-06 03:15:04.568 11507 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 2016-09-06 03:15:04.706 12084 INFO oslo_service.service [-] Child 11507 exited with status 1
Nova API logs:
2016-09-06 03:15:02.287 11428 DEBUG oslo_concurrenc y.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/ python2. 7/dist- packages/ oslo_concurrenc y/lockutils. py:212 y.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/ python2. 7/dist- packages/ oslo_concurrenc y/lockutils. py:225 sqlalchemy. engines [req-a55cfeec- cf17-41ba- bfa0-ed8b990b06 75 - - - - -] MySQL server mode set to STRICT_ TRANS_TABLES, STRICT_ ALL_TABLES, NO_ZERO_ IN_DATE, NO_ZERO_ DATE,ERROR_ FOR_DIVISION_ BY_ZERO, TRADITIONAL, NO_AUTO_ CREATE_ USER,NO_ ENGINE_ SUBSTITUTION _check_ effective_ sql_mode /usr/lib/ python2. 7/dist- packages/ oslo_db/ sqlalchemy/ engines. py:256 y.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/ python2. 7/dist- packages/ oslo_concurrenc y/lockutils. py:212 y.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/ python2. 7/dist- packages/ oslo_concurrenc y/lockutils. py:225 service [-] Child 11428 exited with status 1 service [-] Started child 11915 _start_child /usr/lib/ python2. 7/dist- packages/ oslo_service/ service. py:475 service [-] Child 11429 exited with status 1 service [-] Started child 11923 _start_child /usr/lib/ python2. 7/dist- packages/ oslo_service/ service. py:475 sqlalchemy. engines [req-306febd7- 3f59-49ef- ab24-127303c76c 48 - - - - -] MySQL server mode set to STRICT_ TRANS_TABLES, STRICT_ ALL_TABLES, NO_ZERO_ IN_DATE, NO_ZERO_ DATE,ERROR_ FOR_DIVISION_ BY_ZERO, TRADITIONAL, NO_AUTO_ CREATE_ USER,NO_ ENGINE_ SUBSTITUTION _check_ effective_ sql_mode /usr/lib/ python2. 7/dist- packages/ oslo_db/ sqlalchemy/ engines. py:256 sqlalchemy. engines [req-fc808806- e531-4dc2- 9512-6ce6dc34c8 1d - - - - -] MySQL server mode set to STRICT_ TRANS_TABLES, STRICT_ ALL_TABLES, NO_ZERO_ IN_DATE, NO_ZERO_ DATE,ERROR_ FOR_DIVISION_ BY_ZERO, TRADITIONAL, NO_AUTO_ CREATE_ USER,NO_ ENGINE_ SUBSTITUTION _check_ effective_ sql_mode /usr/lib/ python2. 7/dist- packages/ oslo_db/ sqlalchemy/ engines. py:256 service [req-c9831327- 9d76-46d7- b24e-e572f2bbd0 bd - - - - -] Error starting thread. service Traceback (most recent call last): service File "/usr/lib/ python2. 7/dist- packages/ oslo_service/ service. py", line 680, in run_service service service.start() service File "/usr/lib/ python2. 7/dist- packages/ nova/service. py", line 434, in start service self.binary) service File "/usr/lib/ python2. 7/dist- packages/ oslo_versionedo bjects/ base.py" , line 181, in wrapper service result = fn(cls, context, *args, **kwargs) service File "/usr/lib/ python2. 7/dist- packages/ nova/objects/ service. py", line 230, in get_by_ host_and_ binary service host, binary) service File "/usr/lib/ python2. 7/dist- packages/ nova/db/ api.py" , line 131, in service_ get_by_ host_and_ binary service return IMPL.service_ get_by_ host_and_ binary( context, host, binary) service File "/usr/lib/ python2. 7/dist- packages/ nova/db/ sqlalchemy/ api.py" , line 315, in wrapped service return f(context, *args, **kwargs) service File "/usr/lib/ python2. 7/dist- packages/ nova/db/ sqlalchemy/ api.py" , line 560, in service_ get_by_ host_and_ binary service filter_ by(binary= binary) .\ service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2634, in first service ret = list(self[0:1]) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2457, in __getitem__ service return list(res) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2736, in __iter__ service return self._execute_ and_instances( context) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2751, in _execute_ and_instances service result = conn.execute( querycontext. statement, self._params) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 914, in execute service return meth(self, multiparams, params) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ sql/elements. py", line 323, in _execute_ on_connection service return connection. _execute_ clauseelement( self, multiparams, params) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 1010, in _execute_ clauseelement service compiled_sql, distilled_params service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 1146, in _execute_context service context) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 1337, in _handle_ dbapi_exception service util.raise_ from_cause( newraise, exc_info) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ util/compat. py", line 200, in raise_from_cause service reraise( type(exception) , exception, tb=exc_tb, cause=cause) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 1139, in _execute_context service context) service File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ engine/ default. py", line 450, in do_execute service cursor. execute( statement, parameters) service File "/usr/lib/ python2. 7/dist- packages/ MySQLdb/ cursors. py", line 219, in execute service self.errorhandl er(self, exc, value) service File "/usr/lib/ python2. 7/dist- packages/ MySQLdb/ connections. py", line 38, in defaulterrorhandler service raise errorvalue service OperationalError: (_mysql_ exceptions. OperationalErro r) (1054, "Unknown column 'services. last_seen_ up' in 'field list'") [SQL: u'SELECT services.created_at AS services_ created_ at, services.updated_at AS services_ updated_ at, services.deleted_at AS services_ deleted_ at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services. report_ count AS services_ report_ count, services.disabled AS services_disabled, services. disabled_ reason AS services_ disabled_ reason, services. last_seen_ up AS services_ last_seen_ up, services. forced_ down AS services_ forced_ down, services.version AS services_version \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.`binary` = %s \n LIMIT %s'] [parameters: (0, '10.109.6.9', 'nova-osapi_ compute' , 1)] service y.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/ python2. 7/dist- packages/ oslo_concurrenc y/lockutils. py:212 y.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/ python2. 7/dist- packages/ oslo_concurrenc y/lockutils. py:225 service [-] Child 11507 exited with status 1
2016-09-06 03:15:02.289 11428 DEBUG oslo_concurrenc
2016-09-06 03:15:02.235 11858 DEBUG oslo_db.
2016-09-06 03:15:02.324 11429 DEBUG oslo_concurrenc
2016-09-06 03:15:02.342 11429 DEBUG oslo_concurrenc
2016-09-06 03:15:02.452 12084 INFO oslo_service.
2016-09-06 03:15:02.490 12084 DEBUG oslo_service.
2016-09-06 03:15:02.550 12084 INFO oslo_service.
2016-09-06 03:15:02.667 12084 DEBUG oslo_service.
2016-09-06 03:15:03.306 11915 DEBUG oslo_db.
2016-09-06 03:15:03.325 11923 DEBUG oslo_db.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.361 11507 ERROR oslo_service.
2016-09-06 03:15:04.536 11507 DEBUG oslo_concurrenc
2016-09-06 03:15:04.568 11507 DEBUG oslo_concurrenc
2016-09-06 03:15:04.706 12084 INFO oslo_service.