nova-{api,scheduler,consoleauth,cert} services fail right after the start.
The error message is (from /var/log/nova-api.log):
2016-11-11 14:38:57.074 21442 ERROR oslo_service.service OperationalError: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'services.last_seen_up' in 'field list'") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up, services.forced_down AS services_forced_down, services.version AS services_version \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.`binary` = %s \n LIMIT %s'] [parameters: (0, '192.168.0.8', 'nova-metadata', 1)]
2016-11-11 14:38:57.074 21442 ERROR oslo_service.service
2016-11-11 14:38:57.453 4895 INFO oslo_service.service [-] Child 21412 exited with status 1
root@node-1:~# apt-cache policy nova-api
nova-api:
Installed: 2:13.1.1-7~u14.04+mos20
Candidate: 2:13.1.1-7~u14.04+mos20
Version table:
*** 2:13.1.1-7~u14.04+mos20 0
1050 http://10.20.0.1/mirror/mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-updates/main amd64 Packages
100 /var/lib/dpkg/status
2:13.0.0-6~u14.04+mos43 0
1050 http://10.20.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
1:2014.1.5-0ubuntu1.5 0
500 http://10.20.0.1/mirror/archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
1:2014.1.3-0ubuntu1.1 0
500 http://10.20.0.1/mirror/archive.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
1:2014.1-0ubuntu1 0
500 http://10.20.0.1/mirror/archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
Setting importance to Critical since this breaks the deployment (100% reproducible)
mysql> describe services;
+------
| Field | Type | Null | Key | Default | Extra |
+------
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| deleted_at | datetime | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment |
| host | varchar(255) | YES | MUL | NULL | |
| binary | varchar(255) | YES | | NULL | |
| topic | varchar(255) | YES | | NULL | |
| report_count | int(11) | NO | | NULL | |
| disabled | tinyint(1) | YES | | NULL | |
| deleted | int(11) | YES | | NULL | |
| disabled_reason | varchar(255) | YES | | NULL | |
+------
11 rows in set (0.00 sec)
There are no `last_seen_up' field in the `services' database.