Description
===========
During upgrade we run nova online migration that goes through the list of instances and creates a request spec record in the db if one does not exist. As the online migrations are batched, the request spec migration leaves a marker record in the request_specs table to indicate the last instance uuid that was processed. It continues processing starting from that instances on the next batch.
In our upgrade test, we hit a scenario where the marker instance from the online migration that was run during the Mitaka->Newton upgrade had been deleted and purged from the db by time we ran the Newton->Pike upgrade. This caused the online migration to fail as the marker instance couldn't be found.
Steps to reproduce
==================
- run data online migration on installed Newton load.
nova-manage db online_data_migrations
- delete the instance referenced by the marker (instance_uuid 00000000-0000-0000-0000-000000000000)
- purge db:
nova-manage db purge
- upgrade to Pike.
Expected result
===============
Upgrade successful with no exceptions.
Actual result
=============
Exceptions occur during upgrade with missing marker an upgrade failed.
Error attempting to run <function migrate_instances_add_request_spec at 0x5151050>
14 rows matched query service_uuids_online_data_migration, 14 migrated
13 rows matched query migrate_quota_limits_to_api_db, 13 migrated
Error attempting to run <function migrate_instances_add_request_spec at 0x5151050>
+---------------------------------------------+--------------+-----------+
| Migration | Total Needed | Completed |
+---------------------------------------------+--------------+-----------+
| delete_build_requests_with_no_instance_uuid | 0 | 0 |
| migrate_aggregate_reset_autoincrement | 0 | 0 |
| migrate_aggregates | 0 | 0 |
| migrate_flavor_reset_autoincrement | 0 | 0 |
| migrate_flavors | 0 | 0 |
| migrate_instance_groups_to_api_db | 0 | 0 |
| migrate_instance_keypairs | 0 | 0 |
| migrate_instances_add_request_spec | 0 | 0 |
| migrate_keypairs_to_api_db | 0 | 0 |
| migrate_quota_classes_to_api_db | 0 | 0 |
| migrate_quota_limits_to_api_db | 0 | 0 |
| service_uuids_online_data_migration | 0 | 0 |
+---------------------------------------------+--------------+-----------+
> In our upgrade test, we hit a scenario where the marker instance from the online migration that was run during the Mitaka->Newton upgrade had been deleted and purged from the db
Why/how was the instance marker record deleted? Since it's not owned by any real project/user someone shouldn't be able to list/show it in the API (except maybe an admin when listing servers for all tenants?).