Nova upgrade fails if PCI devices of type-PF or type-PCI are present in the database
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
High
|
Steven Webster | ||
Newton |
Fix Committed
|
High
|
Dan Smith | ||
Ocata |
Fix Committed
|
High
|
Dan Smith |
Bug Description
Description
===========
If a Nova DB is upgraded (migrated) while containing PCI devices with device type 'type-PF' or 'type-PCI',
a validation error similar to this will be thrown:
"ValidationError: There are still 2 unmigrated records in the pci_devices table. Migration cannot continue until all records have been migrated."
The error is generated by the 330_enforce_
The PCI device migration validation will fail if any PCI device entries without a populated parent_addr are found. However, the parent_addr really only applies to PCI device entries of 'type-VF' (ie. SRIOV virtual functions)
This is an example of what the pci_devices table looks like with SRIOV enabled PCI devices if the appropriate entries are whitelisted in nova.conf:
MariaDB [nova]> select * from pci_devices;
+------
| created_at | updated_at | deleted_at | deleted | id | compute_node_id | address | product_id | vendor_id | dev_type | dev_id | label | status | extra_info | instance_uuid | request_id | numa_node | parent_addr |
+------
| 2017-04-06 21:01:21 | 2017-04-06 21:53:13 | NULL | 0 | 1 | 1 | 0000:05:10.1 | 10ed | 8086 | type-VF | pci_0000_05_10_1 | label_8086_10ed | available | {} | NULL | NULL | 0 | 0000:05:00.1 |
| 2017-04-06 21:01:21 | 2017-04-06 21:53:13 | NULL | 0 | 2 | 1 | 0000:05:10.3 | 10ed | 8086 | type-VF | pci_0000_05_10_3 | label_8086_10ed | available | {} | NULL | NULL | 0 | 0000:05:00.1 |
| 2017-04-06 21:01:21 | 2017-04-06 21:53:13 | NULL | 0 | 3 | 1 | 0000:05:10.5 | 10ed | 8086 | type-VF | pci_0000_05_10_5 | label_8086_10ed | available | {} | NULL | NULL | 0 | 0000:05:00.1 |
| 2017-04-06 21:01:21 | 2017-04-06 21:53:13 | NULL | 0 | 4 | 1 | 0000:05:10.7 | 10ed | 8086 | type-VF | pci_0000_05_10_7 | label_8086_10ed | available | {} | NULL | NULL | 0 | 0000:05:00.1 |
| 2017-04-06 21:53:13 | NULL | NULL | 0 | 5 | 1 | 0000:05:00.0 | 10fb | 8086 | type-PF | pci_0000_05_00_0 | label_8086_10fb | available | {} | NULL | NULL | 0 | NULL |
| 2017-04-06 21:53:13 | NULL | NULL | 0 | 6 | 1 | 0000:05:00.1 | 10fb | 8086 | type-PF | pci_0000_05_00_1 | label_8086_10fb | available | {} | NULL | NULL | 0 | NULL |
+------
6 rows in set (0.00 sec)
I think the upgrade script should be checking the PciDevice dev_type field for 'type-VF' when validating the parent_addr.
Steps to reproduce
==================
1. Install a Mitaka control node and edit the nova.conf file to include 1 or more PCI devices in the pci_passthrough
pci_passthrough
2. Install a second Newton or newer control node and edit the nova.conf to point to the SQL database of the Mitaka node. ie:
[database]
connection = mysql+pymysql:
[api_database]
connection = mysql+pymysql:
3. From the new control node, issue the following command:
nova-manage db sync
Expected result
===============
Database migration/upgrade should succeed
Actual result
=============
A ValidationError, similar to:
"ValidationError: There are still <N> unmigrated records in the pci_devices table. Migration cannot continue until all records have been migrated."
Environment
===========
1. Exact version of OpenStack you are running. See the following
Mitaka (old node) Newton, Ocata (new node)
2. Which hypervisor did you use?
libvirt + kvm
2. Which storage type did you use?
lvm
3. Which networking type did you use?
Neutron, OVS
Logs & Configs
==============
See attached
Changed in nova: | |
importance: | Undecided → High |
Fix proposed to branch: master /review. openstack. org/456397
Review: https:/