Nova does not delete or update neutron port for failed VMs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
In Progress
|
Undecided
|
Mohammed Naser |
Bug Description
Environment:
Stable/Liberty with neutron ML2 (Mechanism Driver is a custom asynchronous driver based off the Opendaylight V2 driver)
No agents or OVS bridges in use.
One controller/network node and two compute nodes.
When a VM fails to start on any compute node, nova removes the host binding from the VM but doesn't send a port update to neutron notifying it about the hostbinding change.
When the same error state VM is deleted, nova doesn't send any events to neutron. As a result the driver thinks the port is still active, owned by an existing VM and bound to the host properly.
NOTE: This behavior is only seen with VM's in the ERROR state, ports for VMs in ACTIVE state are deleted properly.
$ nova show vm1
+------
| Property | Value |
+------
| OS-DCF:diskConfig | AUTO |
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-STS:vm_state | error |
| OS-SRV-
| OS-SRV-
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-06-
| fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/opt/
| flavor | m1.small (2) |
| hostId | |
| id | 5d37f0b1-
| image | cirros-
| key_name | - |
| metadata | {} |
| name | vm1 |
| net1 network | 30.0.0.231 |
| os-extended-
| security_groups | default |
| status | ERROR |
| tenant_id | 4c3ddd212d4f4af
| updated | 2016-06-
| user_id | 1e3e7e3addc1486
+------
$ neutron port-show 5efcfa1b-
+------
| Field | Value |
+------
| admin_state_up | True |
| allowed_
| binding:host_id | bxb-ds-50 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": false, "vhostuser_
| binding:vif_type | vhostuser |
| binding:vnic_type | normal |
| device_id | 5d37f0b1-
| device_owner | compute:nova |
| dns_assignment | {"hostname": "host-30-0-0-231", "ip_address": "30.0.0.231", "fqdn": "host-30-
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "880aac1b-
| id | 5efcfa1b-
| mac_address | fa:16:3e:ab:a0:d2 |
| name | |
| network_id | cf31ce68-
| port_security_
| security_groups | ae59a53e-
| status | ACTIVE |
| tenant_id | 4c3ddd212d4f4af
+------
$ nova delete vm1
Request to delete server vm1 has been accepted.
$ nova show vm1
ERROR (CommandError): No server with a name or ID of 'vm1' exists.
$ neutron port-show 5efcfa1b-
+------
| Field | Value |
+------
| admin_state_up | True |
| allowed_
| binding:host_id | bxb-ds-50 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": false, "vhostuser_
| binding:vif_type | vhostuser |
| binding:vnic_type | normal |
| device_id | 5d37f0b1-
| device_owner | compute:nova |
| dns_assignment | {"hostname": "host-30-0-0-231", "ip_address": "30.0.0.231", "fqdn": "host-30-
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "880aac1b-
| id | 5efcfa1b-
| mac_address | fa:16:3e:ab:a0:d2 |
| name | |
| network_id | cf31ce68-
| port_security_
| security_groups | ae59a53e-
| status | ACTIVE |
| tenant_id | 4c3ddd212d4f4af
+------
description: | updated |
Changed in nova: | |
assignee: | nobody → Baodong (Robert) Li (baoli) |
Changed in nova: | |
assignee: | Baodong (Robert) Li (baoli) → melanie witt (melwitt) |
Changed in nova: | |
assignee: | nobody → melanie witt (melwitt) |
status: | New → In Progress |
Changed in nova: | |
assignee: | melanie witt (melwitt) → Charlotte Han (hanrong) |
Changed in nova: | |
assignee: | Charlotte Han (hanrong) → Mohammed Naser (mnaser) |
On further investigation, it seems that the network_info cache in the db is wiped out on VM error: ******* ******* ****** 1. row ******* ******* ******* ******
created_ at: 2016-06-21 14:44:17
updated_ at: 2016-06-21 14:44:19
deleted_ at: NULL
id: 1964
internal_ id: NULL
user_ id: 1e3e7e3addc1486 1afe177f285e43b 40
project_ id: 4c3ddd212d4f4af 9b4d0d55e89b431 25
image_ ref: 50976ce1- c00e-4a80- 9d72-6bf5d1abe5 4a
kernel_ id: 72d053dc- aae8-4559- a7c4-107c980bd6 74
ramdisk_ id: 3d25fd54- ffbc-4370- b971-2196a7963c 24
launch_ index: 9
key_name: NULL
key_data: NULL
power_ state: 0
vm_state: error
memory_ mb: 2048
vcpus: 1
hostname: vm1-10
host: NULL
user_data: NULL
reservation_ id: r-xr60h8pj
scheduled_ at: NULL
launched_ at: NULL
terminated_ at: NULL
display_ name: vm1-10 description: vm1
availability_ zone: nova
locked: 0
os_type: NULL
launched_ on: NULL
instance_ type_id: 5
vm_mode: NULL
uuid: 9e191fd1- 7a5d-430f- 973b-625190f525 b9
architectu re: NULL
root_device_ name: NULL
access_ ip_v4: NULL
access_ ip_v6: NULL
config_ drive:
task_state: NULL ephemeral_ device: NULL swap_device: NULL
progress: 0
auto_disk_ config: 1 terminate: 0
disable_ terminate: 0
root_ gb: 20
ephemeral_ gb: 0
cell_name: NULL
node: NULL
deleted: 0
locked_ by: NULL
cleaned: 0 _key_uuid: NULL
select * from instances where id=1964\G;
*******
display_
default_
default_
shutdown_
ephemeral
1 row in set (0.00 sec)
mysql> select * from instance_ info_caches where id='1964'\G; ******* ******* ****** 1. row ******* ******* ******* ****** 7a5d-430f- 973b-625190f525 b9
*******
created_at: 2016-06-21 14:44:17
updated_at: NULL
deleted_at: NULL
id: 1964
network_info: []
instance_uuid: 9e191fd1-
deleted: 0
1 row in set (0.00 sec)