| 2017-05-22 03:29:31 |
Zhenyu Zheng |
bug |
|
|
added bug |
| 2017-05-22 03:29:38 |
Zhenyu Zheng |
nova: assignee |
|
Zhenyu Zheng (zhengzhenyu) |
|
| 2017-05-22 03:39:00 |
Zhenyu Zheng |
description |
TBA |
Hypervisor statistics could be incorrect:
When we killed a nova-compute service and deleted the service from nova DB, and then
start the nova-compute service again, the result of (nova hypervisor-stats) will be
incorrect;
How to reproduce: |
|
| 2017-05-22 03:50:02 |
Zhenyu Zheng |
description |
Hypervisor statistics could be incorrect:
When we killed a nova-compute service and deleted the service from nova DB, and then
start the nova-compute service again, the result of (nova hypervisor-stats) will be
incorrect;
How to reproduce: |
Hypervisor statistics could be incorrect:
When we killed a nova-compute service and deleted the service from nova DB, and then
start the nova-compute service again, the result of (nova hypervisor-stats) will be
incorrect;
How to reproduce:
Step1. Check the correct statistics before we do anything:
root@SZX1000291919:/opt/stack/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 14 |
| free_disk_gb | 34 |
| free_ram_mb | 6936 |
| local_gb | 35 |
| local_gb_used | 1 |
| memory_mb | 7960 |
| memory_mb_used | 1024 |
| running_vms | 1 |
| vcpus | 8 |
| vcpus_used | 1 |
+----------------------+-------+
Step2. Kill the compute service:
root@SZX1000291919:/var/log/nova# ps -ef | grep nova-com
root 120419 120411 0 11:06 pts/27 00:00:00 sg libvirtd /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root 120420 120419 0 11:06 pts/27 00:00:07 /usr/bin/python /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root@SZX1000291919:/var/log/nova# kill -9 120419
root@SZX1000291919:/var/log/nova# /usr/local/bin/stack: line 19: 120419 Killed sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log' > /dev/null 2>&1
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:37.000000 | - |
| 8 | nova-compute | SZX1000291919 | nova | enabled | down | 2017-05-22T03:23:38.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step3. Delete the service from DB:
root@SZX1000291919:/var/log/nova# nova service-delete 8
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:17.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step4. Start the compute service again:
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:55.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
| 10 | nova-compute | SZX1000291919 | nova | enabled | up | 2017-05-22T03:48:57.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step5. Check again the hyervisor statistics, the result is incorrect:
root@SZX1000291919:/var/log/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 28 |
| free_disk_gb | 68 |
| free_ram_mb | 13872 |
| local_gb | 70 |
| local_gb_used | 2 |
| memory_mb | 15920 |
| memory_mb_used | 2048 |
| running_vms | 2 |
| vcpus | 16 |
| vcpus_used | 2 |
+----------------------+-------+ |
|
| 2017-05-22 03:53:17 |
Zhenyu Zheng |
description |
Hypervisor statistics could be incorrect:
When we killed a nova-compute service and deleted the service from nova DB, and then
start the nova-compute service again, the result of (nova hypervisor-stats) will be
incorrect;
How to reproduce:
Step1. Check the correct statistics before we do anything:
root@SZX1000291919:/opt/stack/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 14 |
| free_disk_gb | 34 |
| free_ram_mb | 6936 |
| local_gb | 35 |
| local_gb_used | 1 |
| memory_mb | 7960 |
| memory_mb_used | 1024 |
| running_vms | 1 |
| vcpus | 8 |
| vcpus_used | 1 |
+----------------------+-------+
Step2. Kill the compute service:
root@SZX1000291919:/var/log/nova# ps -ef | grep nova-com
root 120419 120411 0 11:06 pts/27 00:00:00 sg libvirtd /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root 120420 120419 0 11:06 pts/27 00:00:07 /usr/bin/python /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root@SZX1000291919:/var/log/nova# kill -9 120419
root@SZX1000291919:/var/log/nova# /usr/local/bin/stack: line 19: 120419 Killed sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log' > /dev/null 2>&1
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:37.000000 | - |
| 8 | nova-compute | SZX1000291919 | nova | enabled | down | 2017-05-22T03:23:38.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step3. Delete the service from DB:
root@SZX1000291919:/var/log/nova# nova service-delete 8
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:17.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step4. Start the compute service again:
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:55.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
| 10 | nova-compute | SZX1000291919 | nova | enabled | up | 2017-05-22T03:48:57.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step5. Check again the hyervisor statistics, the result is incorrect:
root@SZX1000291919:/var/log/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 28 |
| free_disk_gb | 68 |
| free_ram_mb | 13872 |
| local_gb | 70 |
| local_gb_used | 2 |
| memory_mb | 15920 |
| memory_mb_used | 2048 |
| running_vms | 2 |
| vcpus | 16 |
| vcpus_used | 2 |
+----------------------+-------+ |
Hypervisor statistics could be incorrect:
When we killed a nova-compute service and deleted the service from nova DB, and then
start the nova-compute service again, the result of Hypervisor/statistics API (nova hypervisor-stats) will be
incorrect;
How to reproduce:
Step1. Check the correct statistics before we do anything:
root@SZX1000291919:/opt/stack/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 14 |
| free_disk_gb | 34 |
| free_ram_mb | 6936 |
| local_gb | 35 |
| local_gb_used | 1 |
| memory_mb | 7960 |
| memory_mb_used | 1024 |
| running_vms | 1 |
| vcpus | 8 |
| vcpus_used | 1 |
+----------------------+-------+
Step2. Kill the compute service:
root@SZX1000291919:/var/log/nova# ps -ef | grep nova-com
root 120419 120411 0 11:06 pts/27 00:00:00 sg libvirtd /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root 120420 120419 0 11:06 pts/27 00:00:07 /usr/bin/python /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root@SZX1000291919:/var/log/nova# kill -9 120419
root@SZX1000291919:/var/log/nova# /usr/local/bin/stack: line 19: 120419 Killed sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log' > /dev/null 2>&1
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:37.000000 | - |
| 8 | nova-compute | SZX1000291919 | nova | enabled | down | 2017-05-22T03:23:38.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step3. Delete the service from DB:
root@SZX1000291919:/var/log/nova# nova service-delete 8
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:17.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step4. Start the compute service again:
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:55.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
| 10 | nova-compute | SZX1000291919 | nova | enabled | up | 2017-05-22T03:48:57.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step5. Check again the hyervisor statistics, the result is incorrect:
root@SZX1000291919:/var/log/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 28 |
| free_disk_gb | 68 |
| free_ram_mb | 13872 |
| local_gb | 70 |
| local_gb_used | 2 |
| memory_mb | 15920 |
| memory_mb_used | 2048 |
| running_vms | 2 |
| vcpus | 16 |
| vcpus_used | 2 |
+----------------------+-------+ |
|
| 2017-05-23 12:50:56 |
OpenStack Infra |
nova: status |
New |
In Progress |
|
| 2017-05-25 02:12:39 |
Matt Riedemann |
nova: importance |
Undecided |
Low |
|
| 2017-05-25 14:45:28 |
Matt Riedemann |
nominated for series |
|
nova/ocata |
|
| 2017-05-25 14:45:28 |
Matt Riedemann |
bug task added |
|
nova/ocata |
|
| 2017-05-25 14:55:37 |
Matt Riedemann |
nova/ocata: status |
New |
Confirmed |
|
| 2017-05-25 14:55:39 |
Matt Riedemann |
nova/ocata: importance |
Undecided |
Low |
|
| 2017-05-25 16:05:15 |
OpenStack Infra |
nova: assignee |
Zhenyu Zheng (zhengzhenyu) |
Matt Riedemann (mriedem) |
|
| 2017-05-26 19:04:46 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2017-05-26 19:48:02 |
OpenStack Infra |
nova/ocata: status |
Confirmed |
In Progress |
|
| 2017-05-26 19:48:02 |
OpenStack Infra |
nova/ocata: assignee |
|
Matt Riedemann (mriedem) |
|
| 2017-05-26 19:59:37 |
Matt Riedemann |
nominated for series |
|
nova/newton |
|
| 2017-05-26 19:59:37 |
Matt Riedemann |
bug task added |
|
nova/newton |
|
| 2017-05-26 19:59:45 |
Matt Riedemann |
nova/newton: assignee |
|
Matt Riedemann (mriedem) |
|
| 2017-05-26 19:59:48 |
Matt Riedemann |
nova/newton: importance |
Undecided |
Low |
|
| 2017-05-26 19:59:51 |
Matt Riedemann |
nova/newton: status |
New |
In Progress |
|
| 2017-05-26 19:59:59 |
Matt Riedemann |
nova: assignee |
Matt Riedemann (mriedem) |
Zhenyu Zheng (zhengzhenyu) |
|
| 2017-05-27 02:59:52 |
OpenStack Infra |
nova/ocata: status |
In Progress |
Fix Committed |
|
| 2017-08-21 18:49:16 |
OpenStack Infra |
nova/newton: status |
In Progress |
Fix Committed |
|
| 2017-11-15 13:42:15 |
Edward Hope-Morley |
bug task added |
|
nova (Ubuntu) |
|
| 2017-11-15 13:42:27 |
Edward Hope-Morley |
bug task added |
|
cloud-archive |
|
| 2017-11-15 13:42:38 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/mitaka |
|
| 2017-11-15 13:42:49 |
Edward Hope-Morley |
nova (Ubuntu): status |
New |
Fix Released |
|
| 2017-11-15 13:43:40 |
Edward Hope-Morley |
cloud-archive: status |
New |
Fix Committed |
|
| 2017-11-15 13:43:45 |
Edward Hope-Morley |
cloud-archive: status |
Fix Committed |
Fix Released |
|
| 2017-11-15 13:45:00 |
Edward Hope-Morley |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-11-15 13:45:19 |
Edward Hope-Morley |
tags |
|
sts sts-sru-needed |
|
| 2017-11-15 13:47:49 |
Eric Desrochers |
bug task added |
|
nova (Ubuntu Xenial) |
|
| 2017-11-15 15:04:32 |
Edward Hope-Morley |
description |
Hypervisor statistics could be incorrect:
When we killed a nova-compute service and deleted the service from nova DB, and then
start the nova-compute service again, the result of Hypervisor/statistics API (nova hypervisor-stats) will be
incorrect;
How to reproduce:
Step1. Check the correct statistics before we do anything:
root@SZX1000291919:/opt/stack/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 14 |
| free_disk_gb | 34 |
| free_ram_mb | 6936 |
| local_gb | 35 |
| local_gb_used | 1 |
| memory_mb | 7960 |
| memory_mb_used | 1024 |
| running_vms | 1 |
| vcpus | 8 |
| vcpus_used | 1 |
+----------------------+-------+
Step2. Kill the compute service:
root@SZX1000291919:/var/log/nova# ps -ef | grep nova-com
root 120419 120411 0 11:06 pts/27 00:00:00 sg libvirtd /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root 120420 120419 0 11:06 pts/27 00:00:07 /usr/bin/python /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root@SZX1000291919:/var/log/nova# kill -9 120419
root@SZX1000291919:/var/log/nova# /usr/local/bin/stack: line 19: 120419 Killed sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log' > /dev/null 2>&1
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:37.000000 | - |
| 8 | nova-compute | SZX1000291919 | nova | enabled | down | 2017-05-22T03:23:38.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step3. Delete the service from DB:
root@SZX1000291919:/var/log/nova# nova service-delete 8
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:17.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step4. Start the compute service again:
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:55.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
| 10 | nova-compute | SZX1000291919 | nova | enabled | up | 2017-05-22T03:48:57.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step5. Check again the hyervisor statistics, the result is incorrect:
root@SZX1000291919:/var/log/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 28 |
| free_disk_gb | 68 |
| free_ram_mb | 13872 |
| local_gb | 70 |
| local_gb_used | 2 |
| memory_mb | 15920 |
| memory_mb_used | 2048 |
| running_vms | 2 |
| vcpus | 16 |
| vcpus_used | 2 |
+----------------------+-------+ |
[Impact]
If you deploy a nova-compute service to a node, delete that service (via the api), then deploy a new nova-compute service to that same node i.e. same hostname, the database will now have two service records one marked as deleted and the other not. So far so good until you do an 'openstack hypervisor stats show' at which point the api will aggregate the resource counts from both services. This has been fixed and backported all the way down to Newton so the problem still exists on Mitaka. I assume the reason why the patch was not backported to Mitaka is that the code in nova.db.sqlalchemy.apy.compute_node_statistics() changed quite a bit. However it only requires a one line change in the old code (that does the same thing as the new code) to fix this issue.
[Test Case]
* Deploy Mitaka with bundle http://pastebin.ubuntu.com/25968008/
* Do 'openstack hypervisor stats show' and verify that count is 3
* Do 'juju remove-unit nova-compute/2' to delete a compute service but not its physical host
* Do 'openstack compute service delete <id>' to delete a compute service we just removed (choosing correct id)
* Do 'openstack hypervisor stats show' and verify that count is 2
* Do juju add-unit nova-compute --to <machine id of deleted unit>
* Do 'openstack hypervisor stats show' and verify that count is 3 (not 4 as it would be before fix)
[Regression Potential]
None anticipated other than for clients that were interpreting invalid counts as correct.
[Other Info]
===========================================================================
Hypervisor statistics could be incorrect:
When we killed a nova-compute service and deleted the service from nova DB, and then
start the nova-compute service again, the result of Hypervisor/statistics API (nova hypervisor-stats) will be
incorrect;
How to reproduce:
Step1. Check the correct statistics before we do anything:
root@SZX1000291919:/opt/stack/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 14 |
| free_disk_gb | 34 |
| free_ram_mb | 6936 |
| local_gb | 35 |
| local_gb_used | 1 |
| memory_mb | 7960 |
| memory_mb_used | 1024 |
| running_vms | 1 |
| vcpus | 8 |
| vcpus_used | 1 |
+----------------------+-------+
Step2. Kill the compute service:
root@SZX1000291919:/var/log/nova# ps -ef | grep nova-com
root 120419 120411 0 11:06 pts/27 00:00:00 sg libvirtd /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root 120420 120419 0 11:06 pts/27 00:00:07 /usr/bin/python /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root@SZX1000291919:/var/log/nova# kill -9 120419
root@SZX1000291919:/var/log/nova# /usr/local/bin/stack: line 19: 120419 Killed sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log' > /dev/null 2>&1
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:37.000000 | - |
| 8 | nova-compute | SZX1000291919 | nova | enabled | down | 2017-05-22T03:23:38.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step3. Delete the service from DB:
root@SZX1000291919:/var/log/nova# nova service-delete 8
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:17.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step4. Start the compute service again:
root@SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:55.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
| 10 | nova-compute | SZX1000291919 | nova | enabled | up | 2017-05-22T03:48:57.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step5. Check again the hyervisor statistics, the result is incorrect:
root@SZX1000291919:/var/log/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 28 |
| free_disk_gb | 68 |
| free_ram_mb | 13872 |
| local_gb | 70 |
| local_gb_used | 2 |
| memory_mb | 15920 |
| memory_mb_used | 2048 |
| running_vms | 2 |
| vcpus | 16 |
| vcpus_used | 2 |
+----------------------+-------+ |
|
| 2017-11-15 15:05:44 |
Edward Hope-Morley |
attachment added |
|
lp1692397-xenial-mitaka.debdiff https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1692397/+attachment/5009496/+files/lp1692397-xenial-mitaka.debdiff |
|
| 2017-11-20 14:47:09 |
Edward Hope-Morley |
tags |
sts sts-sru-needed |
sts sts-sponsor sts-sru-needed |
|
| 2017-11-20 15:33:09 |
Corey Bryant |
bug task added |
|
cloud-archive/mitaka |
|
| 2017-11-20 15:36:02 |
Corey Bryant |
cloud-archive/mitaka: status |
New |
Triaged |
|
| 2017-11-20 15:36:06 |
Corey Bryant |
cloud-archive/mitaka: importance |
Undecided |
Low |
|
| 2017-11-20 15:36:36 |
Corey Bryant |
nova (Ubuntu Xenial): status |
New |
Triaged |
|
| 2017-11-20 15:36:40 |
Corey Bryant |
nova (Ubuntu Xenial): importance |
Undecided |
Low |
|
| 2017-11-20 18:30:25 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-12-09 00:46:31 |
Steve Langasek |
nova (Ubuntu Xenial): status |
Triaged |
Fix Committed |
|
| 2017-12-09 00:46:37 |
Steve Langasek |
bug |
|
|
added subscriber SRU Verification |
| 2017-12-09 00:46:40 |
Steve Langasek |
tags |
sts sts-sponsor sts-sru-needed |
sts sts-sponsor sts-sru-needed verification-needed verification-needed-xenial |
|
| 2017-12-12 11:21:39 |
Edward Hope-Morley |
tags |
sts sts-sponsor sts-sru-needed verification-needed verification-needed-xenial |
sts sts-sponsor sts-sru-needed verification-done-xenial verification-needed |
|
| 2017-12-12 11:21:49 |
Edward Hope-Morley |
tags |
sts sts-sponsor sts-sru-needed verification-done-xenial verification-needed |
sts sts-sponsor sts-sru-needed verification-done verification-done-xenial |
|
| 2017-12-18 18:24:26 |
Launchpad Janitor |
nova (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-12-18 18:24:42 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-12-29 14:00:08 |
Hua Zhang |
attachment added |
|
trusty-mitaka.debdiff https://bugs.launchpad.net/nova/+bug/1692397/+attachment/5028727/+files/trusty-mitaka.debdiff |
|
| 2018-01-02 15:25:03 |
Corey Bryant |
cloud-archive/mitaka: status |
Triaged |
Fix Committed |
|
| 2018-01-02 15:25:05 |
Corey Bryant |
tags |
sts sts-sponsor sts-sru-needed verification-done verification-done-xenial |
sts sts-sponsor sts-sru-needed verification-done verification-done-xenial verification-mitaka-needed |
|
| 2018-01-04 11:49:52 |
Edward Hope-Morley |
tags |
sts sts-sponsor sts-sru-needed verification-done verification-done-xenial verification-mitaka-needed |
sts sts-sponsor sts-sru-done verification-done verification-done-xenial verification-mitaka-done |
|
| 2018-01-04 11:50:06 |
Edward Hope-Morley |
tags |
sts sts-sponsor sts-sru-done verification-done verification-done-xenial verification-mitaka-done |
sts sts-sru-done verification-done verification-done-xenial verification-mitaka-done |
|
| 2018-01-10 19:59:40 |
Corey Bryant |
cloud-archive/mitaka: status |
Fix Committed |
Fix Released |
|
