| 2019-11-19 21:04:34 |
David Coronel |
bug |
|
|
added bug |
| 2019-11-19 21:27:09 |
David Coronel |
description |
When trying to launch an instance in OpenStack Queens on Ubuntu 18.04 with the new kernels, this error happens:
Error: Failed to perform requested operation on instance "david", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance bf8dc8b8-37dd-43fa-ace0-90fe18c1d63b. Last exception: the CPU is incompatible with host CPU: Host CPU does not provide required features: hle, rtm].
This seems to be caused by the new kernels disabling the tsx cpu feature as per https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/TAA_MCEPSC_i915
Disabling tsx also disables hle and rtm, and /usr/share/libvirt/cpu_map.xml has hle and rtm configured for many cpu models:
ubuntu@cloud3:~$ grep -e "model name" -e hle -e rtm -e tsx
[...]
<model name='Haswell'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Haswell-IBRS'>
<feature name='hle'/>
<feature name='rtm'/>
[...]
<model name='Broadwell'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Broadwell-IBRS'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Skylake-Client'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Skylake-Client-IBRS'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Skylake-Server'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Skylake-Server-IBRS'>
<feature name='hle'/>
<feature name='rtm'/>
[...]
Additional info:
I see this issue with the following kernel and libvirt versions:
Linux cloud3 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@cloud3:~$ dpkg -l | grep -e libvirt -e nova
ii libvirt-clients 4.0.0-1ubuntu8.13 amd64 Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8.13 amd64 Virtualization daemon
ii libvirt-daemon-driver-storage-rbd 4.0.0-1ubuntu8.13 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 4.0.0-1ubuntu8.13 amd64 Libvirt daemon configuration files
ii libvirt0:amd64 4.0.0-1ubuntu8.13 amd64 library for interfacing with different virtualization systems
ii nova-common 2:17.0.11-0ubuntu1 all OpenStack Compute - common files
ii nova-compute 2:17.0.11-0ubuntu1 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:17.0.11-0ubuntu1 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:17.0.11-0ubuntu1 all OpenStack Compute - compute node libvirt support
ii python-libvirt 4.0.0-1 amd64 libvirt Python bindings
ii python-nova 2:17.0.11-0ubuntu1 all OpenStack Compute Python libraries
ii python-novaclient 2:9.1.1-0ubuntu1 all client library for OpenStack Compute API - Python 2.7
ubuntu@cloud3:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic |
When trying to launch an instance in OpenStack Queens on Ubuntu 18.04 with the new kernels, this error happens:
Error: Failed to perform requested operation on instance "david", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance bf8dc8b8-37dd-43fa-ace0-90fe18c1d63b. Last exception: the CPU is incompatible with host CPU: Host CPU does not provide required features: hle, rtm].
This seems to be caused by the new kernels disabling the tsx cpu feature as per https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/TAA_MCEPSC_i915
Disabling tsx also disables hle and rtm, and /usr/share/libvirt/cpu_map.xml has hle and rtm configured for many cpu models:
ubuntu@cloud3:~$ grep -e "model name" -e hle -e rtm -e tsx
[...]
<model name='Haswell'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Haswell-IBRS'>
<feature name='hle'/>
<feature name='rtm'/>
[...]
<model name='Broadwell'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Broadwell-IBRS'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Skylake-Client'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Skylake-Client-IBRS'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Skylake-Server'>
<feature name='hle'/>
<feature name='rtm'/>
<model name='Skylake-Server-IBRS'>
<feature name='hle'/>
<feature name='rtm'/>
[...]
This only happens when configuring cpu_mode and cpu_model in /etc/nova/nova.conf:
[libvirt]
cpu_mode = custom
cpu_model = Skylake-Server-IBRS
In my case, this was done by setting the cpu-mode and cpu-model nova-compute charm options.
[Additional info]
I see this issue with the following kernel and libvirt versions:
Linux cloud3 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@cloud3:~$ dpkg -l | grep -e libvirt -e nova
ii libvirt-clients 4.0.0-1ubuntu8.13 amd64 Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8.13 amd64 Virtualization daemon
ii libvirt-daemon-driver-storage-rbd 4.0.0-1ubuntu8.13 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 4.0.0-1ubuntu8.13 amd64 Libvirt daemon configuration files
ii libvirt0:amd64 4.0.0-1ubuntu8.13 amd64 library for interfacing with different virtualization systems
ii nova-common 2:17.0.11-0ubuntu1 all OpenStack Compute - common files
ii nova-compute 2:17.0.11-0ubuntu1 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:17.0.11-0ubuntu1 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:17.0.11-0ubuntu1 all OpenStack Compute - compute node libvirt support
ii python-libvirt 4.0.0-1 amd64 libvirt Python bindings
ii python-nova 2:17.0.11-0ubuntu1 all OpenStack Compute Python libraries
ii python-novaclient 2:9.1.1-0ubuntu1 all client library for OpenStack Compute API - Python 2.7
ubuntu@cloud3:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic
[Workaround]
A workaround is to remove the cpu_mode and cpu_model lines in the libvirt section of /etc/nova/nova.conf.
This can be done with juju like this:
juju config nova-compute-kvm --reset cpu-model
juju config nova-compute-kvm --reset cpu-mode
Apparently another workaround would be to re-enable the tsx cpu feature on the host with tsx=yes on the boot command line, but I have not tested that workaround. |
|
| 2019-11-19 21:32:01 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2019-11-19 22:40:06 |
Steve Beattie |
bug |
|
|
added subscriber Steve Beattie |
| 2019-11-20 06:53:24 |
Christian Ehrhardt |
libvirt (Ubuntu): assignee |
|
Ubuntu Security Team (ubuntu-security) |
|
| 2019-11-20 06:53:26 |
Christian Ehrhardt |
bug |
|
|
added subscriber Christian Ehrhardt |
| 2019-11-20 06:53:31 |
Christian Ehrhardt |
libvirt (Ubuntu): importance |
Undecided |
High |
|
| 2019-11-20 06:53:34 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
New |
Confirmed |
|
| 2019-11-20 07:26:52 |
Christian Ehrhardt |
bug |
|
|
added subscriber Ubuntu Cloud Archive Team |
| 2019-11-22 18:07:25 |
David Coronel |
bug |
|
|
added subscriber Canonical Field Medium |
| 2019-11-25 14:52:06 |
Dan Streetman |
bug |
|
|
added subscriber Dan Streetman |
| 2019-11-27 15:01:33 |
Corey Bryant |
bug |
|
|
added subscriber Corey Bryant |
| 2019-12-06 13:56:26 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2019-12-09 05:52:28 |
Yoshi Kadokawa |
bug |
|
|
added subscriber Yoshi Kadokawa |
| 2019-12-09 07:50:27 |
Christian Ehrhardt |
cve linked |
|
2019-11135 |
|
| 2019-12-09 10:43:12 |
Janåke Rönnblom |
bug |
|
|
added subscriber Janåke Rönnblom |
| 2019-12-09 14:33:57 |
Ove Risberg |
bug |
|
|
added subscriber Ove Risberg |
| 2020-02-26 13:57:47 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
Confirmed |
Won't Fix |
|
| 2020-03-23 10:28:08 |
Christian Ehrhardt |
libvirt (Ubuntu): assignee |
Ubuntu Security Team (ubuntu-security) |
Christian Ehrhardt (paelzer) |
|
| 2020-03-23 10:28:10 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
Won't Fix |
Triaged |
|
| 2020-03-23 11:47:23 |
Christian Ehrhardt |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/380942 |
|
| 2020-03-23 14:54:48 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
Triaged |
Fix Committed |
|
| 2020-03-26 22:36:52 |
Launchpad Janitor |
libvirt (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2020-03-27 13:43:33 |
Christian Ehrhardt |
bug task added |
|
qemu (Ubuntu) |
|
| 2020-03-27 13:43:40 |
Christian Ehrhardt |
qemu (Ubuntu): status |
New |
Fix Released |
|
| 2020-03-27 13:43:45 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Eoan |
|
| 2020-03-27 13:43:45 |
Christian Ehrhardt |
bug task added |
|
qemu (Ubuntu Eoan) |
|
| 2020-03-27 13:43:45 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Eoan) |
|
| 2020-03-27 13:43:45 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Bionic |
|
| 2020-03-27 13:43:45 |
Christian Ehrhardt |
bug task added |
|
qemu (Ubuntu Bionic) |
|
| 2020-03-27 13:43:45 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Bionic) |
|
| 2020-03-27 13:44:02 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): assignee |
|
Ubuntu Security Team (ubuntu-security) |
|
| 2020-03-27 13:44:06 |
Christian Ehrhardt |
libvirt (Ubuntu Eoan): assignee |
|
Ubuntu Security Team (ubuntu-security) |
|
| 2020-03-27 13:44:11 |
Christian Ehrhardt |
qemu (Ubuntu Bionic): assignee |
|
Ubuntu Security Team (ubuntu-security) |
|
| 2020-03-27 13:44:16 |
Christian Ehrhardt |
qemu (Ubuntu Eoan): assignee |
|
Ubuntu Security Team (ubuntu-security) |
|
| 2020-08-18 17:02:32 |
Brian Murray |
libvirt (Ubuntu Eoan): status |
New |
Won't Fix |
|
| 2020-08-18 17:02:37 |
Brian Murray |
qemu (Ubuntu Eoan): status |
New |
Won't Fix |
|
| 2020-10-08 09:24:33 |
Launchpad Janitor |
libvirt (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2020-10-08 09:24:33 |
Launchpad Janitor |
qemu (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2020-10-08 09:24:40 |
Ian Johnson |
bug |
|
|
added subscriber Ian Johnson |
