nova handler "Restart nova-libvirt container" fails

Bug #1842913 reported by Benjamin
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla-ansible
Invalid
Undecided
Unassigned
Stein
Fix Committed
Medium
Mark Goddard

Bug Description

Kolla version: kolla-ansible==8.0.0
Ansible version: 2.9.0.dev0 894d19b1088d8d4872ca0dd2b9588f196b9a34ea

Execution host runs Debian 10, target hosts are CentOS 7.

I have a recently created multinode setup with 1 pure controller node and 1 controller+compute node. Now I wanted to enable support for provider networks (enable_neutron_provider_networks) and ran 'kolla-ansible -i multinode deploy' once more to get this deployed.

For my pure controller node the playbook completes (probably because it needs no libvirt), however, for my compute node it doesn't. There is one VM running on this compute node.

Something in the handler's bool chain goes wrong:
RUNNING HANDLER [nova : Restart nova-libvirt container] ***********************************************************************************************************************************************************
Thursday 05 September 2019 12:34:50 +0200 (0:00:00.180) 0:05:11.410 ****
fatal: [redacted]: FAILED! => {
    "msg": "The conditional check 'config_json.changed | bool or nova_libvirt_confs.changed | bool or nova_libvirt_container.changed | bool or ( ceph_conf is not none and ceph_conf.changed | bool ) or ( nova_ceph_keyring is defined and nova_ceph_keyring.changed | bool ) or ( libvirt_secrets_xml is defined and libvirt_secrets_xml.changed | bool ) or ( libvirt_secrets_key is defined and libvirt_secrets_key.changed | bool )' failed. The error was: error while evaluating conditional (config_json.changed | bool or nova_libvirt_confs.changed | bool or nova_libvirt_container.changed | bool or ( ceph_conf is not none and ceph_conf.changed | bool ) or ( nova_ceph_keyring is defined and nova_ceph_keyring.changed | bool ) or ( libvirt_secrets_xml is defined and libvirt_secrets_xml.changed | bool ) or ( libvirt_secrets_key is defined and libvirt_secrets_key.changed | bool )): 'str object' has no attribute 'changed'\n\nThe error appears to be in '/home/redacted/projects/kolla/venv/share/kolla-ansible/ansible/roles/nova/handlers/main.yml': line 52, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Restart nova-libvirt container\n ^ here\n"
}

Hard to figure out which of these is the culprit. Any hints how to further troubleshoot/workaround this?

Benjamin (tumbl3w33d)
description: updated
Benjamin (tumbl3w33d)
description: updated
Revision history for this message
Benjamin (tumbl3w33d) wrote :

I worked around it by destroying everything and re-deploying from scratch so I can't try possible suggestions anymore. Would still be cool to avoid this for others.

Revision history for this message
Mark Goddard (mgoddard) wrote :

For future reference, the best way to debug an issue like this would be to add debug tasks to display the various variables that are referenced here.

https://docs.ansible.com/ansible/latest/modules/debug_module.html

Mark Goddard (mgoddard)
Changed in kolla-ansible:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for kolla-ansible because there has been no activity for 60 days.]

Changed in kolla-ansible:
status: Incomplete → Expired
Emanuel Evans (shosti)
information type: Public → Public Security
information type: Public Security → Public
Revision history for this message
Emanuel Evans (shosti) wrote :

This happens for me as well on 8.0.1 after restarting the node (it's AIO). I narrowed it down to `ceph_conf` in https://github.com/openstack/kolla-ansible/blob/8.0.1/ansible/roles/nova/handlers/main.yml#L81 (seems to be a string instead of `none`).

Revision history for this message
Mark Goddard (mgoddard) wrote :

I have reproduced something similar:

---
- hosts: localhost
  tasks:
    - command: 'true'
      when: none_var is not none and none_var.changed
      vars:
        none_var: "{{ {} if foo is defined else none }}"

ansible-playbook t.yml

PLAY [localhost] *******************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [localhost]

TASK [command] *********************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'none_var is not none and none_var.changed' failed. The error was: error while evaluating conditional (none_var is not none and none_var.changed): 'unicode object' has no attribute 'changed'\n\nThe error appears to be in '/tmp/t.yml': line 4, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - command: 'true'\n ^ here\n"}

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost

If I change it to this, it works:

---
- hosts: localhost
  tasks:
    - command: 'true'
      when: none_var is mapping and none_var.changed
      vars:
        none_var: "{{ foo if foo is defined else '' }}"

Changed in kolla-ansible:
importance: Undecided → Medium
status: Expired → Triaged
status: Triaged → Invalid
importance: Medium → Undecided
Mark Goddard (mgoddard)
no longer affects: kolla-ansible/rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/698680

Revision history for this message
Mark Goddard (mgoddard) wrote :

This only affects the Stein release since https://review.opendev.org/#/c/636122/ was not backported.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

It looks like it was included in stein, not needing backporting...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/698680
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=067b55505df17995538564c765cb6676218466e1
Submitter: Zuul
Branch: stable/stein

commit 067b55505df17995538564c765cb6676218466e1
Author: Mark Goddard <email address hidden>
Date: Thu Dec 12 10:09:02 2019 +0000

    Fix nova and cinder handlers with no ceph

    Stein only

    If we deploy nova or cinder without internal or external ceph
    configured, and there are no config or container changes, then the
    following task may fail:

    RUNNING HANDLER [nova : Restart nova-libvirt container]

    The error is in evaluating the conditionals:

    'str object' has no attribute 'changed'

    Thank you to Emanuel Evans (shosti) for determining that ceph_conf is the issue
    here. The problem is that if ceph_confs is undefined, then we set ceph_conf to
    {{ none }}, however due to the 'magic' of Jinja this evaluates to a string. The
    check for none therefore fails, and we look up an attribute 'changed' on a
    string.

    This change fixes the issue by changing the condition from 'is not none' to 'is
    mapping'. We also change the value to an empty string, to make it clear that it
    does not end up as None.

    Change-Id: I240ecd6b9a4ef09ef9cbc3f58c9c0169c73c5e58
    Closes-Bug: #1842913

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 8.1.0

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.