bifrost: missing host vars on deprovision

Bug #2038889 reported by Michal Nasiadka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kayobe
Status tracked in Bobcat
Antelope
Fix Committed
Medium
Michal Nasiadka
Bobcat
Fix Released
Medium
Michal Nasiadka
Yoga
Fix Committed
Medium
Michal Nasiadka
Zed
Fix Committed
Medium
Michal Nasiadka

Bug Description

Sometimes there's a case where Bifrost Ironic database is restored from backup and /etc/kolla/bifrost/inventory is not populated - in such cases deprovision will fail.

TASK [Check the ironic node's initial provision state] **********************************************************************************************************************************
task path: /home/stack/michal/venvs/kayobe/share/kayobe/ansible/overcloud-deprovision.yml:34
Tuesday 10 October 2023 05:53:51 +0000 (0:00:00.086) 0:00:00.086 *******
Using module file /home/stack/michal/venvs/kayobe/lib64/python3.9/site-packages/ansible/modules/command.py
Pipelining is enabled.
<192.168.5.254> ESTABLISH SSH CONNECTION FOR USER: stack
<192.168.5.254> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stack"' -o ConnectTimeout=10 -o 'ControlPath="/home/stack/.ansible/cp/127a331874"' 192.168.5.254 '/bin/sh -c '"'"'/opt/kayobe/venvs/kayobe/bin/python && sleep 0'"'"''
<192.168.5.254> (0, b'\n{"changed": true, "stdout": "", "stderr": "[WARNING]: Unable to parse /etc/bifrost/inventory as an inventory source\\n[WARNING]: No inventory was parsed, only implicit localhost is available\\n[WARNING]: provided hosts list is empty, only localhost is available. Note that\\nthe implicit localhost does not match \'all\'\\n[WARNING]: Could not match supplied host pattern, ignoring: baremetal\\n[WARNING]: Could not match supplied host pattern, ignoring: cct1", "rc": 0, "cmd": ["docker", "exec", "bifrost_deploy", "bash", "-c", " export OS_CLOUD=bifrost && export OS_BAREMETAL_API_VERSION=1.34 && export BIFROST_INVENTORY_SOURCE=ironic && export BIFROST_NODE_NAMES=\\"cct1\\" && ansible baremetal --connection local --inventory /etc/bifrost/inventory/ -e @/etc/bifrost/bifrost.yml -e @/etc/bifrost/dib.yml --limit cct1 -m command -a \\"baremetal node show {{ inventory_hostname }} -f value -c provision_state\\""], "start": "2023-10-10 05:53:52.306489", "end": "2023-10-10 05:53:52.986675", "delta": "0:00:00.680186", "msg": "", "invocation": {"module_args": {"_raw_params": "docker exec bifrost_deploy bash -c \' export OS_CLOUD=bifrost && export OS_BAREMETAL_API_VERSION=1.34 && export BIFROST_INVENTORY_SOURCE=ironic && export BIFROST_NODE_NAMES=\\"cct1\\" && ansible baremetal --connection local --inventory /etc/bifrost/inventory/ -e @/etc/bifrost/bifrost.yml -e @/etc/bifrost/dib.yml --limit cct1 -m command -a \\"baremetal node show {{ inventory_hostname }} -f value -c provision_state\\"\'\\n", "_uses_shell": false, "warn": false, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\n', b"OpenSSH_8.7p1, OpenSSL 3.0.7 1 Nov 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for 'final all' host 192.168.5.254 originally 192.168.5.254\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\r\ndebug3: kex names ok: [curve25519-sha256,<email address hidden>,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 192.168.5.254 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for 'final all' host 192.168.5.254 originally 192.168.5.254\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\r\ndebug3: kex names ok: [curve25519-sha256,<email address hidden>,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/stack/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/stack/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 72020\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
ok: [cct1 -> cct0-seed({{ hostvars[seed_host].ansible_host | default(seed_host) }})] => changed=false
  cmd:
  - docker
  - exec
  - bifrost_deploy
  - bash
  - -c
  - ' export OS_CLOUD=bifrost && export OS_BAREMETAL_API_VERSION=1.34 && export BIFROST_INVENTORY_SOURCE=ironic && export BIFROST_NODE_NAMES="cct1" && ansible baremetal --connection local --inventory /etc/bifrost/inventory/ -e @/etc/bifrost/bifrost.yml -e @/etc/bifrost/dib.yml --limit cct1 -m command -a "baremetal node show {{ inventory_hostname }} -f value -c provision_state"'
  delta: '0:00:00.680186'
  end: '2023-10-10 05:53:52.986675'
  invocation:
    module_args:
      _raw_params: |-
        docker exec bifrost_deploy bash -c ' export OS_CLOUD=bifrost && export OS_BAREMETAL_API_VERSION=1.34 && export BIFROST_INVENTORY_SOURCE=ironic && export BIFROST_NODE_NAMES="cct1" && ansible baremetal --connection local --inventory /etc/bifrost/inventory/ -e @/etc/bifrost/bifrost.yml -e @/etc/bifrost/dib.yml --limit cct1 -m command -a "baremetal node show {{ inventory_hostname }} -f value -c provision_state"'
      _uses_shell: false
      argv: null
      chdir: null
      creates: null
      executable: null
      removes: null
      stdin: null
      stdin_add_newline: true
      strip_empty_ends: true
      warn: false
  msg: ''
  rc: 0
  start: '2023-10-10 05:53:52.306489'
  stderr: |-

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kayobe/+/897751

Changed in kayobe:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (master)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/897751
Committed: https://opendev.org/openstack/kayobe/commit/672d6fa24a978009588e66dd19c623ee244a8f8e
Submitter: "Zuul (22348)"
Branch: master

commit 672d6fa24a978009588e66dd19c623ee244a8f8e
Author: Michal Nasiadka <email address hidden>
Date: Tue Oct 10 08:03:47 2023 +0200

    bifrost: Populate bifrost host vars on deprovision

    Closes-Bug: #2038889
    Change-Id: I4609494d009c6e1f97d833a9a11529d107b3216f

Changed in kayobe:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kayobe/+/898561

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kayobe/+/898562

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kayobe/+/898563

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/898562
Committed: https://opendev.org/openstack/kayobe/commit/a1a1f8d3700fa7d274edb47a7e9467c17b923c88
Submitter: "Zuul (22348)"
Branch: stable/zed

commit a1a1f8d3700fa7d274edb47a7e9467c17b923c88
Author: Michal Nasiadka <email address hidden>
Date: Tue Oct 10 08:03:47 2023 +0200

    bifrost: Populate bifrost host vars on deprovision

    Closes-Bug: #2038889
    Change-Id: I4609494d009c6e1f97d833a9a11529d107b3216f
    (cherry picked from commit 672d6fa24a978009588e66dd19c623ee244a8f8e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/898563
Committed: https://opendev.org/openstack/kayobe/commit/f609ae27a073aac351e703751b7731df9230cc20
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f609ae27a073aac351e703751b7731df9230cc20
Author: Michal Nasiadka <email address hidden>
Date: Tue Oct 10 08:03:47 2023 +0200

    bifrost: Populate bifrost host vars on deprovision

    Closes-Bug: #2038889
    Change-Id: I4609494d009c6e1f97d833a9a11529d107b3216f
    (cherry picked from commit 672d6fa24a978009588e66dd19c623ee244a8f8e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/898561
Committed: https://opendev.org/openstack/kayobe/commit/7825b28750dbe3e3a9096448ddc4662be81820dd
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 7825b28750dbe3e3a9096448ddc4662be81820dd
Author: Michal Nasiadka <email address hidden>
Date: Tue Oct 10 08:03:47 2023 +0200

    bifrost: Populate bifrost host vars on deprovision

    Closes-Bug: #2038889
    Change-Id: I4609494d009c6e1f97d833a9a11529d107b3216f
    (cherry picked from commit 672d6fa24a978009588e66dd19c623ee244a8f8e)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.