tripleo

on centos stream 9 neutron dhcp is not working

Bug #1944522 reported by Michele Baldessari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Michele Baldessari
tripleo xena-3

Bug Description

On CS9 baremetal deployment fails. The dhcp log on the undercloud will show:
+ DOCKER_HOST=
+ ARGS='--no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/host --addn-hosts=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/opts --dhcp-leasefile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-eaf4f552-0023-4e4a-9b6e-b5f3727e647e,192.168.24.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=256 --conf-file=/dev/null --domain=home.arpa'
++ ip netns identify
+ NETNS=qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49
+ NAME=neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49
+ CLI='nsenter --net=/run/netns/qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 --preserve-credentials -m -t 1 podman'
+ LOGGING='--log-driver k8s-file --log-opt path=/var/log/containers/stdouts/neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49.log'
+ CMD='/usr/sbin/dnsmasq -k'
++ awk '{print $1}'
++ nsenter --net=/run/netns/qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 --preserve-credentials -m -t 1 podman ps -a --filter name=neutron-dnsmasq- --format '{{.ID}}:{{.Names}}:{{.Status}}'
+ LIST=5a6b8e987ae7:neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49:Created
++ printf '%s\n' 5a6b8e987ae7:neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49:Created
++ grep -E ':(Exited|Created)'
+ ORPHANTS=5a6b8e987ae7:neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49:Created
+ '[' -n 5a6b8e987ae7:neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49:Created ']'
++ printf '%s\n' 5a6b8e987ae7:neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49:Created
++ awk -F: '{print $1}'
+ for orphant in $(printf "%s\n" "${ORPHANTS}" | awk -F':' '{print $1}')
+ echo 'Removing orphaned container 5a6b8e987ae7'
+ nsenter --net=/run/netns/qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 --preserve-credentials -m -t 1 podman stop 5a6b8e987ae7
Error: can only stop created or running containers. 5a6b8e987ae7f5b4e4c41a7c5b6afc3b1ef6aaa76795d1cd7067cace0af525a1 is in state configured: container state improper
+ true
+ nsenter --net=/run/netns/qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 --preserve-credentials -m -t 1 podman rm -f 5a6b8e987ae7
+ grep -q 'neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49$'
+ printf '%s\n' 5a6b8e987ae7:neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49:Created
+ echo 'Starting a new child container neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49'
+ nsenter --net=/run/netns/qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 --preserve-credentials -m -t 1 podman run --detach --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49.log -v /var/lib/config-data/puppet-generated/neutron/etc/neutron:/etc/neutron:ro -v /run/netns:/run/netns:shared -v /var/lib/neutron:/var/lib/neutron:shared -v /dev/log:/dev/log --net host --pid host --cgroupns host --privileged -u root --name neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 undercloud-0.ctlplane.home.arpa:8787/tripleomastercentos9/openstack-neutron-dhcp-agent:8f4fa80dca5a62ca9a9049b664a60868 /usr/sbin/dnsmasq -k --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/host --addn-hosts=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/opts --dhcp-leasefile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-eaf4f552-0023-4e4a-9b6e-b5f3727e647e,192.168.24.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=256 --conf-file=/dev/null --domain=home.arpa
Error: create directory `/sys/fs/cgroup/../../libpod-b43b25a3bf3af73d24d3f4c52b983334ead7d28462e9ed8ba83f207e4264b1ed.scope`: No such file or directory: OCI runtime attempted to invoke a command that was not found

Once we start the dhcp + ovn meta containers with csgroupns=host this works again.

This is basically the cousin LP of https://bugs.launchpad.net/tripleo/+bug/1936005 but for neutron dhcp and ovn-metadata

OpenStack Infra (hudson-openstack)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/807604
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/60d77b37ec64b8bb82ecc08632bd2fd7426fecea
Submitter: "Zuul (22348)"
Branch: master

commit 60d77b37ec64b8bb82ecc08632bd2fd7426fecea
Author: Michele Baldessari <email address hidden>
Date: Mon Sep 6 18:15:27 2021 +0200

    Set cgroupns=host for neutron_dhcp and ovn-metadata

    Without this change on Centos Stream 9 we will get errors like the
    following:
    + nsenter --net=/run/netns/qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 --preserve-credentials -m -t 1 podman run --detach --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49.log -v /var/lib/config-data/puppet-generated/neutron/etc/neutron:/etc/neutron:ro -v /run/netns:/run/netns:shared -v /var/lib/neutron:/var/lib/neutron:shared -v /dev/log:/dev/log --net host --pid host --cgroupns host --privileged -u root --name neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 undercloud-0.ctlplane.home.arpa:8787/tripleomastercentos9/openstack-neutron-dhcp-agent:8f4fa80dca5a62ca9a9049b664a60868 /usr/sbin/dnsmasq -k --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/host --addn-hosts=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/opts --dhcp-leasefile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-eaf4f552-0023-4e4a-9b6e-b5f3727e647e,192.168.24.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=256 --conf-file=/dev/null --domain=home.arpa
    Error: create directory `/sys/fs/cgroup/../../libpod-b43b25a3bf3af73d24d3f4c52b983334ead7d28462e9ed8ba83f207e4264b1ed.scope`: No such file or directory: OCI runtime attempted to invoke a command that was not found

    See also the related libvirt/nova issue in https://bugs.launchpad.net/tripleo/+bug/1936005

    Tested and with this change the error is gone and dhcp/ovn-metadata is functional
    again.

    Closes-Bug: #1944522

    Change-Id: I8a5b885782386a89c463ff0b7afd7ace91958278

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/810585

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/810585
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ab01af939cfdcb1a4a5490dd08fd76f75ac919da
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit ab01af939cfdcb1a4a5490dd08fd76f75ac919da
Author: Michele Baldessari <email address hidden>
Date: Mon Sep 6 18:15:27 2021 +0200

    Set cgroupns=host for neutron_dhcp and ovn-metadata

    Without this change on Centos Stream 9 we will get errors like the
    following:
    + nsenter --net=/run/netns/qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 --preserve-credentials -m -t 1 podman run --detach --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49.log -v /var/lib/config-data/puppet-generated/neutron/etc/neutron:/etc/neutron:ro -v /run/netns:/run/netns:shared -v /var/lib/neutron:/var/lib/neutron:shared -v /dev/log:/dev/log --net host --pid host --cgroupns host --privileged -u root --name neutron-dnsmasq-qdhcp-7083ee69-e139-4844-8af8-7efa798a3c49 undercloud-0.ctlplane.home.arpa:8787/tripleomastercentos9/openstack-neutron-dhcp-agent:8f4fa80dca5a62ca9a9049b664a60868 /usr/sbin/dnsmasq -k --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/host --addn-hosts=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/opts --dhcp-leasefile=/var/lib/neutron/dhcp/7083ee69-e139-4844-8af8-7efa798a3c49/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-eaf4f552-0023-4e4a-9b6e-b5f3727e647e,192.168.24.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=256 --conf-file=/dev/null --domain=home.arpa
    Error: create directory `/sys/fs/cgroup/../../libpod-b43b25a3bf3af73d24d3f4c52b983334ead7d28462e9ed8ba83f207e4264b1ed.scope`: No such file or directory: OCI runtime attempted to invoke a command that was not found

    See also the related libvirt/nova issue in https://bugs.launchpad.net/tripleo/+bug/1936005

    Tested and with this change the error is gone and dhcp/ovn-metadata is functional
    again.

    Closes-Bug: #1944522

    Change-Id: I8a5b885782386a89c463ff0b7afd7ace91958278
    (cherry picked from commit 60d77b37ec64b8bb82ecc08632bd2fd7426fecea)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 15.1.0

This issue was fixed in the openstack/tripleo-heat-templates 15.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.