Activity log for bug #2038914

Date Who What changed Old value New value Message
2023-10-10 10:53:43 Marcel Hauptmann bug added bug
2023-10-10 11:34:03 Marcel Hauptmann summary kolla_toolbox fails to deploy after switching from kolla-ansible 16.0.1 to master tag kolla_toolbox fails to deploy after switching from kolla-ansible 16.1.0 to master tag
2023-10-10 11:34:20 Marcel Hauptmann description **Bug Report** We have successfully deployed openstack with kolla-ansible version 16.0.1. Unfortunately, we have problems with the opensearch container in this version - it does not start and always crashes. Therefore we would like to deploy OpenStack with kolla-ansible master. However, in this version we have the problem that kolla-toolbox is not found during the deployment. The deployment fails on all nodes: ---------------------------- Error in gitlab pipeline: fatal: [10.21.20.2]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1696926350, "HostConfig": {"NetworkMode": "host"}, "Id": "395e3ee92d5146ae6e0f3a2dab16194f7499f898d7e00fb7a2847ee72d44e2b7", "Image": "quay.io/openstack.kolla/kolla-toolbox:2023.1-rocky-9", "ImageID": "sha256:5b8102090b7995252a753d2836c7b10545e40939572cfb8e7ab82c28d901e8f3", "Labels": {"build-date": "20230920", "io.buildah.version": "1.30.0", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "kolla-toolbox"}, "Mounts": [{"Destination": "/dev", "Mode": "rw", "Propagation": "rprivate", "RW": true, "Source": "/dev", "Type": "bind"}, {"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/run", "Mode": "shared", "Propagation": "shared", "RW": true, "Source": "/run", "Type": "bind"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/kolla-toolbox", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/kolla_toolbox"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": "8a180824a28e73a21f8806bebeb90a64aedde3789fbe1365e8fad3a9ac9e8bd2"}}}, "Ports": [], "State": "exited", "Status": "Exited (127) 23 seconds ago", "changed": true, "msg": "Container timed out"} ---------------------------- docker logs kolla_toolbox + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /var/lib/rabbitmq/.erlang.cookie INFO:__main__:Copying /var/lib/kolla/config_files/rabbitmq-erlang.cookie to /var/lib/rabbitmq/.erlang.cookie INFO:__main__:Setting permission for /var/lib/rabbitmq/.erlang.cookie INFO:__main__:Deleting /etc/rabbitmq/rabbitmq-env.conf INFO:__main__:Copying /var/lib/kolla/config_files/rabbitmq-env.conf to /etc/rabbitmq/rabbitmq-env.conf INFO:__main__:Setting permission for /etc/rabbitmq/rabbitmq-env.conf INFO:__main__:Deleting /etc/rabbitmq/erl_inetrc INFO:__main__:Copying /var/lib/kolla/config_files/erl_inetrc to /etc/rabbitmq/erl_inetrc INFO:__main__:Setting permission for /etc/rabbitmq/erl_inetrc INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/log/kolla/ansible.log ++ cat /run_command + CMD=kolla_toolbox + ARGS= + sudo kolla_copy_cacerts + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -f /var/log/kolla/ansible.log ]] +++ stat -c %U:%G /var/log/kolla/ansible.log ++ [[ ansible:kolla != \a\n\s\i\b\l\e\:\k\o\l\l\a ]] +++ stat -c %a /var/log/kolla/ansible.log ++ [[ 664 != \6\6\4 ]] + echo 'Running command: '\''kolla_toolbox'\''' + exec kolla_toolbox Running command: 'kolla_toolbox' /usr/local/bin/kolla_start: line 24: exec: kolla_toolbox: not found ---------------------------- How can we fix kolla_toolbox in kolla-ansible master? Thanks in advance! **Environment** - OS: Rocky Linux 9.2 - OpenStack Release: 2023.1 - Kernel:5.14.0-284.25.1.el9_2.x86_64 - Docker version: 24.0.6 - Kolla-ansible version: switched from 16.0.1 to master - Are you using official images from Docker Hub or self built -> Official images **Bug Report** We have successfully deployed openstack with kolla-ansible version 16.1.0. Unfortunately, we have problems with the opensearch container in this version - it does not start and always crashes. Therefore we would like to deploy OpenStack with kolla-ansible master. However, in this version we have the problem that kolla-toolbox is not found during the deployment. The deployment fails on all nodes: ---------------------------- Error in gitlab pipeline: fatal: [10.21.20.2]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1696926350, "HostConfig": {"NetworkMode": "host"}, "Id": "395e3ee92d5146ae6e0f3a2dab16194f7499f898d7e00fb7a2847ee72d44e2b7", "Image": "quay.io/openstack.kolla/kolla-toolbox:2023.1-rocky-9", "ImageID": "sha256:5b8102090b7995252a753d2836c7b10545e40939572cfb8e7ab82c28d901e8f3", "Labels": {"build-date": "20230920", "io.buildah.version": "1.30.0", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "kolla-toolbox"}, "Mounts": [{"Destination": "/dev", "Mode": "rw", "Propagation": "rprivate", "RW": true, "Source": "/dev", "Type": "bind"}, {"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/run", "Mode": "shared", "Propagation": "shared", "RW": true, "Source": "/run", "Type": "bind"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/kolla-toolbox", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/kolla_toolbox"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": "8a180824a28e73a21f8806bebeb90a64aedde3789fbe1365e8fad3a9ac9e8bd2"}}}, "Ports": [], "State": "exited", "Status": "Exited (127) 23 seconds ago", "changed": true, "msg": "Container timed out"} ---------------------------- docker logs kolla_toolbox + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /var/lib/rabbitmq/.erlang.cookie INFO:__main__:Copying /var/lib/kolla/config_files/rabbitmq-erlang.cookie to /var/lib/rabbitmq/.erlang.cookie INFO:__main__:Setting permission for /var/lib/rabbitmq/.erlang.cookie INFO:__main__:Deleting /etc/rabbitmq/rabbitmq-env.conf INFO:__main__:Copying /var/lib/kolla/config_files/rabbitmq-env.conf to /etc/rabbitmq/rabbitmq-env.conf INFO:__main__:Setting permission for /etc/rabbitmq/rabbitmq-env.conf INFO:__main__:Deleting /etc/rabbitmq/erl_inetrc INFO:__main__:Copying /var/lib/kolla/config_files/erl_inetrc to /etc/rabbitmq/erl_inetrc INFO:__main__:Setting permission for /etc/rabbitmq/erl_inetrc INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/log/kolla/ansible.log ++ cat /run_command + CMD=kolla_toolbox + ARGS= + sudo kolla_copy_cacerts + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -f /var/log/kolla/ansible.log ]] +++ stat -c %U:%G /var/log/kolla/ansible.log ++ [[ ansible:kolla != \a\n\s\i\b\l\e\:\k\o\l\l\a ]] +++ stat -c %a /var/log/kolla/ansible.log ++ [[ 664 != \6\6\4 ]] + echo 'Running command: '\''kolla_toolbox'\''' + exec kolla_toolbox Running command: 'kolla_toolbox' /usr/local/bin/kolla_start: line 24: exec: kolla_toolbox: not found ---------------------------- How can we fix kolla_toolbox in kolla-ansible master? Thanks in advance! **Environment** - OS: Rocky Linux 9.2 - OpenStack Release: 2023.1 - Kernel:5.14.0-284.25.1.el9_2.x86_64 - Docker version: 24.0.6 - Kolla-ansible version: switched from 16.1.0 to master - Are you using official images from Docker Hub or self built -> Official images