tripleo

Scenario 003 failing on standalone deployment - unable to send audit message - container designate_pool_manage

Bug #1960648 reported by Arx Cruz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo yoga-3

Bug Description

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-master/a96d8eb/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

2022-02-11 04:34:41.986159 | fa163e72-b06a-4f15-0c8d-0000000027c2 | TASK | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5
2022-02-11 04:35:46.056207 | | WARNING | ERROR: Can't run container designate_pool_manage
stderr: + sudo -E kolla_set_configs
sudo: unable to send audit message: Operation not permitted
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/designate/designate.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/designate/designate.conf to /etc/designate/designate.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/designate/pools.yaml to /etc/designate/pools.yaml
INFO:__main__:Creating directory /etc/designate/private
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/designate/private/bind1.key to /etc/designate/private/bind1.key
INFO:__main__:Creating directory /etc/my.cnf.d
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/designate
INFO:__main__:Setting permission for /var/log/designate/designate-api.log
INFO:__main__:Setting permission for /var/log/designate/mdns.log
INFO:__main__:Setting permission for /var/log/designate/producer.log
INFO:__main__:Setting permission for /var/log/designate/central.log
INFO:__main__:Setting permission for /var/log/designate/worker.log
++ cat /run_command

The only hint the logs gave us is this:

sudo: unable to send audit message: Operation not permitted

Revision history for this message
Ronelle Landy (rlandy) wrote : Re: Scenario 003 failing on standalone deployment - unable to send audit message: Operation not permitted

comparing logs on passing and failing jobs:

Failing job:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-master/a96d8eb/logs/undercloud/var/log/extra/podman/containers/designate_pool_manage/stdout.log.txt.gz

Passing job:
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-master/74e8a84/logs/undercloud/var/log/extra/podman/containers/designate_pool_manage/stdout.log.txt.gz

summary: - Scenario 003 failing on deployment
+ Scenario 003 failing on standalone deployment - unable to send audit
+ message: Operation not permitted
Revision history for this message
Ronelle Landy (rlandy) wrote :

Failure started on 02/09:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario003-standalone-master

Revision history for this message
Ronelle Landy (rlandy) wrote :

https://bugs.launchpad.net/tripleo/+bug/1960625 - notes change in network scripts

summary: Scenario 003 failing on standalone deployment - unable to send audit
- message: Operation not permitted
+ message - container designate_pool_manage
Revision history for this message
Ronelle Landy (rlandy) wrote :

Possible impact:
https://opendev.org/openstack/tripleo-heat-templates/commit/acf032a098d47f13564e3fc5716fe69f3d4ff813

Revision history for this message
Rabi Mishra (rabi) wrote :
Download full text (3.9 KiB)

Sounds like the same puppet-redis issue.

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-master/a96d8eb/logs/undercloud/var/log/containers/stdouts/redis.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-master/a96d8eb/logs/undercloud/var/log/containers/designate/central.log.txt.gz

2022-02-11 09:32:51.691 2 DEBUG tooz.drivers.redis [req-3c6c572b-9e8d-4efb-b727-31657230a08f - - - - -] Redis connection error, retry limit has been reached, aborting - Connection closed by server. wrapper /usr/lib/python3.9/site-packages/tooz/drivers/redis.py:57
2022-02-11 09:32:51.692 2 ERROR oslo_service.service [req-3c6c572b-9e8d-4efb-b727-31657230a08f - - - - -] Error starting thread.: tooz.coordination.ToozConnectionError: Connection closed by server.
2022-02-11 09:32:51.692 2 ERROR oslo_service.service Traceback (most recent call last):
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/tooz/drivers/redis.py", line 52, in wrapper
2022-02-11 09:32:51.692 2 ERROR oslo_service.service return func(*args, **kwargs)
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/tooz/drivers/redis.py", line 476, in _start
2022-02-11 09:32:51.692 2 ERROR oslo_service.service self._server_info = self._client.info()
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/redis/client.py", line 1304, in info
2022-02-11 09:32:51.692 2 ERROR oslo_service.service return self.execute_command('INFO')
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/redis/client.py", line 898, in execute_command
2022-02-11 09:32:51.692 2 ERROR oslo_service.service conn = self.connection or pool.get_connection(command_name, **options)
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/redis/connection.py", line 1192, in get_connection
2022-02-11 09:32:51.692 2 ERROR oslo_service.service connection.connect()
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/redis/connection.py", line 567, in connect
2022-02-11 09:32:51.692 2 ERROR oslo_service.service self.on_connect()
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/redis/connection.py", line 643, in on_connect
2022-02-11 09:32:51.692 2 ERROR oslo_service.service auth_response = self.read_response()
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/redis/connection.py", line 739, in read_response
2022-02-11 09:32:51.692 2 ERROR oslo_service.service response = self._parser.read_response()
2022-02-11 09:32:51.692 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/redis/connection.py", line 324, in read_response
2022-02-11 09:32:51.692 2 ERROR oslo_service.service raw = self._buffer.readline()
2022-02-11 09:32:51.692 2 ERROR oslo_s...

Read more...

Ronelle Landy (rlandy)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.