OpenSearch services are restarted when no configuration change is made

Bug #2028362 reported by Doug Szumski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Status tracked in Bobcat
Antelope
Fix Committed
Medium
Sven Kieske
Bobcat
Fix Committed
Medium
Doug Szumski
Yoga
Fix Committed
Medium
Sven Kieske
Zed
Fix Committed
Medium
Sven Kieske

Bug Description

When deploying current master (Rocky 9 host), I noticed that on consecutive runs OpenSearch services are restarted, even though no config change has been made. This is caused by the 'Check opensearch containers' task returning that opensearch-dashboards has changed:

```
ok: [controller0] => (item={'key': 'opensearch', 'value': {'container_name': 'opensearch', 'group': 'opensearch', 'enab
led': True, 'image': '192.168.33.3:4000/openstack.kolla/opensearch:210723-1-rocky-9', 'environment': {'OPENSEARCH_JAVA_
OPTS': '-Xms1g -Xmx1g -Dlog4j2.formatMsgNoLookups=true'}, 'volumes': ['/etc/kolla/opensearch/:/var/lib/kolla/config_fil
es/', '/etc/localtime:/etc/localtime:ro', '', 'opensearch:/var/lib/opensearch/data', 'kolla_logs:/var/log/kolla/'], 'di
mensions': {'ulimits': {'nofile': {'soft': 1048576, 'hard': 1048576}}}, 'healthcheck': {'interval': '30', 'retries': '3
', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_curl http://192.168.33.3:9200'], 'timeout': '30'}, 'haproxy'
: {'opensearch': {'enabled': True, 'mode': 'http', 'external': False, 'port': '9200', 'frontend_http_extra': ['option d
ontlog-normal']}}}})

changed: [controller0] => (item={'key': 'opensearch-dashboards', 'value': {'container_name': 'opensearch_dashboards', '
group': 'opensearch-dashboards', 'enabled': True, 'environment': {'OPENSEARCH_DASHBOARDS_SECURITY_PLUGIN': 'False'}, 'i
mage': '192.168.33.3:4000/openstack.kolla/opensearch-dashboards:210723-1-rocky-9', 'volumes': ['/etc/kolla/opensearch-d
ashboards/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'kolla_logs:/var/log/kolla/'], 'di
mensions': {'ulimits': {'nofile': {'soft': 1048576, 'hard': 1048576}}}, 'healthcheck': {'interval': '30', 'retries': '3
', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_curl http://192.168.33.3:5601'], 'timeout': '30'}, 'haproxy'
: {'opensearch-dashboards': {'enabled': True, 'mode': 'http', 'external': False, 'port': '5601', 'auth_user': 'opensear
ch', 'auth_pass': 'foo'}, 'opensearch_dashboards_external': {'enabled': True, 'mod
e': 'http', 'external': True, 'port': '5601', 'auth_user': 'opensearch', 'auth_pass': 'foo'}}}})

```

Then:

```
RUNNING HANDLER [opensearch : Restart opensearch container] *********************
changed: [controller0]

RUNNING HANDLER [opensearch : Restart opensearch-dashboards container] **********
changed: [controller0]
```

Even if something /has/ changed with OpenSearch Dashboards, we shouldn't restart OpenSearch.

Doug Szumski (dszumski)
Changed in kolla-ansible:
assignee: nobody → Doug Szumski (dszumski)
Revision history for this message
Doug Szumski (dszumski) wrote :

Here's the root cause:
https://review.opendev.org/c/openstack/kolla-ansible/+/889189

The remaining fix is to prevent both OpenSearch and OpenSearch Dashboards getting restarted, when a configuration change is only made to one of them.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/890483

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/890484

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/890485

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/890483
Committed: https://opendev.org/openstack/kolla-ansible/commit/b10e7dbe88718f7be49f00f73792c953a30671d7
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit b10e7dbe88718f7be49f00f73792c953a30671d7
Author: Doug Szumski <email address hidden>
Date: Fri Jul 21 16:45:37 2023 +0100

    Fix OpenSearch Dashboards health check

    The OpenSearch Dashboards container does not have a health
    check defined when created. This causes the container to always
    restart when reconfigured, even if no change has been made.

    Change-Id: I0b437a77aeb61bc5ae9238f900a1fa00cbc34e18
    Partial-Bug: #2028362
    (cherry picked from commit bacd6c7f4ff6788feb75a6abc4339e29684195ab)

tags: added: in-stable-zed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/890484
Committed: https://opendev.org/openstack/kolla-ansible/commit/511007f71c2b5e1f1b8f8272a9073bdad09c5c0e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 511007f71c2b5e1f1b8f8272a9073bdad09c5c0e
Author: Doug Szumski <email address hidden>
Date: Fri Jul 21 16:45:37 2023 +0100

    Fix OpenSearch Dashboards health check

    The OpenSearch Dashboards container does not have a health
    check defined when created. This causes the container to always
    restart when reconfigured, even if no change has been made.

    Change-Id: I0b437a77aeb61bc5ae9238f900a1fa00cbc34e18
    Partial-Bug: #2028362
    (cherry picked from commit bacd6c7f4ff6788feb75a6abc4339e29684195ab)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/890485
Committed: https://opendev.org/openstack/kolla-ansible/commit/328f62ddd763c625c8a711e47b4ca1e7b7ae12c5
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 328f62ddd763c625c8a711e47b4ca1e7b7ae12c5
Author: Doug Szumski <email address hidden>
Date: Fri Jul 21 16:45:37 2023 +0100

    Fix OpenSearch Dashboards health check

    The OpenSearch Dashboards container does not have a health
    check defined when created. This causes the container to always
    restart when reconfigured, even if no change has been made.

    Change-Id: I0b437a77aeb61bc5ae9238f900a1fa00cbc34e18
    Partial-Bug: #2028362
    (cherry picked from commit bacd6c7f4ff6788feb75a6abc4339e29684195ab)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.