kolla-ansible reconfigure fails when manila is enabled

Bug #1603943 reported by Bartłomiej Daca
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
High
Bartłomiej Daca
Liberty
Fix Released
High
Bartłomiej Daca
Mitaka
Fix Released
High
Bartłomiej Daca

Bug Description

In multinode environment with manila component enabled when running "kolla-ansible reconfigure" the command will fail with message:
error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'

in step:
TASK: [manila | Remove the containers]

The manila-related inventory configuration:
[control]
gs0104
gs0105
gs0106
[storage]
ssd05
ssd06
[manila:children]
control
[manila-api:children]
manila
[manila-scheduler:children]
manila
[manila-share:children]
storage

The important ansible steps log:

TASK: [manila | Check the configs] ********************************************
skipping: [ssd06] => (item={'group': 'manila-api', 'name': 'manila_api'})
skipping: [ssd06] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [ssd05] => (item={'group': 'manila-api', 'name': 'manila_api'})
skipping: [ssd05] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
ok: [gs0105] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [gs0106] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [gs0104] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [ssd06] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [ssd05] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [gs0105] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
ok: [gs0106] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [gs0105] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [gs0104] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [gs0106] => (item={'group': 'manila-share', 'name': 'manila_share'})
skipping: [gs0104] => (item={'group': 'manila-share', 'name': 'manila_share'})

TASK: [manila | Containers config strategy] ***********************************
skipping: [ssd05] => (item={'group': 'manila-api', 'name': 'manila_api'})
skipping: [ssd06] => (item={'group': 'manila-api', 'name': 'manila_api'})
skipping: [ssd05] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [ssd06] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
ok: [gs0106] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [gs0104] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [gs0105] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [ssd06] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [ssd05] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [gs0106] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
ok: [gs0104] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [gs0106] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [gs0105] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [gs0104] => (item={'group': 'manila-share', 'name': 'manila_share'})
skipping: [gs0105] => (item={'group': 'manila-share', 'name': 'manila_share'})

TASK: [manila | Remove the containers] ****************************************
skipping: [gs0104] => (item=[{'group': 'manila-api', 'name': 'manila_api'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_api'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-api', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_api', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.572993', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.072470', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_api /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.500523'}])
skipping: [gs0105] => (item=[{'group': 'manila-api', 'name': 'manila_api'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_api'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-api', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_api', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.574359', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.070076', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_api /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.504283'}])
skipping: [gs0106] => (item=[{'group': 'manila-api', 'name': 'manila_api'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_api'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-api', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_api', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.574669', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.072411', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_api /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.502258'}])
fatal: [ssd05] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'
fatal: [ssd06] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'
skipping: [gs0104] => (item=[{'group': 'manila-scheduler', 'name': 'manila_scheduler'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_scheduler'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-scheduler', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_scheduler', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.793828', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.072669', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_scheduler /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.721159'}])
skipping: [gs0105] => (item=[{'group': 'manila-scheduler', 'name': 'manila_scheduler'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_scheduler'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-scheduler', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_scheduler', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.794886', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.073903', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_scheduler /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.720983'}])
skipping: [gs0106] => (item=[{'group': 'manila-scheduler', 'name': 'manila_scheduler'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_scheduler'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-scheduler', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_scheduler', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.801423', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.072810', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_scheduler /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.728613'}])
fatal: [gs0104] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'
fatal: [gs0105] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'
fatal: [gs0106] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'

It looks like it fails because some items are skipped in steps "Check the configs" and "Containers config strategy". Because of this, the item[1] (container_envs.results) cannot have the 'KOLLA_CONFIG_STRATEGY' defined.
Why these items are skipped? Because part of the manila components are deployed on the controller ansible group, and the other part on the storage ansible group, which is ok I guess.

From the code analysis it seems that this is not only manila-related problem. The same problem may occur with other components, when someone will try to customize placement of the component "subelements". For example:
# Murano
[murano-api:children]
host001
[murano-engine:children]
host002

Affected components list:
* ceilometer
* glance
* heat
* ironic
* magnum
* mistral
* murano

Solution: change the order of "when" statements and bring "inventory_hostname in groups[item[0]['group']]" to the top.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/343604
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=e61f13d880efd95de8002854ad9e64526d34ab4b
Submitter: Jenkins
Branch: master

commit e61f13d880efd95de8002854ad9e64526d34ab4b
Author: Bartłomiej Daca <email address hidden>
Date: Mon Jul 18 13:51:11 2016 +0200

    Fix 'reconfigure' command of kolla

    Changed the order of the 'when' statements in "remove/restart
    containers" tasks. It will fix the reconfiguration problem when
    deploying different components on different hosts.

    Change-Id: Ibee9dd56b6128b664144deb1d9eb7ec32e39fd5c
    Closes-Bug: #1603943

Changed in kolla:
status: New → Fix Released
Steven Dake (sdake)
Changed in kolla:
importance: Undecided → High
milestone: none → newton-3
Changed in kolla:
assignee: nobody → Bartłomiej Daca (bartek-daca)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/mitaka)

Reviewed: https://review.openstack.org/344400
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=00d2856caf1b266301b0b9b978038c129cf53cc0
Submitter: Jenkins
Branch: stable/mitaka

commit 00d2856caf1b266301b0b9b978038c129cf53cc0
Author: Bartłomiej Daca <email address hidden>
Date: Mon Jul 18 13:51:11 2016 +0200

    Fix 'reconfigure' command of kolla

    Changed the order of the 'when' statements in "remove/restart
    containers" tasks. It will fix the reconfiguration problem when
    deploying different components on different hosts.

    Change-Id: Ibee9dd56b6128b664144deb1d9eb7ec32e39fd5c
    Closes-Bug: #1603943

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/348120

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/liberty)

Reviewed: https://review.openstack.org/348120
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=7abd34284204bbbc10ac304c122280cfdea9b2e5
Submitter: Jenkins
Branch: stable/liberty

commit 7abd34284204bbbc10ac304c122280cfdea9b2e5
Author: Bartłomiej Daca <email address hidden>
Date: Mon Jul 18 13:51:11 2016 +0200

    Fix 'reconfigure' command of kolla

    Changed the order of the 'when' statements in "remove/restart
    containers" tasks. It will fix the reconfiguration problem when
    deploying different components on different hosts.

    Change-Id: Ibee9dd56b6128b664144deb1d9eb7ec32e39fd5c
    Closes-Bug: #1603943
    (cherry picked from commit 00d2856caf1b266301b0b9b978038c129cf53cc0)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 3.0.0.0b3

This issue was fixed in the openstack/kolla 3.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 2.0.3

This issue was fixed in the openstack/kolla 2.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.