Race condition between haproxy and keystone register task

Bug #1699096 reported by Juan J. Martínez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Juan J. Martínez

Bug Description

When keystone register tasks run, it is possible that "Creating default user role" runs before haproxy detects keystone services are up.

When that happens, "Creating default user role" fails:

TASK [keystone : Creating default user role] *********************************************************************************************************************************************************************************************************************************
task path: /usr/share/kolla-ansible/ansible/roles/keystone/tasks/register.yml:10
Using module file /usr/share/kolla-ansible/ansible/library/kolla_toolbox.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1497954431.23-12662551185212 `" && echo ansible-tmp-1497954431.23-12662551185212="` echo /root/.ansible/tmp/ansible-tmp-1497954431.23-12662551185212 `" ) && sleep 0'
<localhost> PUT /tmp/tmpNbygO4 TO /root/.ansible/tmp/ansible-tmp-1497954431.23-12662551185212/kolla_toolbox.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1497954431.23-12662551185212/ /root/.ansible/tmp/ansible-tmp-1497954431.23-12662551185212/kolla_toolbox.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1497954431.23-12662551185212/kolla_toolbox.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1497954431.23-12662551185212/" > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "module_args": {
                "auth": "{{ openstack_keystone_auth }}",
                "name": "_member_"
            },
            "module_extra_vars": {
                "openstack_keystone_auth": {
                    "auth_url": "http://172.28.128.254:35357",
                    "domain_name": "default",
                    "password": "I81yYYS6756P93x10u36hhy5YZOhtPpl7FP4cfSM",
                    "project_name": "admin",
                    "username": "admin"
                }
            },
            "module_name": "os_keystone_role"
        }
    },
    "msg": "Could not determine a suitable URL for the plugin"
}

This issue appeared after this change: https://github.com/openstack/kolla-ansible/commit/38ad05a8729c00ccc012f234135885907065035d

Previous task ("Creating admin project, user, role, service, and endpoint") works fine because it performs a keystone-manage bootstrap, and that uses SQL directly instead of the API.

Suggested solutions include:

* revert part of the mentioned change, so the task retries
* wait for the port to be ready

description: updated
Revision history for this message
Juan J. Martínez (jjmartinez) wrote :

On second thought, waiting for the port is not good enough as haproxy is always bound; unless we check for a specific HTTP response.

Retry should be fine though.

Changed in kolla-ansible:
assignee: nobody → Juan J. Martínez (jjmartinez)
status: New → In Progress
Changed in kolla-ansible:
importance: Undecided → Medium
milestone: none → pike-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.openstack.org/475820
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=ba5c4302788ab6db42782501f79ff6e38748b408
Submitter: Jenkins
Branch: master

commit ba5c4302788ab6db42782501f79ff6e38748b408
Author: Juan J. Martinez <email address hidden>
Date: Tue Jun 20 15:19:48 2017 +0100

    Retry Keystone's default user role creation

    Sometimes Ansible is faster running tasks that haproxy tagging Keystone
    services as UP. Keystone bootstrap uses SQL directly but the default
    user role creation requires the API, and because of that it may fail.

    Retry in case the backend is not yet available.

    Change-Id: I9dfc030bbf92ca0a3dcb008d55e9fa2055f900ec
    Closes-Bug: 1699096

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 5.0.0.0b3

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.