Failure of creating haproxy mysql user

Bug #1577273 reported by Serguei Bezverkhi
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
kolla
Invalid
High
Serguei Bezverkhi
Liberty
Won't Fix
Critical
Unassigned
Mitaka
Invalid
Critical
Unassigned
Newton
Invalid
High
Serguei Bezverkhi

Bug Description

All in one deployment on stable/mitaka and master fails with the following traceback. Suspecting this issue is triggered by recent CentOS update. Here is the failure message: VersionConflict: (setuptools 0.9.8 (/usr/lib/python2.7/site-packages), Requirement.parse('setuptools>=11.3'))

TASK: [mariadb | Waiting for MariaDB service to be ready] *********************
ok: [192.168.80.24]

TASK: [mariadb | Creating haproxy mysql user] *********************************
failed: [192.168.80.24] => {"attempts": 10, "changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='192.168.80.24' login_port='3306' login_user='root' login_password='wPsZfDxD0sxqZ3F8T3fbavjjSzIYHi0hpZQ5UoTs' name='haproxy' password='' host='%' priv=*.*:USAGE"], "delta": "0:00:00.299353", "end": "2016-05-01 23:02:55.623531", "failed": true, "rc": 250, "start": "2016-05-01 23:02:55.324178", "stdout_lines": ["ERROR! Unexpected Exception: (setuptools 0.9.8 (/usr/lib/python2.7/site-packages), Requirement.parse('setuptools>=11.3'))", "the full traceback was:", "", "Traceback (most recent call last):", " File \"/usr/bin/ansible\", line 75, in <module>", " from ansible.cli.adhoc import AdHocCLI as mycli", " File \"/usr/lib/python2.7/site-packages/ansible/cli/adhoc.py\", line 28, in <module>", " from ansible.executor.task_queue_manager import TaskQueueManager", " File \"/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py\", line 28, in <module>", " from ansible.executor.play_iterator import PlayIterator", " File \"/usr/lib/python2.7/site-packages/ansible/executor/play_iterator.py\", line 29, in <module>", " from ansible.playbook.block import Block", " File \"/usr/lib/python2.7/site-packages/ansible/playbook/__init__.py\", line 25, in <module>", " from ansible.playbook.play import Play", " File \"/usr/lib/python2.7/site-packages/ansible/playbook/play.py\", line 27, in <module>", " from ansible.playbook.base import Base", " File \"/usr/lib/python2.7/site-packages/ansible/playbook/base.py\", line 35, in <module>", " from ansible.parsing.dataloader import DataLoader", " File \"/usr/lib/python2.7/site-packages/ansible/parsing/dataloader.py\", line 31, in <module>", " from ansible.parsing.vault import VaultLib", " File \"/usr/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py\", line 67, in <module>", " from cryptography.hazmat.primitives.hashes import SHA256 as c_SHA256", " File \"/usr/lib64/python2.7/site-packages/cryptography/hazmat/primitives/hashes.py\", line 15, in <module>", " from cryptography.hazmat.backends.interfaces import HashBackend", " File \"/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/__init__.py\", line 7, in <module>", " import pkg_resources", " File \"/usr/lib/python2.7/site-packages/pkg_resources.py\", line 3011, in <module>", " parse_requirements(__requires__), Environment()", " File \"/usr/lib/python2.7/site-packages/pkg_resources.py\", line 630, in resolve", " raise VersionConflict(dist,req) # XXX put more info here", "VersionConflict: (setuptools 0.9.8 (/usr/lib/python2.7/site-packages), Requirement.parse('setuptools>=11.3'))"], "warnings": []}
stdout: ERROR! Unexpected Exception: (setuptools 0.9.8 (/usr/lib/python2.7/site-packages), Requirement.parse('setuptools>=11.3'))
the full traceback was:

Traceback (most recent call last):
  File "/usr/bin/ansible", line 75, in <module>
    from ansible.cli.adhoc import AdHocCLI as mycli
  File "/usr/lib/python2.7/site-packages/ansible/cli/adhoc.py", line 28, in <module>
    from ansible.executor.task_queue_manager import TaskQueueManager
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 28, in <module>
    from ansible.executor.play_iterator import PlayIterator
  File "/usr/lib/python2.7/site-packages/ansible/executor/play_iterator.py", line 29, in <module>
    from ansible.playbook.block import Block
  File "/usr/lib/python2.7/site-packages/ansible/playbook/__init__.py", line 25, in <module>
    from ansible.playbook.play import Play
  File "/usr/lib/python2.7/site-packages/ansible/playbook/play.py", line 27, in <module>
    from ansible.playbook.base import Base
  File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 35, in <module>
    from ansible.parsing.dataloader import DataLoader
  File "/usr/lib/python2.7/site-packages/ansible/parsing/dataloader.py", line 31, in <module>
    from ansible.parsing.vault import VaultLib
  File "/usr/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py", line 67, in <module>
    from cryptography.hazmat.primitives.hashes import SHA256 as c_SHA256
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/primitives/hashes.py", line 15, in <module>
    from cryptography.hazmat.backends.interfaces import HashBackend
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 7, in <module>
    import pkg_resources
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3011, in <module>
    parse_requirements(__requires__), Environment()
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 630, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
VersionConflict: (setuptools 0.9.8 (/usr/lib/python2.7/site-packages), Requirement.parse('setuptools>=11.3'))
msg: Task failed as maximum retries was encountered

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/site.retry

192.168.80.24 : ok=74 changed=33 unreachable=0 failed=1

Command failed ansible-playbook -i /root/deployments/kolla_latest/kolla/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /root/deployments/kolla_latest/kolla/ansible/site.yml

Tags: retriage
Changed in kolla:
assignee: nobody → Serguei Bezverkhi (sbezverk)
Revision history for this message
Serguei Bezverkhi (sbezverk) wrote :

After adding these lines to Docker file of base image the issue was resolved:

RUN yum -y install \
         python-pip \
    && yum clean all

RUN pip install -U pip

RUN pip install -U setuptools

Changed in kolla:
importance: Undecided → Critical
milestone: none → newton-1
Revision history for this message
Steven Dake (sdake) wrote :

setuptools should not be installed at all from RPM. If it is being installed it is in error. When python is run from source, it must be all source. Intermixing RPMs with source is super dangerous.

Revision history for this message
Chris Ricker (chris-ricker) wrote :

This error shows up when doing a binary install (centos, stable/mitaka). The workaround in #1 works to get past it

I don't follow how comment #2 helps. Yes, mixing pip and RPM is a potentially fragile thing but it's how kolla works (see docs that have you easy_install pip and then do pip install kolla/ for example)

Steven Dake (sdake)
Changed in kolla:
status: New → Triaged
Changed in kolla:
milestone: newton-1 → newton-2
tags: added: retriage
Changed in kolla:
milestone: newton-2 → newton-3
Changed in kolla:
milestone: newton-3 → newton-rc1
Changed in kolla:
milestone: newton-rc1 → newton-rc2
Revision history for this message
Paul Bourke (pauldbourke) wrote :

Not seeing this in the gates so dropping priority and pushing to Ocata

Changed in kolla:
importance: Critical → High
milestone: newton-rc2 → ocata-1
Changed in kolla:
milestone: ocata-1 → ocata-2
Changed in kolla:
milestone: ocata-2 → ocata-3
Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote :

do not see this issue again. please re-confirm and provide some info.

Changed in kolla:
status: Triaged → Incomplete
Changed in kolla:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.