Fuel for OpenStack

Murano HA is broken

  1. Series 5.0.x
  2. Bug #1325983
Bug #1325983 reported by Anastasia Palkina
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Dmitry Borodaenko
Fuel for OpenStack 5.1
4.1.x
Fix Committed
High
Dmitry Borodaenko
Fuel for OpenStack 4.1.1
5.0.x
Fix Committed
High
Dmitry Borodaenko
Fuel for OpenStack 5.0.1

Bug Description

"build_id": "2014-06-03_00-40-36",
"mirantis": "yes",
"build_number": "338",
"nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b",
"ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467",
"fuelmain_sha": "15805b14f36fc0a896f136051a3be10fea4611d9",
"astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968",
"release": "4.1B",
"fuellib_sha": "377f27df4cb008b400b69013dbd606e6225f5c04"

1. Create new environment (Ubuntu, HA mode)
2. Choose VLAN segmentation
3. Choose both Ceph
4. Choose Murano installation
5. Add 3 controllers, compute and 3 ceph nodes
6. Start deployment. It was successful
7. Login to Horizon
8. Create Murano environment. Try to delete it. Error has appeared. Unable to delete.

There are errors in murano-api.log in /var/log/remote/node-10.domain.tld/murano-api.log

2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 ERROR muranoapi.common.service [-] [Errno 111] ECONNREFUSED
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service Traceback (most recent call last):
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service File "/usr/lib/python2.7/dist-packages/muranoapi/common/service.py", line 79, in _handle_reports
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service with self._create_mq_client() as mqClient:
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service File "/usr/lib/python2.7/dist-packages/muranocommon/messaging/mqclient.py", line 43, in __enter__
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service self.connect()
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service File "/usr/lib/python2.7/dist-packages/muranocommon/messaging/mqclient.py", line 52, in connect
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service promise = self._client.connect()
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service File "/usr/lib/python2.7/dist-packages/puka/client.py", line 19, in wrapper
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service p = method(*args, **kwargs)
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service File "/usr/lib/python2.7/dist-packages/puka/client.py", line 48, in connect
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service return self._connect()
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service File "/usr/lib/python2.7/dist-packages/puka/connection.py", line 91, in _connect
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service self.sd.connect(sockaddr)
2014-06-03T13:09:25.478795+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 194, in connect
2014-06-03T13:09:25.479487+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service socket_checkerr(fd)
2014-06-03T13:09:25.479487+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 46, in socket_checkerr
2014-06-03T13:09:25.479487+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service raise socket.error(err, errno.errorcode[err])
2014-06-03T13:09:25.479487+00:00 debug: 2014-06-03 13:09:24.845 20934 TRACE muranoapi.common.service error: [Errno 111] ECONNREFUSED

Tags: murano
Revision history for this message
Anastasia Palkina (apalkina) wrote :
Tatyanka (tatyana-leontovich)
Changed in fuel:
importance: Undecided → High
Revision history for this message
Anastasia Palkina (apalkina) wrote :
Revision history for this message
Igor Yozhikov (iyozhikov) wrote :

Next was found:
rabbitmq section was set up to use host=172.16.0.2 and port=55572, but there was nothing listening on that endpoint.

Ryan Moe (rmoe)
Changed in fuel:
status: New → Confirmed
Dmitry Borodaenko (angdraug)
Changed in fuel:
assignee: Murano Team (murano) → Dmitry Borodaenko (dborodaenko)
Revision history for this message
Ryan Moe (rmoe) wrote :

The issue is that Murano's rabbitmq instance is not configured in haproxy. Murano services try to connect to the public vip but that vip only exists in the haproxy namespace.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/4.1)

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/97938

Dmitry Borodaenko (angdraug)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/97954

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/97955

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/4.1)

Reviewed: https://review.openstack.org/97938
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3511461a2b529619a787a6306441d9039699e71d
Submitter: Jenkins
Branch: stable/4.1

commit 3511461a2b529619a787a6306441d9039699e71d
Author: Dmitry Borodaenko <email address hidden>
Date: Wed Jun 4 14:15:37 2014 -0700

    Configure Murano RabbitMQ in HAProxy

    Murano is configured to use public VIP for RabbitMQ, so listening on internal
    (management) VIP is disabled.

    Change-Id: I5eabefefba58e2cd5f012d231aa29965afcddb08
    Closes-Bug: #1325983

Vladimir Kuklin (vkuklin)
summary: - Unable to delete Murano environment
+ Murano HA is broken
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/97954
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5da2e8873ce4ad538fdb1aa8e887a34fb4f69601
Submitter: Jenkins
Branch: master

commit 5da2e8873ce4ad538fdb1aa8e887a34fb4f69601
Author: Dmitry Borodaenko <email address hidden>
Date: Wed Jun 4 14:58:20 2014 -0700

    Put Murano RabbitMQ HAProxy service on Public IP

    Unlike OpenStack RabbitMQ, Murano RabbitMQ uses the Public VIP, that has to be
    specified explicitly in openstack::ha::haproxy_service parameters.

    Make openstack::ha::murano consistent with other OpenStack services: declare
    all related HAProxy services in the same file, do not make it a dependency of
    HAProxy itself (only main RabbitMQ HAProxy service needs to have before_start =
    true).

    Change-Id: I0fd7fb1564b322315b88320a331d5a89f0d74c48
    Closes-Bug: #1325983

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/97955
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=253be9bae39a4d7ac4883e958c9453380abc393e
Submitter: Jenkins
Branch: stable/5.0

commit 253be9bae39a4d7ac4883e958c9453380abc393e
Author: Dmitry Borodaenko <email address hidden>
Date: Wed Jun 4 14:58:20 2014 -0700

    Put Murano RabbitMQ HAProxy service on Public IP

    Unlike OpenStack RabbitMQ, Murano RabbitMQ uses the Public VIP, that has to be
    specified explicitly in openstack::ha::haproxy_service parameters.

    Make openstack::ha::murano consistent with other OpenStack services: declare
    all related HAProxy services in the same file, do not make it a dependency of
    HAProxy itself (only main RabbitMQ HAProxy service needs to have before_start =
    true).

    Change-Id: I0fd7fb1564b322315b88320a331d5a89f0d74c48
    Closes-Bug: #1325983

Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #255
"build_id": "2014-06-16_00-31-15",
"mirantis": "yes",
"build_number": "255",
"ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db",
"nailgun_sha": "984aa7a86487f1488c2f83c052904abd9f589b7f",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "6f355160366475d52050d7898a1080a95ecb9cbf",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "99d74172887ab81d38132655d6e5d180e8726437"

Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/5.1.x
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.