devstack

Devstack nova-compute fails with ZeroMQ

Bug #1395721 reported by Oleksii Zamiatin
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
devstack
Fix Released
Undecided
Li Ma
oslo.messaging
Fix Released
Undecided
Oleksii Zamiatin

Bug Description

Deploy devstack with ZeroMQ:

git clone https://github.com/openstack-dev/devstack.git
cd devstack

Copy local.conf from samples

Add this line to local.conf (select zeromq for messaging):
ENABLED_SERVICES+=,-rabbit,-qpid,zeromq,n-cpu,n-net

Run deployment:
./stack.sh

Deployment fails with error code.

Run ./rejoin-stack.sh

Go to nova-compute screen, see:

2014-11-24 11:34:04.079 CRITICAL nova [req-1369c998-f2ef-476f-b1b4-f808ee2f6b73 None None] ImportError: No module named rpc.matchmaker_redis

2014-11-24 11:34:04.079 TRACE nova Traceback (most recent call last):
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/bin/nova-compute", line 10, in <module>
2014-11-24 11:34:04.079 TRACE nova sys.exit(main())
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/cmd/compute.py", line 72, in main
2014-11-24 11:34:04.079 TRACE nova db_allowed=CONF.conductor.use_local)
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/service.py", line 275, in create
2014-11-24 11:34:04.079 TRACE nova db_allowed=db_allowed)
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/service.py", line 157, in __init__
2014-11-24 11:34:04.079 TRACE nova self.conductor_api.wait_until_ready(context.get_admin_context())
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/conductor/api.py", line 313, in wait_until_ready
2014-11-24 11:34:04.079 TRACE nova timeout=timeout)
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/baserpc.py", line 62, in ping
2014-11-24 11:34:04.079 TRACE nova return cctxt.call(context, 'ping', arg=arg_p)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-11-24 11:34:04.079 TRACE nova retry=self.retry)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-11-24 11:34:04.079 TRACE nova timeout=timeout, retry=retry)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py", line 954, in send
2014-11-24 11:34:04.079 TRACE nova return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py", line 945, in _send
2014-11-24 11:34:04.079 TRACE nova allowed_remote_exmods=self._allowed_remote_exmods)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py", line 736, in _multi_send
2014-11-24 11:34:04.079 TRACE nova queues = _get_matchmaker().queues(topic)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py", line 832, in _get_matchmaker
2014-11-24 11:34:04.079 TRACE nova matchmaker = importutils.import_object(mm, *args, **kwargs)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/utils/importutils.py", line 38, in import_object
2014-11-24 11:34:04.079 TRACE nova return import_class(import_str)(*args, **kwargs)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/utils/importutils.py", line 27, in import_class
2014-11-24 11:34:04.079 TRACE nova __import__(mod_str)
2014-11-24 11:34:04.079 TRACE nova ImportError: No module named rpc.matchmaker_redis
2014-11-24 11:34:04.079 TRACE nova
n-cpu failed to start

Tags: zmq
Mehdi Abaakouk (sileht)
tags: added: zmq
Doug Hellmann (doug-hellmann)
Changed in oslo.messaging:
status: New → Confirmed
Revision history for this message
Li Ma (nick-ma-z) wrote :

It seems wrong configuration is set in devstack scripts.

Changed in devstack:
assignee: nobody → Li Ma (nick-ma-z)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to devstack (master)

Fix proposed to branch: master
Review: https://review.openstack.org/143293

Changed in devstack:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to devstack (master)

Reviewed: https://review.openstack.org/143293
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=ce1524d0843894b099c4abd20a64f92acad50c55
Submitter: Jenkins
Branch: master

commit ce1524d0843894b099c4abd20a64f92acad50c55
Author: Li Ma <email address hidden>
Date: Sun Dec 21 00:46:34 2014 -0800

    Fix proper oslo.messaging object for zeromq driver

    Currently, as almost all the rpc driver of projects
    have been switched to oslo.messaging, the object should
    be imported via oslo.messaging rather than its own rpc lib.

    Change-Id: I9633446e78cb5af21f61a26f6fb365a8ed57a85f
    Partially-Implements: blueprint zeromq
    Closes-Bug: #1395721

Changed in devstack:
status: In Progress → Fix Released
Revision history for this message
Oleksii Zamiatin (ozamiatin) wrote :
Download full text (3.8 KiB)

Redis is now ok, in nova have:
2015-02-05 17:25:37.348 TRACE nova Traceback (most recent call last):
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/bin/nova-network", line 9, in <module>
2015-02-05 17:25:37.348 TRACE nova load_entry_point('nova==2015.1.dev837', 'console_scripts', 'nova-network')()
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/cmd/network.py", line 72, in main
2015-02-05 17:25:37.348 TRACE nova db_allowed=CONF.conductor.use_local)
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/service.py", line 275, in create
2015-02-05 17:25:37.348 TRACE nova db_allowed=db_allowed)
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/service.py", line 157, in __init__
2015-02-05 17:25:37.348 TRACE nova self.conductor_api.wait_until_ready(context.get_admin_context())
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/conductor/api.py", line 315, in wait_until_ready
2015-02-05 17:25:37.348 TRACE nova timeout=timeout)
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/baserpc.py", line 62, in ping
2015-02-05 17:25:37.348 TRACE nova return cctxt.call(context, 'ping', arg=arg_p)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-02-05 17:25:37.348 TRACE nova retry=self.retry)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-02-05 17:25:37.348 TRACE nova timeout=timeout, retry=retry)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 926, in send
2015-02-05 17:25:37.348 TRACE nova return self._send(target, ctxt, message, wait_for_reply, timeout)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 917, in _send
2015-02-05 17:25:37.348 TRACE nova allowed_remote_exmods=self._allowed_remote_exmods)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 794, in _multi_send
2015-02-05 17:25:37.348 TRACE nova envelope, allowed_remote_exmods)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 726, in _call
2015-02-05 17:25:37.348 TRACE nova msg = msg_waiter.recv()
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 228, in recv
2015-02-05 17:25:37.348 TRACE nova return self.sock.recv_multipart(**kwargs)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 359, in recv_multipart
2015-02-05 17:25:37.348 TRACE nova return _Socket_recv_multipart(self, flags, copy, track)
2015-02-05 17:25:37.348 TRACE nova File "/usr/lib/python2.7/dist-packages/zmq/sugar/socket.py", line 265, in recv_multipart
2015-02-05 17:25:37.348 TRACE nova parts = [self.recv(flags, copy=copy, track=track)]
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python...

Read more...

Changed in oslo.messaging:
assignee: nobody → Oleksii Zamiatin (ozamiatin)
Revision history for this message
zhangjialong (zhangjl) wrote :

@Oleksii Zamiatin

You must start nova-conductor first, start the others next. Then the errors above would not be occured.

Revision history for this message
Oleksii Zamiatin (ozamiatin) wrote :

zhangjialong, thanks for your reply. Isn't it devstack issue to start services in apropriate order by default?

Revision history for this message
Li Ma (nick-ma-z) wrote :

I run the latest devstack, but everything goes well.

The output of the devstack:

Horizon is now available at http://10.88.88.2/
Keystone is serving at http://10.88.88.2:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: password
This is your host ip: 10.88.88.2

Revision history for this message
Oleksii Zamiatin (ozamiatin) wrote :

Could you please provide local.conf or write how do you configure it?

I replace rabbitmq with zeromq in stackrc and setup the variable ZEROMQ_MATCHMAKER="redis" is there anything else I forgot to do?

Revision history for this message
Li Ma (nick-ma-z) wrote :

Devstack installs oslo_messaging library from pip. The package from pip doesn't reflect the actual master branch of github. This is why you cannot make it work.

There's a critical bug fix that just merged in the upstream to make redis work [1]. I think it has not been released yet.

[1] https://github.com/openstack/oslo.messaging/commit/bd52d08efaa608244254a801c9737a0aed7434e0

Revision history for this message
Li Ma (nick-ma-z) wrote :

I modify the code by myself and finally devstack works.

Revision history for this message
Oleksii Zamiatin (ozamiatin) wrote :

Thanks for reply! Good to hear that it works. I'll check it and update the status.

Revision history for this message
Li Ma (nick-ma-z) wrote :

Well, I didn't notice that I changed the code before. Sorry for the late reply. You can try it later. If there's something wrong, please update this thread asap. Thanks.

Revision history for this message
Oleksii Zamiatin (ozamiatin) wrote :

Works fine on devstack.

local.conf:

[[local|localrc]]
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password

ZEROMQ_MATCHMAKER=redis
LIBS_FROM_GIT=oslo.messaging

    # core compute (glance / keystone / nova (+ nova-network))
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-net,n-cond,n-sch,n-xvnc,n-cauth
    # cinder
ENABLED_SERVICES+=,c-sch,c-api,c-vol
    # heat
ENABLED_SERVICES+=,h-eng,h-api,h-api-cfn,h-api-cw
    # dashboard
ENABLED_SERVICES+=,horizon
    # additional services
ENABLED_SERVICES+=,zeromq,tempest,mysql

Changed in oslo.messaging:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.