oslo.messaging

metadata-agent fills up log-files with " TypeError: 'NoneType' object has no attribute '__getitem__'"

Bug #1496774 reported by L00nix
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned
OpenStack Compute (nova)
New
Undecided
Unassigned
neutron
Expired
Undecided
Unassigned
oslo.messaging
Incomplete
Undecided
Unassigned

Bug Description

Sience Monday, the log file of metadata-agent are getting full with the following error / trace:

2015-09-17 11:06:00.187 8277 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute '__getitem__'
2015-09-17 11:06:00.187 8276 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute '__getitem__'
2015-09-17 11:06:00.186 8267 ERROR oslo_messaging._drivers.amqpdriver [-] Failed to process incoming message, retrying...
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver Traceback (most recent call last):
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 197, in poll
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver self.conn.consume(limit=1)
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1172, in consume
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver six.next(it)
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1083, in iterconsume
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver error_callback=_error_callback)
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 870, in ensure
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver ret, channel = autoretry_method()
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 453, in _ensured
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver return fun(*args, **kwargs)
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 520, in __call__^C
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver self.revive(create_channel())
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 251, in channel
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver chan = self.transport.create_channel(self.connection)^C
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in create_channel
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver return connection.channel()
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 279, in channel
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver return self.channels[channel_id]
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver TypeError: 'NoneType' object has no attribute '__getitem__'
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver

There was no change in the configuration before, i'm not exactly sure why it does that, the configuration looks fine and all good.

I have also a 100% load on the server.

See original description
L00nix (mamudi92)
description: updated
Revision history for this message
Silvan Kaiser (2-silvan) wrote :

We have the same issue on a kilo-release RDO installation, except for the high load.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

It looks like a bug in oslo.messaging. The self.channels var is set to None somehow.

Revision history for this message
Gary Kotton (garyk) wrote :

This was a result of the change https://review.openstack.org/#q,7267d75fdd3f90af759d71e9490cd41d41ba6d98,n,z
If you are not running a plugin that is in the main tree you need to add the following:
        self.conn.create_consumer(topics.REPORTS,
                                  [agents_db.AgentExtRpcCallback()],
                                  fanout=False)

Does this bug still affect you?

Changed in neutron:
status: New → Confirmed
status: Confirmed → Incomplete
Davanum Srinivas (DIMS) (dims-v)
Changed in oslo.messaging:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.