| 2014-07-07 19:11:50 |
Noel Burton-Krahn |
bug |
|
|
added bug |
| 2014-07-14 14:10:34 |
Doug Hellmann |
bug task added |
|
oslo.messaging |
|
| 2014-07-14 14:10:42 |
Doug Hellmann |
bug task deleted |
oslo |
|
|
| 2014-07-17 13:57:32 |
Ruediger Rissmann |
bug |
|
|
added subscriber Ruediger Rissmann |
| 2014-07-23 17:18:44 |
Bogdan Dobrelya |
oslo.messaging: status |
New |
Confirmed |
|
| 2014-07-24 00:37:47 |
Koji Iida |
bug |
|
|
added subscriber Koji Iida |
| 2014-07-24 18:19:16 |
OpenStack Infra |
oslo.messaging: status |
Confirmed |
In Progress |
|
| 2014-07-24 18:19:16 |
OpenStack Infra |
oslo.messaging: assignee |
|
Noel Burton-Krahn (noel-burton-krahn) |
|
| 2014-08-20 18:17:10 |
Zhen Qin |
bug |
|
|
added subscriber Zhen Qin |
| 2014-08-26 07:57:09 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto MURATA |
| 2014-09-06 14:31:11 |
Tianpeng Wang |
bug |
|
|
added subscriber Tianpeng Wang |
| 2014-09-11 03:53:15 |
gustavo panizzo |
bug |
|
|
added subscriber gustavo panizzo |
| 2014-09-15 15:46:50 |
David TARDIVEL |
bug |
|
|
added subscriber TARDIVEL |
| 2014-09-21 06:54:22 |
Andrey Pavlov |
bug |
|
|
added subscriber Andrey Pavlov |
| 2014-09-22 12:51:55 |
OpenStack Infra |
oslo.messaging: assignee |
Noel Burton-Krahn (noel-burton-krahn) |
Andrey Pavlov (apavlov-e) |
|
| 2014-11-10 12:24:21 |
Tomoko Inoue |
bug |
|
|
added subscriber Tomoko Inoue |
| 2014-11-11 03:35:59 |
Jian Wen |
bug |
|
|
added subscriber Jian Wen |
| 2014-11-13 10:23:35 |
Mantissa James |
bug |
|
|
added subscriber Mantissa James |
| 2014-11-26 09:16:50 |
Dr. Jens Harbott |
bug |
|
|
added subscriber Dr. Jens Rosenboom |
| 2014-12-03 11:44:49 |
Mehdi Abaakouk |
oslo.messaging: status |
In Progress |
Incomplete |
|
| 2014-12-03 11:46:06 |
Mehdi Abaakouk |
oslo.messaging: status |
Incomplete |
In Progress |
|
| 2014-12-03 12:27:47 |
Mehdi Abaakouk |
oslo.messaging: status |
In Progress |
Incomplete |
|
| 2014-12-03 12:38:47 |
Andrey Pavlov |
oslo.messaging: assignee |
Andrey Pavlov (apavlov-e) |
|
|
| 2014-12-19 21:41:35 |
Vish Ishaya |
oslo.messaging: status |
Incomplete |
Confirmed |
|
| 2014-12-30 18:49:56 |
OpenStack Infra |
oslo.messaging: status |
Confirmed |
In Progress |
|
| 2014-12-30 18:49:56 |
OpenStack Infra |
oslo.messaging: assignee |
|
Vish Ishaya (vishvananda) |
|
| 2015-01-06 17:28:35 |
OpenStack Infra |
oslo.messaging: assignee |
Vish Ishaya (vishvananda) |
Mehdi Abaakouk (sileht) |
|
| 2015-01-06 17:43:18 |
Mehdi Abaakouk |
oslo.messaging: importance |
Undecided |
High |
|
| 2015-01-06 17:43:20 |
Mehdi Abaakouk |
oslo.messaging: milestone |
|
next-kilo |
|
| 2015-01-20 02:04:59 |
Shintaro Mizuno |
bug |
|
|
added subscriber Shintaro Mizuno |
| 2015-01-26 11:02:07 |
Andrea Rosa |
bug |
|
|
added subscriber Andrea Rosa |
| 2015-01-27 19:01:43 |
Doug Hellmann |
oslo.messaging: milestone |
1.6.0 |
next-kilo |
|
| 2015-01-29 17:21:12 |
OpenStack Infra |
oslo.messaging: status |
In Progress |
Fix Committed |
|
| 2015-02-25 16:11:37 |
Mehdi Abaakouk |
oslo.messaging: status |
Fix Committed |
Fix Released |
|
| 2015-03-05 07:33:56 |
Xav Paice |
bug |
|
|
added subscriber Xav Paice |
| 2015-03-26 01:29:18 |
OpenStack Infra |
tags |
|
in-stable-juno |
|
| 2015-04-15 10:16:00 |
James Page |
bug task added |
|
oslo.messaging (Ubuntu) |
|
| 2015-04-15 10:16:08 |
James Page |
nominated for series |
|
Ubuntu Vivid |
|
| 2015-04-15 10:16:08 |
James Page |
bug task added |
|
oslo.messaging (Ubuntu Vivid) |
|
| 2015-04-15 10:16:08 |
James Page |
nominated for series |
|
Ubuntu Trusty |
|
| 2015-04-15 10:16:08 |
James Page |
bug task added |
|
oslo.messaging (Ubuntu Trusty) |
|
| 2015-04-15 10:16:08 |
James Page |
nominated for series |
|
Ubuntu Utopic |
|
| 2015-04-15 10:16:08 |
James Page |
bug task added |
|
oslo.messaging (Ubuntu Utopic) |
|
| 2015-04-15 10:16:18 |
James Page |
oslo.messaging (Ubuntu Vivid): status |
New |
Fix Released |
|
| 2015-04-15 10:16:20 |
James Page |
oslo.messaging (Ubuntu Utopic): status |
New |
Fix Released |
|
| 2015-04-15 10:16:25 |
James Page |
oslo.messaging (Ubuntu Trusty): status |
New |
Confirmed |
|
| 2015-04-15 10:16:27 |
James Page |
oslo.messaging (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2015-04-17 11:53:52 |
Edward Hope-Morley |
description |
Icehouse
oslo-messaging 1.3.0 (stable/icehouse branch from github, c21a2c2)
rabbitmq-server 3.1.3
Nova rpc calls fail often after rabbit restarts. I've tracked it down to oslo/rabbit/kombu timing out if it's forced to reconnect to rabbit. The code below times out waiting for a reply if the topic has been used in a previous run. The reply always arrives the first time a topic is used, or if the topic is none. But, the second run with the same topic will hang with this error:
MessagingTimeout: Timed out waiting for a reply to message ID ...
This problem seems too basic to not be caught earlier in oslo, but the program below does really reproduce the same symptoms we see in nova when run against a live rabbit server.
Here's a log from a test run: https://gist.github.com/noelbk/12adcfe188d9f54f971d
#! /usr/bin/python
from oslo.config import cfg
import threading
from oslo import messaging
import logging
import time
log = logging.getLogger(__name__)
class OsloTest():
def test(self):
# The code below times out waiting for a reply if the topic
# has been used in a previous run. The reply always arrives
# the first time a topic is used, or if the topic is none.
# But, the second run with the same topic will hang with this
# error:
#
# MessagingTimeout: Timed out waiting for a reply to message ID ...
#
topic = 'will_hang_on_second_usage'
#topic = None # never hangs
url = "%(proto)s://%(user)s:%(password)s@%(host)s/" % dict(
proto = 'rabbit',
host = '1.2.3.4',
password = 'xxxxxxxx',
user = 'rabbit-mq-user',
)
transport = messaging.get_transport(cfg.CONF, url)
driver = transport._driver
target = messaging.Target(topic=topic)
listener = driver.listen(target)
ctxt={"context": True}
timeout = 10
def send_main():
log.debug("sending msg")
reply = driver.send(target,
ctxt,
{'send': 1},
wait_for_reply=True,
timeout=timeout)
# times out if topic was not None and used before
log.debug("received reply=%r" % (reply,))
send_thread = threading.Thread(target=send_main)
send_thread.daemon = True
send_thread.start()
msg = listener.poll()
log.debug("received msg=%r" % (msg,))
msg.reply({'reply': 1})
log.debug("sent reply")
send_thread.join()
if __name__ == '__main__':
FORMAT = '%(asctime)-15s %(process)5d %(thread)5d %(filename)s %(funcName)s %(message)s'
logging.basicConfig(level=logging.DEBUG, format=FORMAT)
OsloTest().test() |
Icehouse
oslo-messaging 1.3.0 (stable/icehouse branch from github, c21a2c2)
rabbitmq-server 3.1.3
Nova rpc calls fail often after rabbit restarts. I've tracked it down to oslo/rabbit/kombu timing out if it's forced to reconnect to rabbit. The code below times out waiting for a reply if the topic has been used in a previous run. The reply always arrives the first time a topic is used, or if the topic is none. But, the second run with the same topic will hang with this error:
MessagingTimeout: Timed out waiting for a reply to message ID ...
This problem seems too basic to not be caught earlier in oslo, but the program below does really reproduce the same symptoms we see in nova when run against a live rabbit server.
Here's a log from a test run: https://gist.github.com/noelbk/12adcfe188d9f54f971d
#! /usr/bin/python
from oslo.config import cfg
import threading
from oslo import messaging
import logging
import time
log = logging.getLogger(__name__)
class OsloTest():
def test(self):
# The code below times out waiting for a reply if the topic
# has been used in a previous run. The reply always arrives
# the first time a topic is used, or if the topic is none.
# But, the second run with the same topic will hang with this
# error:
#
# MessagingTimeout: Timed out waiting for a reply to message ID ...
#
topic = 'will_hang_on_second_usage'
#topic = None # never hangs
url = "%(proto)s://%(user)s:%(password)s@%(host)s/" % dict(
proto = 'rabbit',
host = '1.2.3.4',
password = 'xxxxxxxx',
user = 'rabbit-mq-user',
)
transport = messaging.get_transport(cfg.CONF, url)
driver = transport._driver
target = messaging.Target(topic=topic)
listener = driver.listen(target)
ctxt={"context": True}
timeout = 10
def send_main():
log.debug("sending msg")
reply = driver.send(target,
ctxt,
{'send': 1},
wait_for_reply=True,
timeout=timeout)
# times out if topic was not None and used before
log.debug("received reply=%r" % (reply,))
send_thread = threading.Thread(target=send_main)
send_thread.daemon = True
send_thread.start()
msg = listener.poll()
log.debug("received msg=%r" % (msg,))
msg.reply({'reply': 1})
log.debug("sent reply")
send_thread.join()
if __name__ == '__main__':
FORMAT = '%(asctime)-15s %(process)5d %(thread)5d %(filename)s %(funcName)s %(message)s'
logging.basicConfig(level=logging.DEBUG, format=FORMAT)
OsloTest().test()
---- ---- ---- ---- ----
[Impact]
* This patch along with those from LP #1400268 and LP #1408370 fixes rabbitmq reconnects
* We are backporting this to Icehouse since oslo.messaging 1.3.0
fails to reconnect to Rabbit properly, particularly nova-compute.
* This patch alond with it's dependencies metnioend above, will ensure that
multiple reconnect attempts happen by having connections timout and retry.
[Test Case]
* Start a service that uses oslo.messaging with rabbitmq e.g. nova-compute
* Stop rabbitmq while tail-F /var/log/nova/nova-compute.log
* Observe that nova-compute amqp times out and it is trying to reconnect
* Restart rabbitmq
* Observe that rabbitmq connection has re-established
[Regression Potential]
* None. I have tested in my local cloud environment and it appears to be
reliable. |
|
| 2015-04-17 11:55:15 |
Edward Hope-Morley |
attachment added |
|
oslo.messaging_1.3.0-0ubuntu1hf1338732v20150415.1.debdiff https://bugs.launchpad.net/oslo.messaging/+bug/1338732/+attachment/4378279/+files/oslo.messaging_1.3.0-0ubuntu1hf1338732v20150415.1.debdiff |
|
| 2015-04-17 11:55:33 |
Edward Hope-Morley |
oslo.messaging (Ubuntu Trusty): status |
Confirmed |
In Progress |
|
| 2015-04-17 16:46:34 |
Mehdi Abaakouk |
tags |
in-stable-juno |
icehouse-backport-potential in-stable-juno |
|
| 2015-04-19 22:40:48 |
Edward Hope-Morley |
attachment added |
|
oslo.messaging_1.3.0-0ubuntu1hf1338732v20150415.1.debdiff https://bugs.launchpad.net/oslo.messaging/+bug/1338732/+attachment/4379697/+files/oslo.messaging_1.3.0-0ubuntu1hf1338732v20150415.1.debdiff |
|
| 2015-04-20 11:29:30 |
James Page |
bug task added |
|
cloud-archive |
|
| 2015-04-23 14:52:17 |
James Page |
oslo.messaging (Ubuntu Trusty): assignee |
|
James Page (james-page) |
|
| 2015-04-23 14:52:56 |
James Page |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2015-04-29 13:20:53 |
Chris J Arges |
oslo.messaging (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2015-04-29 13:20:58 |
Chris J Arges |
bug |
|
|
added subscriber SRU Verification |
| 2015-04-29 13:20:59 |
Chris J Arges |
tags |
icehouse-backport-potential in-stable-juno |
icehouse-backport-potential in-stable-juno verification-needed |
|
| 2015-05-07 13:48:29 |
Ante Karamatić |
tags |
icehouse-backport-potential in-stable-juno verification-needed |
icehouse-backport-potential in-stable-juno verification-done |
|
| 2015-05-07 19:05:37 |
Launchpad Janitor |
oslo.messaging (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2015-05-07 19:05:43 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2015-05-08 15:32:35 |
Adam Collard |
bug |
|
|
added subscriber Landscape |
| 2015-05-12 13:32:08 |
James Page |
oslo.messaging (Ubuntu Utopic): status |
Fix Released |
New |
|
| 2015-05-12 13:32:11 |
James Page |
oslo.messaging (Ubuntu Utopic): importance |
Undecided |
High |
|
| 2015-05-15 14:16:16 |
OpenStack Infra |
tags |
icehouse-backport-potential in-stable-juno verification-done |
icehouse-backport-potential in-stable-icehouse in-stable-juno verification-done |
|
| 2015-05-24 12:14:33 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-proposed/oslo.messaging |
|
| 2015-05-24 15:30:41 |
Andrey Pavlov |
removed subscriber Andrey Pavlov |
|
|
|
| 2015-05-26 06:54:54 |
Launchpad Janitor |
oslo.messaging (Ubuntu Utopic): status |
New |
Confirmed |
|
| 2015-06-22 04:01:42 |
Yoshi Kadokawa |
bug |
|
|
added subscriber Yoshi Kadokawa |
| 2015-06-25 23:28:36 |
Shu Shen |
bug |
|
|
added subscriber Shu Shen |
| 2015-07-08 14:18:24 |
Chris J Arges |
oslo.messaging (Ubuntu Utopic): status |
Confirmed |
Fix Committed |
|
| 2015-07-08 14:18:27 |
Chris J Arges |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2015-07-08 14:18:37 |
Chris J Arges |
tags |
icehouse-backport-potential in-stable-icehouse in-stable-juno verification-done |
icehouse-backport-potential in-stable-icehouse in-stable-juno |
|
| 2015-07-08 14:18:37 |
Chris J Arges |
tags |
icehouse-backport-potential in-stable-icehouse in-stable-juno |
icehouse-backport-potential in-stable-icehouse in-stable-juno verification-needed |
|
| 2015-07-08 14:33:08 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/utopic-proposed/oslo.messaging |
|
| 2015-07-08 15:35:25 |
Bogdan Dobrelya |
bug |
|
|
added subscriber Davanum Srinivas (DIMS) |
| 2015-08-16 22:15:08 |
Szymon Dziech |
bug |
|
|
added subscriber Szymon Dziech |
| 2015-09-09 21:24:13 |
Jorge Niedbalski |
attachment added |
|
Utopic / Juno Patch https://bugs.launchpad.net/oslo.messaging/+bug/1338732/+attachment/4460080/+files/fix-lp-utopic-1338732.patch |
|
| 2015-09-09 21:24:35 |
Jorge Niedbalski |
oslo.messaging (Ubuntu Utopic): assignee |
|
Jorge Niedbalski (niedbalski) |
|
| 2015-09-09 21:24:38 |
Jorge Niedbalski |
oslo.messaging (Ubuntu Utopic): status |
Fix Committed |
In Progress |
|
| 2015-09-09 21:39:32 |
Jorge Niedbalski |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2015-09-09 21:39:40 |
Jorge Niedbalski |
bug |
|
|
added subscriber Jorge Niedbalski |
| 2015-09-10 21:14:25 |
Jorge Niedbalski |
branch linked |
|
lp:~niedbalski/ubuntu/trusty/oslo.messaging/fix-lp-1338732 |
|
| 2015-09-28 16:01:50 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-cloud-archive/ubuntu/trusty/oslo.messaging/juno |
|
| 2016-04-24 10:43:15 |
Rolf Leggewie |
oslo.messaging (Ubuntu Utopic): status |
In Progress |
Won't Fix |
|
| 2016-09-08 09:49:12 |
James Page |
cloud-archive: status |
New |
Fix Released |
|
| 2018-11-07 11:52:27 |
Hu Zhou |
bug |
|
|
added subscriber Hu Zhou |
