n-cpu raising MessageUndeliverable when replying to RPC call

I think there is some conflation of terminology here.

RabbitMQ *does* have the "direct reply-to" feature described here:

https://www.rabbitmq.com/direct-reply-to.html

However, in the oslo.messaging rabbitmq driver, the references to "direct" is not referring to the same thing. Instead, it's simply differentiating the "direct" exchange type from the others, "fanout" and "topic" ("headers" is unused):

https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchanges

As far as I can tell, within the intended RPC architecture, the mandatory flag is working as intended here. There is no special reply-to shortcut being used. The RPC sender (should) declare the reply_XXX queue/binding/exchange triplet that the RPC server uses to return the reply. However if the server can't send the reply back for some reason (error somewhere in that exchange->binding->queue flow) then we *do* want to raise the error, otherwise we would blackhole the msg and the sender would never know it was lost.

I've seen this same/similar behavior in the past but I've never been able to reproduce it successfully. See for example from 3+ years ago:

https://bugzilla.redhat.com/show_bug.cgi?id=1484543#c5

Something happens to break the normal exchange->binding->queue flow. I'm not sure if this is something internal to RabbitMQ where object(s) get lost/corrupted during failover? Or perhaps this is a bug in oslo.messaging/kombu/pyamqp wherein the necessary objects are not properly redeclared on reconnection.

Thanks for the input John. You are correct - the current implementation of oslo.messaging does not use the special Direct Reply To feature provided by rabbitmq [0]. Whether or not it should is an entire discussion in itself :D

Currently oslo.messaging uses a "somewhat long lived" reply queue that is associated with the direct exchange. It is "somewhat long lived" in that its lifecycle is associated with the client's transport, and therefore will be used for all requests issued by the client (each individual request is identified by a correlation id - essentially multiplexing replies on the same queue).

Back the the issue at hand: if I understand you correctly oslo.messaging's current use of the mandatory flag - to force an error exception on the server when the reply cannot be delivered - is correct. Logically it seems correct to me as well.

However from the user's point of view (in the case nova) it's a breaking API change. Mea culpa - turning on the mandatory feature by default should not have been done without making a whole lot more public noise than I did.

Sounds like the best path forward would be to turn off the mandatory behavior by default (it's configurable) until folks have a chance to update their code to handle the new behavior.

[0] https://www.rabbitmq.com/direct-reply-to.html

Thanks John for your feedback.

You're right I was misled by the terminology.

I agree with Ken, the mandatory flag is a good feature but the communication around that wasn't enough for our users.

I think that we should provide some examples on how to handle this kind of failure on the user side (service etc...), it could be fixed by providing some doc patches.

Do you've any idea of scenarios that we should provide on our examples?

Concerning nova I don't think that we broke something (by adding the mandatory flag). if I correctly understand John's analyze the issue could come from a connection issue or from an internal server issue, so even without the flag nova would face more or less the same issue.

So on one side I think we should improve our doc and our communication about this feature, and on the other side, I think we should provide some workaround to execute in the case if we face a similar scenario in the future. Apparently this kind of issue is hard to reproduce so it could be worth to provide a reusable workaround publicly referenced somewhere (in our doc, downstream?).

@John do you have some idea of workarounds to restore similar situation?

I am seeing same behavior with senlin project, Here is the detail - https://bugs.launchpad.net/senlin/+bug/1909016

When i am trying to add following value in /etc/senlin/senlin.conf

[DEFAULT]
transport_url = rabbit://senlin:94d7aecb853145779db8f1dcb@10.65.6.176:5671//senlin?ssl=1

[oslo_messaging_rabbit]
ssl = True
direct_mandatory_flag = False

I am getting error in logs

Dec 22 14:43:26 os-lab-infra-1-senlin-container-16f24bbe senlin-conductor[8591]: 2020-12-22 14:43:25.999 8591 ERROR oslo_service.service [-] Error starting t
hread.: oslo_config.cfg.ConfigFileValueError: Value for option direct_mandatory_flag from LocationInfo(location=<Locations.user: (4, True)>, detail='/etc/senlin/senlin.conf') is not
 valid: invalid literal for int() with base 10: 'False'

re: Ken's comment:

"""
However from the user's point of view (in the case nova) it's a breaking API change. Mea culpa - turning on the mandatory feature by default should not have been done without making a whole lot more public noise than I did.
"""

I agree with this. The main issue is that oslo.messaging will now raise a MessageUndeliverable exception with the mandatory flag set. This exception was added here:

https://github.com/openstack/oslo.messaging/commit/c50076b4efb79cef46d618d6d80eecbcebb72898

And then later started getting raised when the mandatory flag was added to direct_send here:

https://github.com/openstack/oslo.messaging/commit/b7e9faf6590086b79f9696183b5c296ecc12b7b6

This is the detail that is API-breaking; nova and the like are not prepared to catch this exception and handle it.

However, we already do almost the exact same thing presently, albeit in a slightly different manner.

In the base amqpdriver here:

https://github.com/openstack/oslo.messaging/blob/e44c988/oslo_messaging/_drivers/amqpdriver.py#L151-L169

We catch AMQPDestinationNotFound and loop for missing_destination_retry_timeout seconds. This is accomplished by setting the passive=True flag on the exchange, and then probing to see if the exchange exists by issuing a exchange.declare. If the exchange is already present, exchange.declare-ok is returned. If the exchange is not present, a channel error is raised and the exchange is *not* declared. This is so the replying end can probe to see when the original requester re-establishes itself (it is on the sender to declare the exchange).

I think for now we could get away with simply catching the MessageUndeliverable exception in the same place, and we could loop for the same timeout while it remains undeliverable. The only problem I see is that the blueprint spec for the transport options is there explicitly so that the rpc client itself can declare at_least_once=True and expect to get back MessageUndeliverable if that fails. So we could keep track of the transport_options originally provided by the client. If the client explicitly set at_least_once=True, then when MessageUndeliverable is raised, we should re-raise it back to the client, with the assumption that it knows how to handle it (possibly after retrying some number of times?). However, if the client did not explicitly ask for it (the current case where nova and such do not use this API), then we can still use the functionality within oslo.messaging, but we need to catch it and not re-raise it, the same way AMQPDestinationNotFound works currently.

+1 John that's a great idea.

Hello,

Today I submitted changes related to John's proposition (c.f PS3 https://review.opendev.org/c/openstack/oslo.messaging/+/764776/3 ).

These changes are still "WIP" to allow us to discuss about the related details of the implementation.

The first that we could discuss about is can we drop the `direct_mandatory_flag` in favor of `at_least_once`.

I think it doesn't really make sense to keep the both params flags in parallel.

Either we want to re-raise the `MessageUndelivered` exception or to loop over a timeout, but I don't think we want to stop to use the mandatory flag option, isn't? For this reason I think we can simply define the `at_least_once` option as did in PS3 and simply abandon the `direct_mandatory_flag` option.

Also, notice that I tried to simulate the `MessageUndelivered` scenario by using oslo.messaging's simulator, unfortunatelly I couldn't reproduce the MessageUndeliverable exception.

Find bellow the followed process, maybe you will have an idea oon how to simulate that.

1. Run a rabbitmq server and allow to access to the dashboard to monitor things:

```
$ sudo podman run -d -e RABBITMQ_NODENAME=rabbitmq \
     -p 5672:5672 \
     -p 15672:15672 \
     --name rabbitmq rabbitmq:3-management
```

2. Prepare some configuration to use with our simulator:

```
# tuto.conf
[DEFAULT]
transport_url = rabbit://localhost/
at_least_once = False
```

3. Run the RPC server

```
$ tox -e venv -- python tools/simulator.py --config-file ./tuto.conf -d rpc-server -w 40
```

Notice that I asked to the server to wait before answer to allow me to manually delete the reply queue on the rabbitmq server side.

4. Run the RPC client

```
$ tox -e venv -- python tools/simulator.py --config-file ./tuto.conf -d rpc-client
```

5. Delete the corresponding reply queue

```
$ sudo podman exec -it rabbitmq rabbitmqctl delete_queue reply_<id>
```

Unfortunatelly I can't fall into the expected situation by following this process...

I suppose that some other mechanisms catch that something went wrong with this scenario and it could explain why I didn't reach my expected scenario.

Please let me know if these changes fit our expectations and if they make sense to you or if they needs changes.

Hi Hervé,

We can consistently reproduce the MessageUndelivered error in nova-compute when trying to attach 7 cinder volumes right after a VM is created, all via a heat template.

Maybe you can give use case a try.

The way to get rid of the error is: (1) Set direct_mandatory_flag to 0 for nova-compute, and use a very big rpc_response_timeout (240) in nova_api.

The same heat template runs OK in Rocky, we are also looking at why it fails now in Train.

Jing

This is to provide update for our issue,

(1) Use a very big rpc_response_time for nova_api can not reliably get rid of the issue

(2) Use single mod_wsgi thread for nova_api is able to resolve the issue reliably, plus the other "difficult" cinder volume test cases that run in Rocky are able to work in Train.

The following ticket filtered out the rabbitmq hearbeat errors in nova_api.log, but luckily the errors are still shown in rabbitmq.log, that helped us to nail down the issue.

https://bugs.launchpad.net/nova/+bug/1825584

Jing

This issue was fixed in the openstack/oslo.messaging 12.7.1 release.

This issue was fixed in the openstack/oslo.messaging 12.1.6 release.

This issue was fixed in the openstack/oslo.messaging 12.5.2 release.

This issue was fixed in the openstack/oslo.messaging 10.2.4 release.

This is being fixed via LP:1940858