parallel volume-attachment requests might starve out nova-api for others

So I think I can reproduce this in a basic devstack env with a simple 2 process uwsgi config and enable-threads = true:

$ cat /<email address hidden>

[Unit]
Description = Devstack <email address hidden>

[Service]
RestartForceExitStatus = 100
NotifyAccess = all
Restart = always
KillMode = process
Type = notify
ExecReload = /usr/bin/kill -HUP $MAINPID
ExecStart = /usr/local/bin/uwsgi --procname-prefix nova-api --ini /etc/nova/nova-api-uwsgi.ini
User = stack
SyslogIdentifier = <email address hidden>

[Install]
WantedBy = multi-user.target

$ cat /etc/nova/nova-api-uwsgi.ini

[uwsgi]
chmod-socket = 666
socket = /var/run/uwsgi/nova-api-wsgi.socket
lazy-apps = true
add-header = Connection: close
buffer-size = 65535
hook-master-start = unix_signal:15 gracefully_kill_them_all
thunder-lock = true
plugins = http,python3
enable-threads = true
worker-reload-mercy = 90
exit-on-reload = false
die-on-term = true
master = true
processes = 2
wsgi-file = /usr/local/bin/nova-api-wsgi

$ for i in $(seq 1 9); do openstack volume create --size 1 test-$i ; done
$ for i in $(seq 1 9); do openstack server add volume test test-$i & done
[1] 888087
[2] 888088
[3] 888089
[4] 888090
[5] 888091
[6] 888092
[7] 888093
[8] 888094
[9] 888095
[5] Done openstack server add volume test test-$i
[2] Done openstack server add volume test test-$i
[4] Done openstack server add volume test test-$i
[8]- Done openstack server add volume test test-$i
[9]+ Done openstack server add volume test test-$i
[1] Done openstack server add volume test test-$i
[3] Done openstack server add volume test test-$i
[6]- Done openstack server add volume test test-$i
[7]+ Done openstack server add volume test test-$i
$ openstack server event list test
+------------------------------------------+--------------------------------------+---------------+----------------------------+
| Request ID | Server ID | Action | Start Time |
+------------------------------------------+--------------------------------------+---------------+----------------------------+
| req-df477597-befd-4122-bc3d-d54e9b0daaae | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:10.000000 |
| req-17e8c4af-a435-4d91-854a-9308a19284d3 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:10.000000 |
| req-336f0d30-716a-4294-90d9-33869192e961 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:03.000000 |
| req-7b9d3b8a-e94d-446e-890f-216cc8f8d6c3 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:03.000000 |
| req-a2e80b18-113f-436f-8fa8-4c2373b9d831 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:59:57.000000 |
| req-8d24416c-b80d-4230-8cf4-9a979690e7ff | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:59:56.000000 |
| req-0d91f187-e4c7-4469-bd66-4880add4b2df | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:5...

I can't capture it in an update here but uwsgitop graphically demonstrates this as well.

I've wanted to drop the RPC call to reserve_block_device_name on the compute for a while as the virt drivers can't guarantee the assigned device name within the guestOS and it's just quicker to create our BDMs in the API workers before returning:

https://docs.openstack.org/nova/latest/reference/attach-volume.html

This would require a microversion as the response of os-volume_attachments would change, making device optional:

https://docs.openstack.org/api-ref/compute/?expanded=attach-a-volume-to-an-instance-detail#attach-a-volume-to-an-instance

I can try to write up a spec for this in the coming days.

just an FYI the nova-api when run under uwsgi is still using eventlet monkey patching.
in the past it did not but it has for about 18month or more since we added the multi
cell sactter gatter code around Stine/train.
currently we do not recommend using multiple threads when usign nova-api with uwsgi

+1 what Sean said. We have had a known issue documented since Stein indicating that threads=1 has to be used otherwise it will be possible for oslo.messaging to attempt a reply on a different eventlet thread than the request came from and cause timeouts.

Unfortunately I'm not aware of anything we can do here other than increase the number of processes.

This might be yet another reason we will want to enable nova to run with regular threads -- then we wouldn't have to run oslo.messaging rpc with the eventlet executor and theoretically could run wsgi with threads > 1.

Is there any update on this topic?