parallel volume-attachment requests might starve out nova-api for others

Bug #1930406 reported by Johannes Kulik
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Lee Yarwood

Bug Description

When doing volume-attachemnts, nova-api does an RPC-call (with a long_rpc_timeout) into nova-compute to reserve_block_device_name(). This takes a lock on the instance. If a volume-attachment was in progress, which also takes the instance-lock, nova-api's RPC-call needs to wait.

Having RPC-calls in nova-api, that can take a long time, will block the process handling the request. If a project does a lot of volume-attachments (e.g. for a k8s workload > 10 attachments per instance), this could starve out other users of nova-api by occupying all available processes.

When running nova-api with eventlet, a small number of processes can handle a lot of requests in parallel and some blocking rpc-calls don't matter too much.

When switching to uWSGI, the number of processes would have to be increased drastically to accommodate for that - unless it's possible to map those requests to threads and use a high number of threads instead.

What's the recommended way to run nova-api on uWSGI to handle this? Low number of processes with high number of threads to mimic eventlet?

Revision history for this message
Lee Yarwood (lyarwood) wrote :
Download full text (11.5 KiB)

So I think I can reproduce this in a basic devstack env with a simple 2 process uwsgi config and enable-threads = true:

$ cat /<email address hidden>

Description = Devstack <email address hidden>

RestartForceExitStatus = 100
NotifyAccess = all
Restart = always
KillMode = process
Type = notify
ExecReload = /usr/bin/kill -HUP $MAINPID
ExecStart = /usr/local/bin/uwsgi --procname-prefix nova-api --ini /etc/nova/nova-api-uwsgi.ini
User = stack
SyslogIdentifier = <email address hidden>

WantedBy =

$ cat /etc/nova/nova-api-uwsgi.ini

chmod-socket = 666
socket = /var/run/uwsgi/nova-api-wsgi.socket
lazy-apps = true
add-header = Connection: close
buffer-size = 65535
hook-master-start = unix_signal:15 gracefully_kill_them_all
thunder-lock = true
plugins = http,python3
enable-threads = true
worker-reload-mercy = 90
exit-on-reload = false
die-on-term = true
master = true
processes = 2
wsgi-file = /usr/local/bin/nova-api-wsgi

$ for i in $(seq 1 9); do openstack volume create --size 1 test-$i ; done
$ for i in $(seq 1 9); do openstack server add volume test test-$i & done
[1] 888087
[2] 888088
[3] 888089
[4] 888090
[5] 888091
[6] 888092
[7] 888093
[8] 888094
[9] 888095
[5] Done openstack server add volume test test-$i
[2] Done openstack server add volume test test-$i
[4] Done openstack server add volume test test-$i
[8]- Done openstack server add volume test test-$i
[9]+ Done openstack server add volume test test-$i
[1] Done openstack server add volume test test-$i
[3] Done openstack server add volume test test-$i
[6]- Done openstack server add volume test test-$i
[7]+ Done openstack server add volume test test-$i
$ openstack server event list test
| Request ID | Server ID | Action | Start Time |
| req-df477597-befd-4122-bc3d-d54e9b0daaae | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:10.000000 |
| req-17e8c4af-a435-4d91-854a-9308a19284d3 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:10.000000 |
| req-336f0d30-716a-4294-90d9-33869192e961 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:03.000000 |
| req-7b9d3b8a-e94d-446e-890f-216cc8f8d6c3 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:03.000000 |
| req-a2e80b18-113f-436f-8fa8-4c2373b9d831 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:59:57.000000 |
| req-8d24416c-b80d-4230-8cf4-9a979690e7ff | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:59:56.000000 |
| req-0d91f187-e4c7-4469-bd66-4880add4b2df | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:5...

Revision history for this message
Lee Yarwood (lyarwood) wrote :

I can't capture it in an update here but uwsgitop graphically demonstrates this as well.

I've wanted to drop the RPC call to reserve_block_device_name on the compute for a while as the virt drivers can't guarantee the assigned device name within the guestOS and it's just quicker to create our BDMs in the API workers before returning:

This would require a microversion as the response of os-volume_attachments would change, making device optional:

I can try to write up a spec for this in the coming days.

Revision history for this message
sean mooney (sean-k-mooney) wrote :

just an FYI the nova-api when run under uwsgi is still using eventlet monkey patching.
in the past it did not but it has for about 18month or more since we added the multi
cell sactter gatter code around Stine/train.
currently we do not recommend using multiple threads when usign nova-api with uwsgi

Revision history for this message
melanie witt (melwitt) wrote :

+1 what Sean said. We have had a known issue documented since Stein indicating that threads=1 has to be used otherwise it will be possible for oslo.messaging to attempt a reply on a different eventlet thread than the request came from and cause timeouts.

Unfortunately I'm not aware of anything we can do here other than increase the number of processes.

This might be yet another reason we will want to enable nova to run with regular threads -- then we wouldn't have to run oslo.messaging rpc with the eventlet executor and theoretically could run wsgi with threads > 1.

Lee Yarwood (lyarwood)
Changed in nova:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Lee Yarwood (lyarwood)
Revision history for this message
Johannes Kulik (jkulik) wrote :

Is there any update on this topic?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.