VGPUs is not recreated on host reboot

Thanks for reporting !

Oh shit, you're right, we can't lookup the existing mdev info [1] to know its parent PCI device since the mdev disappeared after rebooting...

[1] https://github.com/openstack/nova/blob/450213f/nova/virt/libvirt/driver.py#L816

So, honestly, there are no ways to know the parent PCI device since we don't persist mdevs and to be honest, that's not something Nova should do since it's a KVM/kernel issue.

Leaving the bug open until we figure out a good way to either document or fix this.

I guess we in that case should simply free all GPU-resources registered in placement for a certain compute-node when the compute-node boots? And give the VM's new GPU's when they boot?

I guess it is not an issue that they do not get the exact same GPU after a reboot; as the GPU looses state at reboot anyway?

Sylvain, Sean and I had a discussion about this and some of the possible solutions (long term and short term):

http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-04-15.log.html#t2021-04-15T13:12:29

A possible workaround for the trivial case where all GPUs are set with a unique type could be :
 - stop the compute service
 - lookup all the existing guest domains and fetch each of the mdev UUIDs
 - for each UUID, ask sysfs to recreate the mdev by firing this command :

echo $UUID > /sys/bus/pci/devices/$PCI_ADDRR/mdev_supported_types/$THE_RIGHT_TYPE/create

where $UUID is the mdev UUID, $PCI_ADDR the parent GPU and $THE_RIGHT_TYPE is the mdev type this pGPU was using.

Hope that helps, I'll amend the upstream docs with this limitation and I'll work on a patch for looking up the allocations to get the Resource Provider that matches, so I'll know the physical GPU and then be able to lookup the config option for finding the type.

Sylvain, I had been thinking a possible workaround might be shelve offloading the server and then unshelving it. Would that work or no?

Melanie: That is exactly what we're doing to workaround this in our environment. Work nicely for planned reboots. But of course, in the event of an unexpected reboot, we're still faced with a major manual cleanup job.

Yeah, shelving before rebooting does work as the unshelve will look at the allocations and then providing back mdevs for the instance.
But as saif by Lars, the problem here is that the nova-compute service can't be restarted after rebooting if you don't recreate the mdevs or removing the guests.

Hm, OK, I was thinking about shelve offloading after an unexpected reboot. AFAICT the placement allocations should remain after a reboot, so if you shelve offload, it will remove the placement allocations and then I was thinking the unshelve would recreate the placement allocations and mdevs when recreating the virt domain.

I take it that this is not actually what happens in practice?

Correct Melanie. Shelving after an unexpected reboot won't work, because it will require nova-compute to actually start to succeed.

I had other stuff to do so I stopped to work on this bugfix, but I'm now working on using init_instance() as we discussed here https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-05-05.log.html

IMO, I think this issue deserves higher importance...

I also think this should have a higher importance, especially as the bug means that the nova-compute service fails to start.

Related bug report about: Persisting mediated devices on reboot
https://bugzilla.redhat.com/show_bug.cgi?id=1699274

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/810220

1. libvirt 7.3 allows to define nodedev device
2. libvirt 6.5 starts to use mdevctl for handling mdev device
3. libvirt 7.6 adds support to autostart nodedev device but requires udev event which might need some work for containerized deployment

I've run in to this same issue using an Nvidia RTX A5000, however the Ampere cards present an additional issue. The SR-IOV VFs that the MDEVs use are not persisted after reboot either. I've worked around this by having a system service run before nova-compute to run the sriov-manage command provided by Nvidia to enable the VFs. I used "mdevctl define" to persist the mdev definitions and this has worked for me.

@zrsolis: We had to work aorund the SR-IOV issue as well, with the A100. Did it even easier with just a simple:

@reboot /usr/lib/nvidia/sriov-manage -e ALL

in root's crontab.

Works like a charm. I've actually been in contact with Nvidia Enterprise Support about persisting VFs over a reboot, and this was the only solution they came up with (which we actually implemented before we asked them anyways..)

But sriov-manage -e ALL doesn't re-create mdev devices with same UUIDs - it just activate virtual functions?

At least I don't have mdev devices after sriov-manage, so need to re-create them on A10. Maybe with MIG mode this does work though...

so the expecation here is that nova will creat the mdevs automatically on host reboot
there is justa bug in our currnt code.
you as an operator should not have to use mdevctl or any other host tooling to persist them.

we also dont support the use of mdevctl to precreat the mdevs today
if we want to move to that model we need to signifcatly change how we manage mdevs today.

@noonedeadpunk: sriov-manage does not create mdevs at all. It creates the SR-IOV VFs that you later can create mdevs on. So it's a prerequisite that imho nova should just assume is in place. I once tried to as Nvidia Enterprise Support for "fix" for this, but all they came up with was exactly the same @reboot cronjob-hack I already had in place.

MIG mode is still something nova does not support, I guess?

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/851924

"MIG mode is still something nova does not support, I guess?"

no nova support mig mode but we expect that you use udev or systemd services or other tolling to precreate the VFs sriov-manage or other tooling liek nvidia's mig-parted tool.

we have tested using mig backed mdevs downstream.

nova will create teh mdevs on teh VFs but you need to list the VFs not the PF in nova's config

it would be nice if the error message here described the details of what is missing, and saying it should be a udev rule, etc.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/864418