nova/compute/manager.py throws TypeError: 'NoneType' object is not iterable

Looking at what fails:

def _notify_volume_usage_detach(self, context, instance, bdm):
        if CONF.volume_usage_poll_interval <= 0:
            return

        vol_stats = []
        mp = bdm.device_name
        # Handle bootable volumes which will not contain /dev/
        if '/dev/' in mp:
            mp = mp[5:]
        try:
            vol_stats = self.driver.block_stats(instance, mp)
        except NotImplementedError:
            return

        LOG.debug("Updating volume usage cache with totals", instance=instance)
rd_req, rd_bytes, wr_req, wr_bytes, flush_ops = vol_stats

Clearly volume_usage_poll_interval has to be configured, which it's not by default. Which compute driver are you using that returns None but doesn't raise NotImplementedError?

Looks like the libvirt driver is the only driver that implements the block_stats() method, and it that method encounters an error it logs it and returns None:

https://github.com/openstack/nova/blob/6bf11e1dc14afad78b11d980c2544a3dc41579ff/nova/virt/libvirt/driver.py#L6364

Do you see one of those log messages in the compute log before the NoneType failure?

Fix proposed to branch: master
Review: https://review.openstack.org/609518

based on the calls to Log.info in

https://github.com/openstack/nova/blob/6bf11e1dc14afad78b11d980c2544a3dc41579ff/nova/virt/libvirt/driver.py#L6364

cd /var/log/nova
grep -e "Updating volume usage cache with totals" -e "Getting block stats failed" -e "Could not find domain in libvirt" *.log

There are no entries found, Is this the correct text to be looking for ?

bump.. Is this the correct text to be looking for in the log files ?
"Updating volume usage cache with totals"
"Getting block stats failed"
"Could not find domain in libvirt"

Reviewed: https://review.openstack.org/609518
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4da54c07861a1542a8e30a768d4506d3e81b5598
Submitter: Zuul
Branch: master

commit 4da54c07861a1542a8e30a768d4506d3e81b5598
Author: Matt Riedemann <email address hidden>
Date: Wed Oct 10 15:07:00 2018 -0400

    Fix NoneType error in _notify_volume_usage_detach

    If the driver.block_stats() method returns None, like the
    libvirt driver will if the guest is gone during an evacuate,
    we'll get a NoneType error trying to unpack the return value
    from the driver. Instead, simply return as if the driver
    raised NotImplementedError.

    Since handling None is changing the contract on the virt
    driver API, the docstring is updated to explain the acceptable
    return values of the driver method.

    Change-Id: I98a2785c07f7af02ad83650c72d9e1868290ece4
    Closes-Bug: #1796981

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/611326

Chris, yes those are the correct text to look for in the log files. But you said you had none of those entries, so that means there was no error encountered while the libvirt driver was getting block stats. It implies that the domain.blockStats() libvirt python binding returned None, but I don't know why it would do that. This is the method that's called at the lower level:

https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats

Re-reading the commit message of the fix, indeed it's probably that the method will return None if the guest is gone during the evacuate.

We handle InstanceNotFound in the block_stats method:

https://github.com/openstack/nova/blob/6bf11e1dc14afad78b11d980c2544a3dc41579ff/nova/virt/libvirt/driver.py#L6364-L6381

But this is probably a race where we are able to get a guest reference to the instance on the host _before_ it's gone (so we don't get InstanceNotFound) but by the time we call dev.blockStats(), the guest is gone, so we get None returned.

Anyway, the merged patch should fix the issue.

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/614868

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/614872

Reviewed: https://review.openstack.org/611326
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=940034c27c611b8b344f8371eb0458b16fb58f1d
Submitter: Zuul
Branch: stable/rocky

commit 940034c27c611b8b344f8371eb0458b16fb58f1d
Author: Matt Riedemann <email address hidden>
Date: Wed Oct 10 15:07:00 2018 -0400

    Fix NoneType error in _notify_volume_usage_detach

    If the driver.block_stats() method returns None, like the
    libvirt driver will if the guest is gone during an evacuate,
    we'll get a NoneType error trying to unpack the return value
    from the driver. Instead, simply return as if the driver
    raised NotImplementedError.

    Since handling None is changing the contract on the virt
    driver API, the docstring is updated to explain the acceptable
    return values of the driver method.

    Change-Id: I98a2785c07f7af02ad83650c72d9e1868290ece4
    Closes-Bug: #1796981
    (cherry picked from commit 4da54c07861a1542a8e30a768d4506d3e81b5598)

Reviewed: https://review.openstack.org/614868
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=62fbfdfa8f2dda1a5d9c3733b2f3da3a9d00c93a
Submitter: Zuul
Branch: stable/queens

commit 62fbfdfa8f2dda1a5d9c3733b2f3da3a9d00c93a
Author: Matt Riedemann <email address hidden>
Date: Wed Oct 10 15:07:00 2018 -0400

    Fix NoneType error in _notify_volume_usage_detach

    If the driver.block_stats() method returns None, like the
    libvirt driver will if the guest is gone during an evacuate,
    we'll get a NoneType error trying to unpack the return value
    from the driver. Instead, simply return as if the driver
    raised NotImplementedError.

    Since handling None is changing the contract on the virt
    driver API, the docstring is updated to explain the acceptable
    return values of the driver method.

    Change-Id: I98a2785c07f7af02ad83650c72d9e1868290ece4
    Closes-Bug: #1796981
    (cherry picked from commit 4da54c07861a1542a8e30a768d4506d3e81b5598)
    (cherry picked from commit 940034c27c611b8b344f8371eb0458b16fb58f1d)

Reviewed: https://review.openstack.org/614872
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d004b0ecfa6f20ceefd029f4ed09de191f0d76b3
Submitter: Zuul
Branch: stable/pike

commit d004b0ecfa6f20ceefd029f4ed09de191f0d76b3
Author: Matt Riedemann <email address hidden>
Date: Wed Oct 10 15:07:00 2018 -0400

    Fix NoneType error in _notify_volume_usage_detach

    If the driver.block_stats() method returns None, like the
    libvirt driver will if the guest is gone during an evacuate,
    we'll get a NoneType error trying to unpack the return value
    from the driver. Instead, simply return as if the driver
    raised NotImplementedError.

    Since handling None is changing the contract on the virt
    driver API, the docstring is updated to explain the acceptable
    return values of the driver method.

    Conflicts:
          nova/tests/unit/compute/test_compute_mgr.py

    NOTE(mriedem): The conflict is due to not having the
    test_get_scheduler_hints test from change
    I49ffebcd129990f1835f404d98b51732a32171eb which was
    added in Queens.

    Change-Id: I98a2785c07f7af02ad83650c72d9e1868290ece4
    Closes-Bug: #1796981
    (cherry picked from commit 4da54c07861a1542a8e30a768d4506d3e81b5598)
    (cherry picked from commit 940034c27c611b8b344f8371eb0458b16fb58f1d)
    (cherry picked from commit 62fbfdfa8f2dda1a5d9c3733b2f3da3a9d00c93a)

This issue was fixed in the openstack/nova 18.1.0 release.

This issue was fixed in the openstack/nova 16.1.7 release.

This issue was fixed in the openstack/nova 17.0.9 release.

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.