OpenStack Compute (nova)

nova service-list for new compute service causes TypeError with servicegroup_driver mc

  1. Series rocky
  2. Bug #1781880
Bug #1781880 reported by s10
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Zhenyu Zheng
Pike
Fix Committed
Medium
s10
Queens
Fix Committed
Medium
s10
Rocky
Fix Committed
Medium
Zhenyu Zheng

Bug Description

Description
===========
nova service-list for new compute service causes TypeError.
Related commit: https://git.openstack.org/cgit/openstack/nova/commit/?id=0df91a7f799060cd2e9b8a0adac1efacb974bcb3

Steps to reproduce
==================
1. Set servicegroup_driver=mc and enable_new_services=False in /etc/nova/nova.conf
2. Add new compute host, start nova-compute service. Service becomes disabled, and field updated_at is NULL in nova.services table in DB.
3. Execute nova service-list.

Expected result
===============
List of the nova services

Actual result
=============
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-cbd9f4ca-7634-4378-8dcc-5a03d9d4193f)

Environment
===========
OpenStack Pike

Logs
==============
Trace in the nova-api-os-compute logs:

ERROR nova.api.openstack.extensions [req-6034517c-bc29-4ebe-931e-8726fd934bee e05fb82b34cd4265a839f2482debb973 b5d263d6d7c24b84b335f4f00ae6d7c9] Unexpected exception in API method: TypeError: can't compare datetime.datetime to NoneType
TRACE nova.api.openstack.extensions Traceback (most recent call last):
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 336, in wrapped
TRACE nova.api.openstack.extensions return f(*args, **kwargs)
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 261, in index
TRACE nova.api.openstack.extensions _services = self._get_services_list(req, ['forced_down'])
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 115, in _get_services_list
TRACE nova.api.openstack.extensions for svc in _services]
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 83, in _get_service_detail
TRACE nova.api.openstack.extensions updated_time = self.servicegroup_api.get_updated_time(svc)
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/servicegroup/api.py", line 84, in get_updated_time
TRACE nova.api.openstack.extensions return self._driver.updated_time(member)
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/mc.py", line 81, in updated_time
TRACE nova.api.openstack.extensions if updated_time_in_db <= updated_time_in_mc:
TRACE nova.api.openstack.extensions TypeError: can't compare datetime.datetime to NoneType
TRACE nova.api.openstack.extensions
2018-07-16 11:40:16,437.437 2096 INFO nova.api.openstack.wsgi [req-6034517c-bc29-4ebe-931e-8726fd934bee e05fb82b34cd4265a839f2482debb973 b5d263d6d7c24b84b335f4f00ae6d7c9] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>

See original description
s10 (vlad-esten)
description: updated
description: updated
Pavel Abalikhin (anpavl)
Changed in nova:
status: New → Confirmed
s10 (vlad-esten)
description: updated
Revision history for this message
Matt Riedemann (mriedem) wrote :

Looks like the correct analysis of the bug. We don't test the MC server group driver or enable_new_services config option very well, or together (obviously).

Changed in nova:
importance: Undecided → Medium
status: Confirmed → Triaged
tags: added: memcache servicegroup
Matt Riedemann (mriedem)
tags: added: low-hanging-fruit
Zhenyu Zheng (zhengzhenyu)
Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
Revision history for this message
Matt Riedemann (mriedem) wrote :

It should be pretty easy to recreate this in a test which uses the MC driver and something like the oslo_cache.dict or some dogpile.cache.null backend.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/583927

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/583936

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/583927
Reason: This type of functional test won't work with the oslo_cache.dict since the cache will be a dict member variable of the server group API driver, and there are different server group API instances between the compute service and the API.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/583936
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0f4e5c442c0ab446274979dafda873c1e1c20585
Submitter: Zuul
Branch: master

commit 0f4e5c442c0ab446274979dafda873c1e1c20585
Author: Kevin_Zheng <email address hidden>
Date: Thu Jul 19 19:09:31 2018 +0800

    Fix service list for disabled compute using MC driver

    In 0df91a7f799060cd2e9b8a0adac1efacb974bcb3 a comparison
    between ``updated_at`` field from services table and update time
    from memcache was added if the servergroup_driver is set to be
    mc.

    This could be problematic if servergroup_driver=mc and
    ``[DEFAULT]/enable_new_services`` option in /etc/nova/nova.conf
    is set to be ``False`` at the same time, as the ``updated_at``
    field will always be ``Null`` in services record.

    This patch fixes the bug by returning ``updated_time_in_mc`` instead
    under the above mentioned scenario.

    Closes-Bug: #1781880

    Change-Id: I27160140cbccf3e6914aed2dcff54d547e446c1f

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/592310

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/592311

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/592337

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/592310
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=161d38679fec13bdb10e89536f7e0db43176396f
Submitter: Zuul
Branch: stable/rocky

commit 161d38679fec13bdb10e89536f7e0db43176396f
Author: Kevin_Zheng <email address hidden>
Date: Thu Jul 19 19:09:31 2018 +0800

    Fix service list for disabled compute using MC driver

    In 0df91a7f799060cd2e9b8a0adac1efacb974bcb3 a comparison
    between ``updated_at`` field from services table and update time
    from memcache was added if the servergroup_driver is set to be
    mc.

    This could be problematic if servergroup_driver=mc and
    ``[DEFAULT]/enable_new_services`` option in /etc/nova/nova.conf
    is set to be ``False`` at the same time, as the ``updated_at``
    field will always be ``Null`` in services record.

    This patch fixes the bug by returning ``updated_time_in_mc`` instead
    under the above mentioned scenario.

    Closes-Bug: #1781880

    Change-Id: I27160140cbccf3e6914aed2dcff54d547e446c1f
    (cherry picked from commit 0f4e5c442c0ab446274979dafda873c1e1c20585)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/592311
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3b17251da6e06260ee4b499dd18b46cc95f97fe5
Submitter: Zuul
Branch: stable/queens

commit 3b17251da6e06260ee4b499dd18b46cc95f97fe5
Author: Kevin_Zheng <email address hidden>
Date: Thu Jul 19 19:09:31 2018 +0800

    Fix service list for disabled compute using MC driver

    In 0df91a7f799060cd2e9b8a0adac1efacb974bcb3 a comparison
    between ``updated_at`` field from services table and update time
    from memcache was added if the servergroup_driver is set to be
    mc.

    This could be problematic if servergroup_driver=mc and
    ``[DEFAULT]/enable_new_services`` option in /etc/nova/nova.conf
    is set to be ``False`` at the same time, as the ``updated_at``
    field will always be ``Null`` in services record.

    This patch fixes the bug by returning ``updated_time_in_mc`` instead
    under the above mentioned scenario.

    Closes-Bug: #1781880

    Change-Id: I27160140cbccf3e6914aed2dcff54d547e446c1f
    (cherry picked from commit 0f4e5c442c0ab446274979dafda873c1e1c20585)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/592337
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b9ea6939db92662f4978866e9d48c43e67568922
Submitter: Zuul
Branch: stable/pike

commit b9ea6939db92662f4978866e9d48c43e67568922
Author: Vlad Gusev <email address hidden>
Date: Thu Aug 16 13:14:01 2018 +0300

    Fix service list for disabled compute using MC driver

    In 0df91a7f799060cd2e9b8a0adac1efacb974bcb3 a comparison
    between ``updated_at`` field from services table and update time
    from memcache was added if the servergroup_driver is set to be
    mc.

    This could be problematic if servergroup_driver=mc and
    ``[DEFAULT]/enable_new_services`` option in /etc/nova/nova.conf
    is set to be ``False`` at the same time, as the ``updated_at``
    field will always be ``Null`` in services record.

    This patch fixes the bug by returning ``updated_time_in_mc`` instead
    under the above mentioned scenario.

    Closes-Bug: #1781880

    Conflict is caused by I10b84a9cc7f0cc8ea213a377e4c4f16168b6e45b
    not being in Pike.

    Change-Id: I27160140cbccf3e6914aed2dcff54d547e446c1f
    (Cherry-picked from 0f4e5c442c0ab446274979dafda873c1e1c20585)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.2

This issue was fixed in the openstack/nova 18.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.1.6

This issue was fixed in the openstack/nova 16.1.6 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.7

This issue was fixed in the openstack/nova 17.0.7 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.