kolla-ansible

DBNotAllowed error in nova compute log

  1. Series rocky
  2. Bug #1829705
Bug #1829705 reported by Lei Zhang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Unassigned
kolla-ansible 9.0.0 "Train"
Queens
New
Medium
Unassigned
Rocky
New
Medium
Unassigned
Stein
Fix Released
Medium
Unassigned
Train
Fix Released
Medium
Unassigned
kolla-ansible 9.0.0 "Train"

Bug Description

After deploying successfully, found some errors of database accessing in nova compute log:

2019-05-09 23:24:51.998 7 ERROR nova.context [req-24633cf0-350f-48f0-8462-ac307deffe3f - - - - -] Error gathering result from cell 00000000-0000-0000-0000-000000000000: DBNotAllowed: nova-compute
2019-05-09 23:24:51.998 7 ERROR nova.context Traceback (most recent call last):
2019-05-09 23:24:51.998 7 ERROR nova.context File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/context.py", line 437, in gather_result
2019-05-09 23:24:51.998 7 ERROR nova.context with target_cell(context, cell_mapping) as cctxt:
2019-05-09 23:24:51.998 7 ERROR nova.context File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2019-05-09 23:24:51.998 7 ERROR nova.context return self.gen.next()
2019-05-09 23:24:51.998 7 ERROR nova.context File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/context.py", line 406, in target_cell
2019-05-09 23:24:51.998 7 ERROR nova.context set_target_cell(cctxt, cell_mapping)
2019-05-09 23:24:51.998 7 ERROR nova.context File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/context.py", line 380, in set_target_cell
2019-05-09 23:24:51.998 7 ERROR nova.context get_or_set_cached_cell_and_set_connections()
2019-05-09 23:24:51.998 7 ERROR nova.context File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-05-09 23:24:51.998 7 ERROR nova.context return f(*args, **kwargs)
2019-05-09 23:24:51.998 7 ERROR nova.context File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/context.py", line 370, in get_or_set_cached_cell_and_set_connections
2019-05-09 23:24:51.998 7 ERROR nova.context db_connection_string)
2019-05-09 23:24:51.998 7 ERROR nova.context File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/api.py", line 80, in create_context_manager
2019-05-09 23:24:51.998 7 ERROR nova.context return IMPL.create_context_manager(connection=connection)
2019-05-09 23:24:51.998 7 ERROR nova.context File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cmd/common.py", line 50, in __call__
2019-05-09 23:24:51.998 7 ERROR nova.context raise exception.DBNotAllowed(service_name)
2019-05-09 23:24:51.998 7 ERROR nova.context DBNotAllowed: nova-compute
2019-05-09 23:24:51.998 7 ERROR nova.context
2019-05-09 23:24:52.000 7 ERROR nova.cmd.common [req-24633cf0-350f-48f0-8462-ac307deffe3f - - - - -] No db access allowed in nova-compute: File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/utils.py", line 810, in context_wrapper
    return func(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/context.py", line 437, in gather_result
    with target_cell(context, cell_mapping) as cctxt:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/context.py", line 406, in target_cell
    set_target_cell(cctxt, cell_mapping)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/context.py", line 380, in set_target_cell
    get_or_set_cached_cell_and_set_connections()
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
    return f(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/context.py", line 370, in get_or_set_cached_cell_and_set_connections
    db_connection_string)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/api.py", line 80, in create_context_manager
    return IMPL.create_context_manager(connection=connection)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cmd/common.py", line 46, in __call__
    stacktrace = "".join(traceback.format_stack())

devstack guys met similar issue before, see https://bugs.launchpad.net/devstack/+bug/1812398. They handled this by removing any database configuration from nova compute conf.

Revision history for this message
Mark Goddard (mgoddard) wrote :

Thanks for raising and for the Devstack hint. I have also seen this. Will try removing the DB conf.

Changed in kolla-ansible:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Lei Zhang (lei-a-zhang) wrote :

Hi Mark, I've submitted a fix for this, https://review.opendev.org/#/c/660032/, but got stuck by some CI failure, I'm figuring out what's wrong with the patch.

Revision history for this message
Mark Goddard (mgoddard) wrote :

Sorry, I didn't see your patch and proposed one of my own: https://review.opendev.org/#/c/660059/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by Lei Zhang (<email address hidden>) on branch: master
Review: https://review.opendev.org/660032

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/660059
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=002eec951f336b07ef3a7f277e9c0fe6aae9a952
Submitter: Zuul
Branch: master

commit 002eec951f336b07ef3a7f277e9c0fe6aae9a952
Author: Mark Goddard <email address hidden>
Date: Mon May 20 10:17:32 2019 +0100

    nova: Fix DBNotAllowed during compute startup

    backport: stein, rocky

    During startup of nova-compute, we see the following error message:

    Error gathering result from cell 00000000-0000-0000-0000-000000000000:
    DBNotAllowed: nova-compute

    This issue was observed in devstack [1], and fixed [2] by removing
    database configuration from the compute service.

    This change takes the same approach, removing DB config from nova.conf
    in the nova-compute* containers.

    [1] https://bugs.launchpad.net/devstack/+bug/1812398
    [2] https://opendev.org/openstack/devstack/commit/82537871376afe98a286e1ba424cf192ae60869a

    Change-Id: I18c99ff4213ce456868e64eab63a4257910b9b8e
    Closes-Bug: #1829705

Changed in kolla-ansible:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/663303

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/663304

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/rocky)

Reviewed: https://review.opendev.org/663304
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=153ce503fc4038ee1af558e89cc0176474e2548c
Submitter: Zuul
Branch: stable/rocky

commit 153ce503fc4038ee1af558e89cc0176474e2548c
Author: Mark Goddard <email address hidden>
Date: Mon May 20 10:17:32 2019 +0100

    nova: Fix DBNotAllowed during compute startup

    backport: stein, rocky

    During startup of nova-compute, we see the following error message:

    Error gathering result from cell 00000000-0000-0000-0000-000000000000:
    DBNotAllowed: nova-compute

    This issue was observed in devstack [1], and fixed [2] by removing
    database configuration from the compute service.

    This change takes the same approach, removing DB config from nova.conf
    in the nova-compute* containers.

    [1] https://bugs.launchpad.net/devstack/+bug/1812398
    [2] https://opendev.org/openstack/devstack/commit/82537871376afe98a286e1ba424cf192ae60869a

    Change-Id: I18c99ff4213ce456868e64eab63a4257910b9b8e
    Closes-Bug: #1829705
    (cherry picked from commit 002eec951f336b07ef3a7f277e9c0fe6aae9a952)

tags: added: in-stable-rocky
tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/663303
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=615028f354fa6566107164a833bb16a70d71ce7d
Submitter: Zuul
Branch: stable/stein

commit 615028f354fa6566107164a833bb16a70d71ce7d
Author: Mark Goddard <email address hidden>
Date: Mon May 20 10:17:32 2019 +0100

    nova: Fix DBNotAllowed during compute startup

    backport: stein, rocky

    During startup of nova-compute, we see the following error message:

    Error gathering result from cell 00000000-0000-0000-0000-000000000000:
    DBNotAllowed: nova-compute

    This issue was observed in devstack [1], and fixed [2] by removing
    database configuration from the compute service.

    This change takes the same approach, removing DB config from nova.conf
    in the nova-compute* containers.

    [1] https://bugs.launchpad.net/devstack/+bug/1812398
    [2] https://opendev.org/openstack/devstack/commit/82537871376afe98a286e1ba424cf192ae60869a

    Change-Id: I18c99ff4213ce456868e64eab63a4257910b9b8e
    Closes-Bug: #1829705
    (cherry picked from commit 002eec951f336b07ef3a7f277e9c0fe6aae9a952)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 7.1.1

This issue was fixed in the openstack/kolla-ansible 7.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/666096

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/queens)

Reviewed: https://review.opendev.org/666096
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=11b9eba2a48b793d00ba04d7ace8f644c2a14f97
Submitter: Zuul
Branch: stable/queens

commit 11b9eba2a48b793d00ba04d7ace8f644c2a14f97
Author: Mark Goddard <email address hidden>
Date: Mon May 20 10:17:32 2019 +0100

    nova: Fix DBNotAllowed during compute startup

    backport: stein, rocky

    During startup of nova-compute, we see the following error message:

    Error gathering result from cell 00000000-0000-0000-0000-000000000000:
    DBNotAllowed: nova-compute

    This issue was observed in devstack [1], and fixed [2] by removing
    database configuration from the compute service.

    This change takes the same approach, removing DB config from nova.conf
    in the nova-compute* containers.

    [1] https://bugs.launchpad.net/devstack/+bug/1812398
    [2] https://opendev.org/openstack/devstack/commit/82537871376afe98a286e1ba424cf192ae60869a

    Change-Id: I18c99ff4213ce456868e64eab63a4257910b9b8e
    Closes-Bug: #1829705
    (cherry picked from commit 002eec951f336b07ef3a7f277e9c0fe6aae9a952)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 8.0.0.0rc2

This issue was fixed in the openstack/kolla-ansible 8.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 6.2.2

This issue was fixed in the openstack/kolla-ansible 6.2.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.