OpenStack Compute (nova)

Instances always get mapped into the last processed cell in conductor

Bug #1715493 reported by Matt Riedemann on 2017-09-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Dan Smith
	Pike	Fix Committed	High	Matt Riedemann

Bug Description

This was a regression introduced in Pike here: https://review.openstack.org/#/c/416521/

The schedule_and_build_instances method in conductor was split into two for loops, where the first loop creates the instance record in the cell database and the cell is looked up via the host mapping for the chosen host for that instance.

The problem is the 2nd for loop doesn't do the same cell lookup based on the host:

https://github.com/openstack/nova/blob/b79492f70257754f960eaf38ad6a3f56f647cb3d/nova/conductor/manager.py#L1023

It just re-uses the last set cell variable from the first for loop, so we could have a case where an instance is created in cell1, and then another instance is created in cell2, and then when the 2nd loop maps the first instance, it maps it to cell2 but it really lives in cell1.

Not to mention the BDMs and tags would be created in the wrong cell.

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-06: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/501452

Changed in nova:
assignee:	nobody → Dan Smith (danms)
status:	Triaged → In Progress

OpenStack Infra (hudson-openstack) on 2017-09-06

Changed in nova:
assignee:	Dan Smith (danms) → Matt Riedemann (mriedem)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-07: Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/501513

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-07:

Related fix proposed to branch: master
Review: https://review.openstack.org/501514

Matt Riedemann (mriedem) on 2017-09-07

Changed in nova:
assignee:	Matt Riedemann (mriedem) → Dan Smith (danms)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-07: Related fix merged to nova (master)

Reviewed: https://review.openstack.org/501513
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=53c2a979e25c6b017f2e8c7b8c357f275bb13a7a
Submitter: Jenkins
Branch: master

commit 53c2a979e25c6b017f2e8c7b8c357f275bb13a7a
Author: Matt Riedemann <email address hidden>
Date: Wed Sep 6 21:22:53 2017 -0400

Allow setting up multiple cells in the base TestCase

This lays the groundwork for other tests to get multiple
cells setup in the base TestCase by overriding a variable.

Change-Id: Ic27369dd3711e5b6bf11c98f742b5dc20a3a2799
Related-Bug: #1715493

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-07:

Reviewed: https://review.openstack.org/501514
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bcd1f1d8c833c4c12399ba84934a9a72a2bde122
Submitter: Jenkins
Branch: master

commit bcd1f1d8c833c4c12399ba84934a9a72a2bde122
Author: Matt Riedemann <email address hidden>
Date: Wed Sep 6 21:29:25 2017 -0400

Make ConductorTaskTestCase run with 2 cells

This lays the groundwork for ConductorTaskTestCase to run
with two cells which is needed for an upcoming fix.

    As a result, some mocks needed to be removed for targeting
    a cell context, but we still assert a targeted context is
    used where necessary, just in a different way. Some other
    DB-hitting operations needed to be mocked out in the unshelve
    tests since the instance being worked on isn't actually in
    a database, and these should have probably been mocked out
    anyway.

Change-Id: I6548548825c00a71f3d317ceb37be26da312be31
Related-Bug: #1715493

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-07: Fix merged to nova (master)

Reviewed: https://review.openstack.org/501452
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e801775959a8ae0f9c09767021a40b6a746d1928
Submitter: Jenkins
Branch: master

commit e801775959a8ae0f9c09767021a40b6a746d1928
Author: Dan Smith <email address hidden>
Date: Wed Sep 6 14:19:05 2017 -0700

Track which cell each instance is created in and use it consistently

    In conductor's instance create path we were using the last cell of the
    create loop for the second loop that created extra resources like BDMs,
    after the quota recheck.

Co-Authored-By: Matt Riedemann <email address hidden>

Change-Id: Iea8db52e7026166fb6623e0cc1aebdcc0af1d691
Closes-Bug: #1715493

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-07: Related fix proposed to nova (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/501818

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-07:

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/501819

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-07: Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/501820

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-08: Related fix merged to nova (stable/pike)

#10

Reviewed: https://review.openstack.org/501818
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9a791df1ecf5a60cd4e7bfedd0df8aae294d2df1
Submitter: Jenkins
Branch: stable/pike

commit 9a791df1ecf5a60cd4e7bfedd0df8aae294d2df1
Author: Matt Riedemann <email address hidden>
Date: Wed Sep 6 21:22:53 2017 -0400

Allow setting up multiple cells in the base TestCase

This lays the groundwork for other tests to get multiple
cells setup in the base TestCase by overriding a variable.

    Change-Id: Ic27369dd3711e5b6bf11c98f742b5dc20a3a2799
    Related-Bug: #1715493
    (cherry picked from commit 53c2a979e25c6b017f2e8c7b8c357f275bb13a7a)

tags:

added: in-stable-pike

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-08:

#11

Reviewed: https://review.openstack.org/501819
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=72e50bedd65787ebbfddf41759babebcb281e7fb
Submitter: Jenkins
Branch: stable/pike

commit 72e50bedd65787ebbfddf41759babebcb281e7fb
Author: Matt Riedemann <email address hidden>
Date: Wed Sep 6 21:29:25 2017 -0400

Make ConductorTaskTestCase run with 2 cells

This lays the groundwork for ConductorTaskTestCase to run
with two cells which is needed for an upcoming fix.

    Change-Id: I6548548825c00a71f3d317ceb37be26da312be31
    Related-Bug: #1715493
    (cherry picked from commit bcd1f1d8c833c4c12399ba84934a9a72a2bde122)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-08: Fix merged to nova (stable/pike)

#12

Reviewed: https://review.openstack.org/501820
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2ad865fb36fd92cf6d950fe05cd3d3eaa3488ed5
Submitter: Jenkins
Branch: stable/pike

commit 2ad865fb36fd92cf6d950fe05cd3d3eaa3488ed5
Author: Dan Smith <email address hidden>
Date: Wed Sep 6 14:19:05 2017 -0700

Track which cell each instance is created in and use it consistently

    In conductor's instance create path we were using the last cell of the
    create loop for the second loop that created extra resources like BDMs,
    after the quota recheck.

Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: Iea8db52e7026166fb6623e0cc1aebdcc0af1d691
    Closes-Bug: #1715493
    (cherry picked from commit e801775959a8ae0f9c09767021a40b6a746d1928)