OpenStack Compute (nova)

Microversion 2.47 punches nova in its special place

Series pike
Bug #1719966

Bug #1719966 reported by Dan Smith on 2017-09-27

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Dan Smith
	Pike	Fix Committed	High	Matt Riedemann

Bug Description

Testing with 500 instances in ACTIVE, and 500 in ERROR state, using curl to pull all 1000 instances ten times in a row, 2.47 clearly shows a knee in the curve on average response time:

https://imgur.com/a/2lmiw

We should...fix that and stuff.

Tags:

Revision history for this message

Matt Riedemann (mriedem) wrote on 2017-09-27:

We were always joining on flavor in the database:

https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L58

And we were always pulling the embedded instance.flavor:

https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L263

So it would probably be something in here:

https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L248

Could it be the policy check on all 1000 instances?

tags:	added: api performance
Changed in nova:
status:	New → Confirmed
importance:	Undecided → High

OpenStack Infra (hudson-openstack) on 2017-09-27

Changed in nova:
assignee:	nobody → Dan Smith (danms)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-27: Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/507965

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-28: Fix merged to nova (master)

Reviewed: https://review.openstack.org/507948
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b1591d1080e4dbfbaf5351e009efa3abd11d88d4
Submitter: Jenkins
Branch: master

commit b1591d1080e4dbfbaf5351e009efa3abd11d88d4
Author: Dan Smith <email address hidden>
Date: Wed Sep 27 11:46:34 2017 -0700

Fix policy check performance in 2.47+

    In 2.47 we introduced inlined flavors, which include extra_specs if so
    authorized. The code was doing a policy check deep in the per-instance
    handler that generated this blob. For queries with a lot of instances,
    this policy check is a lot of overhead, especially as it introduces a
    check of the policy file on disk, in case it has changed.

This patch makes us calculate the permission flag once per list operation
and pass it down to the lower layers to honor.

Closes-Bug: #1719966
Change-Id: I036623ae7409c2e6c6a754b4c8d5c9120f777774

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-28: Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/507965
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bdd8b58351e9ebe69f1c5ab52bb5085e03be5202
Submitter: Jenkins
Branch: stable/pike

commit bdd8b58351e9ebe69f1c5ab52bb5085e03be5202
Author: Dan Smith <email address hidden>
Date: Wed Sep 27 11:46:34 2017 -0700

Fix policy check performance in 2.47+

This patch makes us calculate the permission flag once per list operation
and pass it down to the lower layers to honor.

    Closes-Bug: #1719966
    Change-Id: I036623ae7409c2e6c6a754b4c8d5c9120f777774
    (cherry picked from commit b1591d1080e4dbfbaf5351e009efa3abd11d88d4)