Nova API in Horizon sometimes act as a bottleneck

Bug #1442310 reported by Timur Sufiev
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned
OpenStack Dashboard (Horizon)
Confirmed
Medium
Unassigned

Bug Description

The scale testing of Horizon faced a several problems with a lot of data being received from Nova side when Horizon actually doesn't need so much data.

First case is getting the volumes list from cinder, see https://github.com/openstack/horizon/blob/2015.1.0b3/openstack_dashboard/dashboards/project/volumes/tabs.py#L90 - here all the instances are being requested from Nova side just to show the names of Instances that Cinder volumes are attached to. In cases when the total number of Instances is far more bigger than the number of Volumes this causes a serious performance issues in Horizon where it shouldn't.

Second case is getting Floating IPs list from Neutron, https://github.com/openstack/horizon/blob/2015.1.0b3/openstack_dashboard/dashboards/project/access_and_security/tabs.py#L109 - here we again request all the instances from Nova just to show their names for some Floating IPs (which number could be truly inferior to the number of Instances).

Both cases could be avoided if we were able to request details for a limited set of servers (specified by their IDs) from Nova.

Tags: large-scale
Revision history for this message
jichenjc (jichenjc) wrote :

not very familiar with horizon, but If I understand it correctly, you want nova to return the instances and do the query in horizon side , instead , specify some options to nova and ask nova to query ?

there is a 'search option' but your request may not be there currently ... wait for response and wait for more confirm

Changed in nova:
assignee: nobody → jichenjc (jichenjc)
Revision history for this message
Timur Sufiev (tsufiev-x) wrote :

Yes, it is correct: instead of querying Nova for _all_ existing instances and then picking the required ones at Horizon side, I'd rather specify in request to Nova which instances exactly (preferably by `instance_id`) I need in Horizon.

Revision history for this message
jichenjc (jichenjc) wrote :

there is a rule that API changes need a spec
let me draft one and see someone has opinion and sugestions, will paste the link here later

Changed in nova:
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
Timur Sufiev (tsufiev-x) wrote :

Okay, thank you very much for the efforts! Looking forward to the spec :).

Revision history for this message
jichenjc (jichenjc) wrote :

I dig into code more and got several questions:

1) seems the logic of horizon is get volumes and instances then use local loop
if we offer an option to search, e.g filter instances by each volume ,should this increase the
REST API request and db hit? maye performance is not that good?

2) Also, seems the attach detach info is recorded at cinder db ,should it be better to
submit this to cinder instead of nova ?

both apply to 2nd question on floating IP, thanks

Revision history for this message
Timur Sufiev (tsufiev-x) wrote :

Sorry for the long time it took me to answer, I was being distracted by other tasks. So, regarding you you questions...

1. The data received from Cinder already contains instance ids. The only reason Horizon does a separate call to Nova is that he needs to get these instances' names (and possibly some other metadata). Asking the Nova to filter the instance list by volumes seems like a duplication of work to me ('cause we already know these instances), passing just instance ids would suffice.

2. Most likely Cinder doesn't know about instances' names, that's why Horizon does a request to Nova. This architecture is okay IMO, because each OpenStack component should do its _own_ job, with Horizon aggregating the data across many components just for the user's convenience.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to horizon (master)

Fix proposed to branch: master
Review: https://review.openstack.org/213260

Changed in horizon:
status: New → In Progress
Timur Sufiev (tsufiev-x)
tags: added: large-scale
jichenjc (jichenjc)
Changed in nova:
assignee: jichenjc (jichenjc) → nobody
Changed in nova:
assignee: nobody → Sergey Nikitin (snikitin)
Changed in horizon:
assignee: Timur Sufiev (tsufiev-x) → Itxaka Serrano (itxakaserrano)
Timur Sufiev (tsufiev-x)
Changed in horizon:
assignee: Itxaka Serrano (itxakaserrano) → Timur Sufiev (tsufiev-x)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to horizon (master)

Reviewed: https://review.openstack.org/213260
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=ed076e623458467c9dc4f2a9f49f3abb8466d562
Submitter: Jenkins
Branch: master

commit ed076e623458467c9dc4f2a9f49f3abb8466d562
Author: Timur Sufiev <email address hidden>
Date: Fri Aug 14 20:20:45 2015 +0300

    Do not make unnecessary calls to Nova from FIPs and Volumes tabs

    If none of the floating IPs listed within Horizon are attached to an
    instance, do not request a list of Nova instances (because we have no
    use for it). Apply the same behavior for Volumes tab (both Admin and
    Project). This patch both saves us from making unnecessary and
    potentially expensive call to Nova and lays the ground for a more
    selective request of instance details from Nova. The latter will be
    possible once Nova supports filtering instance details by >1
    instance_id. For reference see
    https://blueprints.launchpad.net/nova/+spec/get-multi-servers-filter

    Change-Id: Ie7563b9e03b286b4cf51507463213162af3383b1
    Partial-Bug: #1442310

Revision history for this message
Sean Dague (sdague) wrote :

What would the impact be to actually do the GET /servers/{id} when needed here? That's the thing I don't understand or see the implications of.

Revision history for this message
Andrea Rosa (andrea-rosa-m) wrote :

@sergey are you working on this bug? If so please mark it as "in progress" otherwise if you do not have any plans to work on it anytime soon, please assign the ticket to 'nobody'. Thanks

Revision history for this message
Maciej Szankin (mszankin) wrote :

Removing assignee due to lack of visible progress for a period of over one month.

Changed in nova:
assignee: Sergey Nikitin (snikitin) → nobody
Akihiro Motoki (amotoki)
Changed in horizon:
assignee: Timur Sufiev (tsufiev-x) → nobody
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.