nova-manage floating commands are slow

Bug #1023560 reported by Joel Friedly
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Michael Still

Bug Description

nova-manage floating (list|create|delete) all execute very slowly. Creating/deleting realistic numbers of IPs can take hours, and simply listing them takes multiple seconds. These operations are essentially inserting/deleting/selecting from a mysql database, and should all be fast (< 1 second).

Reference values:
On a development cloud that we have, creating a /16 block of IPs (65,536 IPs) took around two hours, and simply doing the SQL SELECT to print them out with nova-manage floating list took 10 seconds.

Implementation details:
Looking at the nova-manage command, the create/delete commands spend the majority of their time in the db.floating_ip_create()/destroy() functions, spending roughly .2s to execute each on our machines.
The list command needs to be investigated more.
We suspect that the issues lies in using sqlalchemy inefficiently, most likely using a new transaction to update every single row. Assuming that sqlalchemy supports pipelining multiple row updates into a single transaction, we think this could improve execution speeds a lot.

Tags: nova-manage
Revision history for this message
Thierry Carrez (ttx) wrote :
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
tags: added: nova-manage
Revision history for this message
Michael Still (mikal) wrote :

Some indexes got added to the floating IPs table in schema 125, and I was hoping that that would improve things. A quick check with devstack says not though. I suspect the problem here is that each IP is created with its own call to the database.

Michael Still (mikal)
Changed in nova:
assignee: nobody → Michael Still (mikalstill)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/12315

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
Michael Still (mikal) wrote :

Listing 65,000 floating IPs on my laptop takes 7.5 seconds of wall time, which seems reasonable to me. Its a single database operation unless the floating IP has a fixed IP associated with it for what its worth. Deletion suffers from similar problems to create and I'll take a pass at fixing that now.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/12315
Committed: http://github.com/openstack/nova/commit/0bd4a04afb35d7e60b34b8a1aaab45b7805cad15
Submitter: Jenkins
Branch: master

commit 0bd4a04afb35d7e60b34b8a1aaab45b7805cad15
Author: Michael Still <email address hidden>
Date: Mon Sep 3 22:23:24 2012 +1000

    Speed up creating floating ips.

    Previously, we would make two separate DB calls for each floating ip
    we created in floating_ip_bulk_create. This was just a little slow
    (several hours to create a /16). This patch speeds up floating ip
    creation just a little:

    $ time bin/nova-manage floating create --ip_range=10.250.0.0/16

    real 0m48.766s
    user 0m34.342s
    sys 0m1.984s

    Resolves bug 1023560.

    Change-Id: Ie3663440d414b102111178899db0c571769edba8

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Michael Still (mikal) wrote :

I've changed this back to "in progress" because it still needs a delete speed up patch (which I will send today).

Changed in nova:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/12490

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/12490
Committed: http://github.com/openstack/nova/commit/5339141f04bf3ce470677e5fc3800ce4ac3fc0ae
Submitter: Jenkins
Branch: master

commit 5339141f04bf3ce470677e5fc3800ce4ac3fc0ae
Author: Michael Still <email address hidden>
Date: Tue Sep 4 20:46:00 2012 +1000

    Improve floating IP delete speed.

    This is similar to the change to improve bulk creation speed. Before:

    ./bin/nova-manage floating delete --ip_range 10.250.0.0/16
    337.20user 10.22system 44:07.10elapsed 13%CPU (0avgtext+0avgdata 119536maxresident)k
    400inputs+0outputs (4major+11704minor)pagefaults 0swaps

    After:

    ./bin/nova-manage floating delete --ip_range 10.250.0.0/16
    6.95user 0.16system 1:04.04elapsed 11%CPU (0avgtext+0avgdata 120016maxresident)k
    0inputs+0outputs (0major+11750minor)pagefaults 0swaps

    Resolves bug 1023560.

    Change-Id: If4f23222025cae88576f469022e0f7d907c9c4fa

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-rc1 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.