performance of plugin_rpc.get_routers is bad

Fix proposed to branch: master
Review: https://review.openstack.org/174806

Fix proposed to branch: master
Review: https://review.openstack.org/174812

Fix proposed to branch: master
Review: https://review.openstack.org/174815

Fix proposed to branch: master
Review: https://review.openstack.org/174818

Related fix proposed to branch: master
Review: https://review.openstack.org/174821

Fix proposed to branch: master
Review: https://review.openstack.org/174842

Fix proposed to branch: master
Review: https://review.openstack.org/174843

Fix proposed to branch: master
Review: https://review.openstack.org/174844

Fix proposed to branch: master
Review: https://review.openstack.org/174845

Fix proposed to branch: master
Review: https://review.openstack.org/174846

Reviewed: https://review.openstack.org/174842
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3310c3c3d4c05c0d13f32f08f978ba4813e2a39a
Submitter: Jenkins
Branch: master

commit 3310c3c3d4c05c0d13f32f08f978ba4813e2a39a
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:28:58 2015 -0700

    Remove double queries in l3 DB get methods

    Two frequently called functions were querying the routerport table
    and the corresponding ports just to get the port ID. Then they were
    calling get_ports again with those port IDs, resulting in two queries
    to the port table when there should have only been one.

    This eliminates the second call to get_ports since all of the necessary
    data hase been retrieved from the port table.

    Change-Id: I806e9c380b7de048fe084b2baf4b6f92ab0edf6b
    Partial-Bug: #1445412

Reviewed: https://review.openstack.org/174806
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=10b17a884452736a6b214bcb7705b955192a1748
Submitter: Jenkins
Branch: master

commit 10b17a884452736a6b214bcb7705b955192a1748
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 03:36:50 2015 -0700

    Set loading strategy to joined for Routerport/Port

    The RouterPort model has a relationship to the ports model which
    is frequently relied on to get the port IDs of interfaces attached
    to a router. However, this defaults to the loading strategy to
    'select', which meant a new query was being emitted for every
    interface to the ports table just to get the ID.

    This patch adjusts the relationship to be 'joined' by default so
    one query will fetch the related ports.

    Another option would have been not to use the port object at all since
    the ID is all that the callers were usually interested in. However,
    they would end up using the ID to do a port lookup, which is being
    optimized away in another patch anyway so the full port object from
    the relationship will end up getting used.

    Change-Id: Id1ae35f845f7367d5f1f065c6fa637da7b980a2b
    Partial-Bug: #1445412

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/175588

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/175589

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/175590

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/175591

Reviewed: https://review.openstack.org/174821
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6c6d3c9cca17a788a31526bb652dcdfc7bb54326
Submitter: Jenkins
Branch: master

commit 6c6d3c9cca17a788a31526bb652dcdfc7bb54326
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:18:56 2015 -0700

    Strip unnecessary overrides in extraroute_db mixin

    The extra route DB mixin seemed to be overriding the
    get_router and get_routers method for no reason. They
    both just called the super version of themselves with
    the same arguments.

    This patch just pulls those functions out. Found in
    tracebacks while working on a related bug.

    Change-Id: Ifd1a0676073e91104db3a13df6fe1eb2189f20f5
    Related-bug: #1445412

Reviewed: https://review.openstack.org/174812
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=649599457e29b58ad0aec9ace990e0a2b59b05d0
Submitter: Jenkins
Branch: master

commit 649599457e29b58ad0aec9ace990e0a2b59b05d0
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 03:53:45 2015 -0700

    Defer creation of router JSON in get_routers RPC

    The get_routers method in the l3 RPC code has a log.debug
    statement that formats all of the router data as indented
    JSON. This method can be expensive if there are hundreds
    of routers being synced and it happens even if debugging
    is disabled since the function call result is the parameter
    to the debug statement.

    This patch adds and leverages a small helper class that takes a
    callable and its args and defers calling it until the __str__ method
    is called on it when it's actually trying to be rendered to a string.

    Change-Id: I2bfceb286ce30f2a3595381b62bdc6dd71ed8483
    Partial-Bug: #1445412

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/176450

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/176451

Reviewed: https://review.openstack.org/174843
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9701bd479529ccc243e48fdb944c284d2921c376
Submitter: Jenkins
Branch: master

commit 9701bd479529ccc243e48fdb944c284d2921c376
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:46:11 2015 -0700

    L3 DB: Defer port DB subnet lookups

    _populate_subnets_for_ports was being called multiple
    times for different interface types during the get_routers
    process.

    This patch eliminates those extra queries by deferring the
    subnet information population until after all of the interfaces
    have been looked up. Includes a function rename as well to
    indicate that a function is only used internally.

    Change-Id: Ib46f685d72eb61ecbaa2869e28fb173cd6d49552
    Partial-bug: #1445412

Reviewed: https://review.openstack.org/174844
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4be5c2f6dc635e1139e268b078ba3c28c3bcefb6
Submitter: Jenkins
Branch: master

commit 4be5c2f6dc635e1139e268b078ba3c28c3bcefb6
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:51:26 2015 -0700

    Replace unnecessary call to get_sync_routers

    Replaces a call to get_sync_routers with a single-column
    router DB query in a method that doesn't use any of the
    gateway information get_sync_routers spends extra
    time populating.

    Change-Id: I35eae975209316aad6b2c97c909dce385729864d
    Partial-Bug: #1445412

Reviewed: https://review.openstack.org/174845
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=682c0fdcc2faad07e82968a7a7739f2dedd7173f
Submitter: Jenkins
Branch: master

commit 682c0fdcc2faad07e82968a7a7739f2dedd7173f
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:54:41 2015 -0700

    Eliminate extra queries used to retrieve gw_ports

    The _get_sync_routers method was calling get_routers and
    then getting the gateway ports from the db in a separate
    get_ports call. This extra call is unnecessary since is
    already an SQL relationship directly between the router
    and it's gw_port.

    This patch eliminates all of the additional gw_port retrieval
    logic by replacing the get_routers call with a _get_collection
    call to make use of the gw_port object already present on
    each router object.

    Change-Id: I478bfef8b0273b343aa72bcd6787a486eba4f006
    Partial-Bug: #1445412

Reviewed: https://review.openstack.org/174818
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a80924dc3e648984873833399350ba4817f1eaa9
Submitter: Jenkins
Branch: master

commit a80924dc3e648984873833399350ba4817f1eaa9
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:09:48 2015 -0700

    Don't update port with host id of None

    In the L3 RPC code if the host for a port is not
    present, it ends up calling update_port with the
    host_id set to None. This does not update the host
    id at all because it's treated as an unset attribute
    which leads to the same thing happening on the next
    iteration. These pointless update calls are expensive
    because they involve a semaphore and calls to mechanism
    drivers.

    This patch adjusts the logic to only send a port
    update if it actually has a host to ensure is on
    the port.

    Change-Id: Ic55496dd2ba3abcef0a2de9fc8699c391b79fa51
    Partial-Bug: #1445412

Reviewed: https://review.openstack.org/175588
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1a46332a4661841d219888a4bec952d6f9859331
Submitter: Jenkins
Branch: stable/kilo

commit 1a46332a4661841d219888a4bec952d6f9859331
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 03:36:50 2015 -0700

    Set loading strategy to joined for Routerport/Port

    The RouterPort model has a relationship to the ports model which
    is frequently relied on to get the port IDs of interfaces attached
    to a router. However, this defaults to the loading strategy to
    'select', which meant a new query was being emitted for every
    interface to the ports table just to get the ID.

    This patch adjusts the relationship to be 'joined' by default so
    one query will fetch the related ports.

    Another option would have been not to use the port object at all since
    the ID is all that the callers were usually interested in. However,
    they would end up using the ID to do a port lookup, which is being
    optimized away in another patch anyway so the full port object from
    the relationship will end up getting used.

    Change-Id: Id1ae35f845f7367d5f1f065c6fa637da7b980a2b
    Partial-Bug: #1445412
    (cherry picked from commit 10b17a884452736a6b214bcb7705b955192a1748)

Reviewed: https://review.openstack.org/175589
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3fbb39c3fdb6de342a58b9a8258390a1a522170a
Submitter: Jenkins
Branch: stable/juno

commit 3fbb39c3fdb6de342a58b9a8258390a1a522170a
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 03:36:50 2015 -0700

    Set loading strategy to joined for Routerport/Port

    The RouterPort model has a relationship to the ports model which
    is frequently relied on to get the port IDs of interfaces attached
    to a router. However, this defaults to the loading strategy to
    'select', which meant a new query was being emitted for every
    interface to the ports table just to get the ID.

    This patch adjusts the relationship to be 'joined' by default so
    one query will fetch the related ports.

    Another option would have been not to use the port object at all since
    the ID is all that the callers were usually interested in. However,
    they would end up using the ID to do a port lookup, which is being
    optimized away in another patch anyway so the full port object from
    the relationship will end up getting used.

    Change-Id: Id1ae35f845f7367d5f1f065c6fa637da7b980a2b
    Partial-Bug: #1445412
    (cherry picked from commit 10b17a884452736a6b214bcb7705b955192a1748)

Reviewed: https://review.openstack.org/175590
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c4d2b8ca6cf72d93704f62c0ddf649a42bc0863c
Submitter: Jenkins
Branch: stable/kilo

commit c4d2b8ca6cf72d93704f62c0ddf649a42bc0863c
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:28:58 2015 -0700

    Remove double queries in l3 DB get methods

    Two frequently called functions were querying the routerport table
    and the corresponding ports just to get the port ID. Then they were
    calling get_ports again with those port IDs, resulting in two queries
    to the port table when there should have only been one.

    This eliminates the second call to get_ports since all of the necessary
    data hase been retrieved from the port table.

    Change-Id: I806e9c380b7de048fe084b2baf4b6f92ab0edf6b
    Partial-Bug: #1445412
    (cherry picked from commit 3310c3c3d4c05c0d13f32f08f978ba4813e2a39a)

Reviewed: https://review.openstack.org/175591
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4a69a64c8a5c89970c5216c060ad3e6c79e3cdcc
Submitter: Jenkins
Branch: stable/juno

commit 4a69a64c8a5c89970c5216c060ad3e6c79e3cdcc
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:28:58 2015 -0700

    Remove double queries in l3 DB get methods

    Two frequently called functions were querying the routerport table
    and the corresponding ports just to get the port ID. Then they were
    calling get_ports again with those port IDs, resulting in two queries
    to the port table when there should have only been one.

    This eliminates the second call to get_ports since all of the necessary
    data hase been retrieved from the port table.

    Change-Id: I806e9c380b7de048fe084b2baf4b6f92ab0edf6b
    Partial-Bug: #1445412
    (cherry picked from commit 3310c3c3d4c05c0d13f32f08f978ba4813e2a39a)

Reviewed: https://review.openstack.org/174846
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=26284228dfc3c5f121f869dd6b2d2a492afaf659
Submitter: Jenkins
Branch: master

commit 26284228dfc3c5f121f869dd6b2d2a492afaf659
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 05:10:26 2015 -0700

    Get all interfaces for get_snat_sync_interfaces

    The get_snat_sync_interfaces method was being called for
    each router individually during a sync, which resulted
    in a new query to the database.

    This patch eliminates that waste by querying for the snat
    interfaces for all of the routers in the list at once.

    Change-Id: I1e44a0cf15a70632e8b62ac89ce807a7a457747d
    Partial-Bug: #1445412

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072

Reviewed: https://review.openstack.org/176451
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=aafe03f30b75c4f4e255f295c352e9eb2f2e1110
Submitter: Jenkins
Branch: stable/juno

commit aafe03f30b75c4f4e255f295c352e9eb2f2e1110
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 03:53:45 2015 -0700

    Defer creation of router JSON in get_routers RPC

    The get_routers method in the l3 RPC code has a log.debug
    statement that formats all of the router data as indented
    JSON. This method can be expensive if there are hundreds
    of routers being synced and it happens even if debugging
    is disabled since the function call result is the parameter
    to the debug statement.

    This patch adds and leverages a small helper class that takes a
    callable and its args and defers calling it until the __str__ method
    is called on it when it's actually trying to be rendered to a string.

    Change-Id: I2bfceb286ce30f2a3595381b62bdc6dd71ed8483
    Partial-Bug: #1445412
    (cherry picked from commit 649599457e29b58ad0aec9ace990e0a2b59b05d0)

Reviewed: https://review.openstack.org/176450
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=58af5e101d30027549bda9308536e9f45ed0bfef
Submitter: Jenkins
Branch: stable/kilo

commit 58af5e101d30027549bda9308536e9f45ed0bfef
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 03:53:45 2015 -0700

    Defer creation of router JSON in get_routers RPC

    The get_routers method in the l3 RPC code has a log.debug
    statement that formats all of the router data as indented
    JSON. This method can be expensive if there are hundreds
    of routers being synced and it happens even if debugging
    is disabled since the function call result is the parameter
    to the debug statement.

    This patch adds and leverages a small helper class that takes a
    callable and its args and defers calling it until the __str__ method
    is called on it when it's actually trying to be rendered to a string.

    Change-Id: I2bfceb286ce30f2a3595381b62bdc6dd71ed8483
    Partial-Bug: #1445412
    (cherry picked from commit 649599457e29b58ad0aec9ace990e0a2b59b05d0)

Reviewed: https://review.openstack.org/174815
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c44506bfd60b2dd6036e113464f1ea682cfaeb6c
Submitter: Jenkins
Branch: master

commit c44506bfd60b2dd6036e113464f1ea682cfaeb6c
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:03:38 2015 -0700

    Don't update floating IP status if no change

    The floating IP status was going through all of the
    status update code every time the L3 agent sent in
    an update, even if the status didn't change.

    This patch skips sending updates to the server if the
    agent doesn't change the status.

    Change-Id: Ic3736bed3dc3e4ccb91f4acfabbf033949e09ce0
    Partial-Bug: #1445412

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/187796

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/187802

Reviewed: https://review.openstack.org/187796
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cf7647502093aa8a1a04462ef52261e1f2c5f465
Submitter: Jenkins
Branch: stable/kilo

commit cf7647502093aa8a1a04462ef52261e1f2c5f465
Author: Kevin Benton <email address hidden>
Date: Fri Apr 17 04:03:38 2015 -0700

    Don't update floating IP status if no change

    The floating IP status was going through all of the
    status update code every time the L3 agent sent in
    an update, even if the status didn't change.

    This patch skips sending updates to the server if the
    agent doesn't change the status.

    Change-Id: Ic3736bed3dc3e4ccb91f4acfabbf033949e09ce0
    Partial-Bug: #1445412
    (cherry picked from commit c44506bfd60b2dd6036e113464f1ea682cfaeb6c)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/196701

Change abandoned by Kyle Mestery (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/196701
Reason: This is lacking the functional fix [1], so I'll propose a new merge commit which includes that one.

[1] https://review.openstack.org/#/c/196711/

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/196920

Reviewed: https://review.openstack.org/196920
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7f759c077f8f860c13db92d2ea6b353ef6b70900
Submitter: Jenkins
Branch: feature/pecan

commit 8123144fadd7c5d5e6e56a76ea860512619a2cf6
Author: Moshe Levi <email address hidden>
Date: Sun Jun 28 14:37:14 2015 +0300

    Fix Consolidate sriov agent and driver code

    This patch add mising __init to mech_sriov/mech_driver/
    and update the setup.cfg to the new agent entrypoint

    Trivial Fix

    Change-Id: I53a527081feb78472f496675bbb3c5121d38a14a

commit 8942fccf02e6e179d47582fdb2792a1ca972da21
Author: Assaf Muller <email address hidden>
Date: Mon Jun 29 11:38:51 2015 -0400

    Remove failing SafeFixture tests

    The fixtures 1.3 release attempted to fix the fixtures resource
    leak issue, but failed to do so completely. Our own SafeFixture
    is still needed: The 1.3 release broke our SafeFixture tests,
    but not the usage of SafeFixture itself. This patch removes
    those failing tests for now to unbreak the gate. Jakub reported
    a bug on fixtures 1.3:
    https://bugs.launchpad.net/python-fixtures/+bug/1469759

    We will continue to use SafeFixture until that bug is fixed
    in fixtures, at which point we will be able to require
    fixtures > 1.3.

    Change-Id: I59457c3bb198ff86d5ad55a1e623d008f0034b8f
    Closes-Bug: #1469734

commit 71dffb0a2c1720cd8233a329d32958a0160dd6f5
Author: Kevin Benton <email address hidden>
Date: Mon Jun 29 08:27:41 2015 +0000

    Revert "Removed test_lib module"

    This reverts commit 9a6536de6e1a7fe9b2552adc142e254426b82b6f.

    We pulled all of the plugins out of the tree, many of which still inherit
    from neutron test classes. This change then stated that we no longer
    support testing other plugins. I think this is a bit premature and should
    have been discussed under the subject
    "Neutron plugins can't use neutron plugin unit tests" or something
    similar.

    Change-Id: I68318589f010b731574ea3bfa8df98492bab31fc

commit b20fd81dbd497e058384a0af065dd0f1fdc4c728
Author: Jakub Libosvar <email address hidden>
Date: Fri Jun 5 14:32:51 2015 +0000

    Refactor NetcatTester class

    Following capabilities were added:
       - used transport protocol is passed as a constant instead of bool
       - src port for testing was added
       - connection can be established explicitly
       - change constructor parameters of NetcatTester

    As a part of removing bool for protocol definition
    get_free_namespace_port() was also modified to match the behavior.

    Change-Id: Id2ec322e7f731c05a3754a65411c9a5d8b258126

commit 83e37980dcd0b2bad6d64dd2cb23bcd2891cafca
Author: jingliuqing <email address hidden>
Date: Sat Jun 27 13:41:54 2015 +0800

    Use REST rather than ReST

    Change-Id: I06c9deaab58c5ec13bfeec39fb8fd4b1fe21f42d

commit 1b60df85ba3ad442c2e4e7e52538e1b9a1bf9378
Author: Kevin Benton <email address hidden>
Date: Thu Jun 25 18:34:38 2015 -0700

    Add a double-mock guard to the base test case

    Use mock to patch mock with a check to prevent multiple active
    patches to the...

Change abandoned by Kevin Benton (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/187802