DBDuplicateError found sometimes when router_interface_delete issued with DVR

Marking incomplete until further details are provided.

This issue is a side effect of the DB lockwait.
We use two scripts, one to create and one for cleanup.

The issue is seen with the Cleanup script.

The Create script:
1. Creates three networks and subnets
2. Creates a router.
3. Creates a public network.
4. Add networks to the router.
5. Set gateway to the router.
6. Create VM on the network
7. Create a FloatingIP and attach a FIP to a VM.
9. Try to ping the VM.

The CleanUP Script:
1. Deletes the FloatingIP
2. Deletes the VM.
3. Removes the interfaces from the Router.
4. Gateway Clear
5. Delete Router
6. Delete Subnet
7. Delete Networks etc.,

When running the CleanUP script, for some reason the Gateway clear happens first before the interfaces are removed, and then when interfaces are removed, the _schedule_router is called.
In the _schedule_router, the check for the "external_gw_info" succeeds even though the Gateway clear happened earlier. ( This call probably did not remove the router completely from the DB).

So because of this, the schedule_snat_router is called again and then we see a DBDuplicate error.

2014-10-07 15:55:56.692 DEBUG neutron.context [req-0f3120c9-3e1b-49db-84cb-ac69d1fbf7df None None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} from (pid=51358) __init__ /opt/stack/neutron/neutron/context.py:83
2014-10-07 15:55:56.781 DEBUG neutron.api.rpc.agentnotifiers.l3_rpc_agent_api [req-96d3bbc8-da54-4aa4-aa59-a9e7fc878eff admin 5e0ec017b5ab4ee288b9c85b8a6e87f3] Notify agent at l3_agent.ubuntu the message routers_updated from (pid=51358) _agent_notification /opt/stack/neutron/neutron/api/rpc/agentnotifiers/l3_rpc_agent_api.py:66
2014-10-07 15:55:56.782 DEBUG neutron.common.rpc [req-96d3bbc8-da54-4aa4-aa59-a9e7fc878eff admin 5e0ec017b5ab4ee288b9c85b8a6e87f3] neutron.api.rpc.agentnotifiers.l3_rpc_agent_api.L3AgentNotifyAPI method cast called with arguments (<neutron.context.Context object at 0x7f4559930650>, {'args': {'routers': [u'7a0a1515-52dd-4a7b-9f4d-ab5b0b89cc8b']}, 'namespace': None, 'method': 'routers_updated'}) {'topic': u'l3_agent.ubuntu', 'version': '1.1'} from (pid=51358) wrapper /opt/stack/neutron/neutron/common/log.py:33
2014-10-07 15:55:56.804 INFO neutron.wsgi [req-96d3bbc8-da54-4aa4-aa59-a9e7fc878eff admin 5e0ec017b5ab4ee288b9c85b8a6e87f3] 192.168.15.144 - - [07/Oct/2014 15:55:56] "PUT /v2.0/routers/7a0a1515-52dd-4a7b-9f4d-ab5b0b89cc8b/remove_router_interface.json HTTP/1.1" 200 389 1.824291
2014-10-07 15:55:56.936 WARNING neutron.scheduler.l3_agent_scheduler [req-aae0cf4f-b6b4-4977-8146-97572eb1681f admin 5e0ec017b5ab4ee288b9c85b8a6e87f3] SWAMI: Router gateway exists: {'network_id': u'611fbc19-0e66-4bff-b324-aec6d6cc4c71', 'enable_snat': True, 'external_fixed_ips': [{'subnet_id': u'52f5a6f9-c6ee-4448-8376-5ce097f851b0', 'ip_address': u'153.1.1.100'}]}
2014-10-07 15:55:56.947 WARNING neutron.scheduler.l3_agent_scheduler [req-aae0cf4f-b6b4-4977-8146-97572eb1681f admin 5e0ec017b5ab4ee288b9c85b8a6e87f3] SWAMI: Binding: [<neutron.db.l3_dvrscheduler_db.CentralizedSnatL3AgentBinding[object at 7f4559b7bed0] {router_id=u'7a0a1515-52dd-4a7b-9f4d-ab5b0b89cc8b', l3_agent_id=u'aa2683b9-d7d8-4d21-9dc2-db47c6f43ffa', host_id=None, csnat_gw_port_id=None}>]
2014-10-07 15:55:57.001 DEBUG neutron.api.rpc.agentnotifiers.l3_rpc_agent_api [req-aae0cf4f-b6b4-4977-8146-97572eb1681f admin 5e0ec017b5ab4ee288b9c85b8a6e87f3] Notify agent at l3_agent.ubuntu the message routers_updated from (pid=51358) _agent_notification /opt/stack/neutron/neutron/api/rpc/agentnotifiers/l3_rpc_agent_api.py:66
2014-10-07 15:55:57.002 DEBUG neutron.common.rpc [req-aae0cf4f-b6b4-4977-8146-97572eb1681f admin 5e0ec017b5ab4ee288b9c85b8a6e87f3] neutron.api.rpc.agentnotifiers.l3_rpc_agent_api.L3AgentNotifyAPI method cast called with arguments (<neutron.context.Context object at 0x7f4559b01f50>, {'args': {'routers': [u'7a0a1515-52dd-4a7b-9f4d-ab5b0b89cc8b']}, 'namespace': None, 'method': 'routers_updated'}) {'topic': u'l3_agent.ubuntu', 'version': '1.1'} from (pid=51358) wrapper /opt/stack/neutron/neutron/common/log.py:33
2014-10-07 15:55:57.014 INFO neutron.wsgi [req-aae0cf4f-b6b4-4977-8146-97572eb1681f admin 5e0ec017b5ab4ee288b9c85b8a6e87f3] 192.168.15.144 - - [07/Oct/2014 15:55:57] "PUT /v2.0/routers/7a0a1515-52dd-4a7b-9f4d-ab5b0b89cc8b...

Trace captured in the file.

Fix proposed to branch: master
Review: https://review.openstack.org/126793

This bug was addressed by one of patches that Kevin Benton pushed in.
So this bug can be closed.

This is the patch ID that Kevin Benton pushed in.
https://review.openstack.org/#/c/128857/4

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/126793
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.