DB out of sync

Bug #1733545 reported by Ricardo Noriega
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
networking-l2gw
New
Undecided
Unassigned

Bug Description

I haven't found any deterministic way to reproduce the issue. Basically, executing commands that could modify the database (create, update, delete) could lead to out of syncs.

It seems the agent performs actions without taking into account the transactions to the database.

Revision history for this message
Ricardo Noriega (rnoriega-b) wrote :
Download full text (7.3 KiB)

Adding the following exception:

diff --git a/networking_l2gw/db/l2gateway/l2gateway_db.py b/networking_l2gw/db/l2gateway/l2gateway_db.py
index 806647b..227d5d5 100644
--- a/networking_l2gw/db/l2gateway/l2gateway_db.py
+++ b/networking_l2gw/db/l2gateway/l2gateway_db.py
@@ -277,6 +277,7 @@ class L2GatewayMixin(l2gateway.L2GatewayPluginBase,

     def create_l2_gateway_connection(self, context, l2_gateway_connection):
         """Create l2 gateway connection."""
+ raise Exception("GET OUT OF SYNC!")
         gw_connection = l2_gateway_connection[self.connection_resource]
         l2_gw_id = gw_connection.get('l2_gateway_id')
         network_id = gw_connection.get('network_id')

This will trigger the inconsistency between the DB and the OVSDB of the managed switch:

centos@devstack ~]$ ./l2gw_create.txt
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-79016e1e-9c9a-44f8-9186-ebc5ceda8039']
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-fef00891-1477-462a-be44-df068a5989ab']

[centos@devstack ~]$ cat l2gw_create.txt
 neutron l2-gateway-connection-create gw1 private --default-segmentation-id 200
 neutron l2-gateway-connection-create gw1 private2 --default-segmentation-id 300
[centos@devstack ~]$ neutron l2-gateway-connection-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.

[centos@devstack ~]$

However, the OVSDB has been modified adding new vlan bindings:

[centos@hwvtep ~]$ sudo ovsdb-client dump hardware_vtep [35/8267]
ACL table
_uuid acl_entries acl_fault_status acl_name
----- ----------- ---------------- --------

ACL_entry table
_uuid acle_fault_status action dest_ip dest_mac dest_mask dest_port_max dest_port_min direction ethertype icmp_code icmp_type protocol sequence source_ip source_mac source_mask source_port_max source_port_m
in tcp_flags tcp_flags_mask
----- ----------------- ------ ------- -------- --------- ------------- ------------- --------- --------- --------- --------- -------- -------- --------- ---------- ----------- --------------- -------------
-- --------- --------------

Arp_Sources_Local table
_uuid locator src_mac
----- ------- -------

Arp_Sources_Remote table
_uuid locator src_mac
----- ------- -------

Global table
_uuid managers switches
------------------------------------ -------------------------------------- --------------------------------------
0d30fbd3-e040-47c0-9841-c011baa554c3 [6f32515f-59bf-44ae-b12a-43255d2019b9] [1a412a01-943e-47fb-9779-d631d4df4e2a]

Logical_Binding_Stats table
_uuid bytes_from_local bytes_to_local packets_from_local packets_to_local
----- ---------------- -------------- ------------------ ------------...

Read more...

Revision history for this message
Ricardo Noriega (rnoriega-b) wrote :

A possible way to repair it would be to use an synchronizer using external uuids in the OVSDB. For example, if we use the "description" field in the Physical_port table to store Neutron L2Gateway Connection UUID:

[centos@hwvtep ~]$ sudo vtep-ctl list Physical_port
_uuid : d53ee688-e763-4165-acc4-134acf447f50
acl_bindings : {}
description : ""
name : "eth2"
port_fault_status : []
vlan_bindings : {200=f5760373-6242-4b9d-ab70-e09b3eb7ad92, 300=adc60ecd-3c46-477e-ad47-f6cedb382234}
vlan_stats : {}

_uuid : 77c6c08f-1520-4698-8808-abd4f7eeb61b
acl_bindings : {}
description : "9cd3ad7d-71aa-498e-8ce8-befac2ad95fc"
name : "eth1"
port_fault_status : []
vlan_bindings : {200=f5760373-6242-4b9d-ab70-e09b3eb7ad92, 300=adc60ecd-3c46-477e-ad47-f6cedb382234}
vlan_stats : {}

neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new l2_gateway_connection:
+-----------------+--------------------------------------+
| Field | Value |
+-----------------+--------------------------------------+
| id | 9cd3ad7d-71aa-498e-8ce8-befac2ad95fc |----------> Check in the table above
| l2_gateway_id | 8196df15-c9f0-4f3f-b4a4-2377f2505ebe |
| network_id | 0f8f9ccd-a85a-4d7e-b71e-1df65b90f18e |
| segmentation_id | 200 |
| tenant_id | 802531e233854bffbac67504eaed1cd3 |
+-----------------+--------------------------------------+

If there is an orphan object, we could apply changes to solve this inconsistency.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.