[R 2.20] SNAT: Instance VN not created sometimes leading to SNAT failures

32 private VNs are connected to a public GW via a LR. Setting up/tearing down will sometimes cause the instance VN to not be created. Details below:

----

I see that the instance_vn is not created for some reason. The LR has been created and a private VN and GW have been attached to it. The service instance has been created with right interface in the GW VN, but because the instance VN isn't there, SNAT doesn't work.

Also seems like the message queue has pending messages on it that haven't been dequeued by svc-mon.

svc_mon.csol2-node5 2185
svc_mon.csol2-node8 4410

root@csol2-node5:/var/log/contrail# neutron net-list
+--------------------------------------+-------------------------+-------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-------------------------+-------------------------------------------------+
| 4c4dace9-9584-4dd8-8200-2f90cdc42740 | ip-fabric | |
| 1ca51aae-7587-4ee0-ab68-dd2539dce72b | C_VN | a8ab1252-1f7a-46a2-a35a-969320f8580e 5.1.1.0/24 |
| d8d62093-1ca3-4e5e-a511-f8868932f9bf | default-virtual-network | |
| c58c620a-3d26-4cdb-bfee-a38997faeede | __link_local__ | |
| 867f4a38-d074-43d9-b8fb-514b4e8eeb4c | P_VN1 | d65badcb-0aae-4e85-8282-8528f4308524 6.1.1.0/24 |
| 2c3a5e5f-4f7d-4d35-bc0e-ad76093225b4 | GW1 | f2af2852-7681-4474-accf-645fc191e9a8 7.1.1.0/24 |
+--------------------------------------+-------------------------+-------------------------------------------------+
root@csol2-node5:/var/log/contrail# neutron router-list
+--------------------------------------+------+-----------------------------------------------------------------------------+
| id | name | external_gateway_info |
+--------------------------------------+------+-----------------------------------------------------------------------------+
| d98369dd-efae-46d0-9a79-a21d10ec5399 | R1 | {"network_id": "2c3a5e5f-4f7d-4d35-bc0e-ad76093225b4", "enable_snat": true} |
+--------------------------------------+------+-----------------------------------------------------------------------------+

Can you please take a look at the system and help debug this? The controller node is at 10.87.26.145 (.147 and .148 are the other 2 nodes of the cluster).

The service-monitor log shows this error:

127.0.0.1 - - [02/Jun/2015 07:16:12] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 3630
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 110, in _connection_watch
    self._conn.drain_events()
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 279, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 90, in drain_events
    return connection.drain_events(**kwargs)
  File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 322, in drain_events
    return amqp_method(channel, args, content)
  File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1908, in _basic_deliver
    fun(msg)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 589, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 558, in receive
    [callback(body, message) for callback in callbacks]
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 137, in _subscribe
    self._subscribe_cb(body)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 285, in _vnc_subscribe_callback
    self._delete_interface_route_table(irt_id)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 735, in _delete_interface_route_table
    self._vnc_lib.interface_route_table_delete(id=irt_uuid)
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 3652, in interface_route_table_delete
    content = self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 368, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 429, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error
<Greenlet at 0x7fab683914b0: <bound method VncKombuClientV2._connection_watch of <cfgm_common.vnc_kombu.VncKombuClientV2 object at 0x7fab684ccfd0>>> failed with HttpError