Juniper Openstack

schema transformer could crash during port update if port disappears in the middle of the update

  1. Series trunk
  2. Bug #1691590
Bug #1691590 reported by Sachin Bansal
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.21.x
Fix Committed
Undecided
Sahil Sabharwal
Juniper Openstack r2.21.3
R4.0
Fix Committed
Undecided
Sahil Sabharwal
Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk
Fix Committed
Undecided
Sahil Sabharwal
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

Observed on a customer setup that schema transformer could crash with the following traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/schema_transformer/to_bgp.py", line 3870, in launch_arc
    transformer.process_poll_result(result)
  File "/usr/lib/python2.6/site-packages/schema_transformer/to_bgp.py", line 3445, in process_poll_result
    func(idents, meta)
  File "/usr/lib/python2.6/site-packages/schema_transformer/to_bgp.py", line 3113, in add_virtual_machine_interface_virtual_network
    vmi.set_virtual_network(vn_name)
  File "/usr/lib/python2.6/site-packages/schema_transformer/to_bgp.py", line 2323, in set_virtual_network
    _vnc_lib.virtual_machine_interface_update(self.obj)
  File "/usr/lib/python2.6/site-packages/vnc_api/gen/vnc_api_client_gen.py", line 4340, in virtual_machine_interface_update
    self.ref_update('virtual-machine-interface', obj.uuid, ref_name, ref[0], list(ref[1]), 'ADD', ref[2])
  File "/usr/lib/python2.6/site-packages/vnc_api/vnc_api.py", line 577, in ref_update
    raise he
HttpError: HTTP Status: 500 Content: Internal Server Error

This happened because api server didn't handle the disappearance of the port while it tried to update last modified timestamp with the traceback below:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bottle.py", line 763, in _handle
    return route.call(**args)
  File "/usr/lib/python2.6/site-packages/bottle.py", line 1624, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.6/site-packages/bottle.py", line 1574, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 496, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 659, in ref_update_http_post
    id = self._db_conn.ref_update(obj_type, obj_uuid, ref_type, ref_uuid, {'attr': attr}, operation)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1540, in ref_update
    ref_data, operation)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 550, in ref_update
    self.update_last_modified(bch, obj_uuid)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 675, in update_last_modified
    ['prop:id_perms'])['prop:id_perms'])
  File "/usr/lib/python2.6/site-packages/cfgm_common/vnc_cassandra.py", line 108, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pycassa/columnfamily.py", line 652, in get
    read_consistency_level or self.read_consistency_level)
  File "/usr/lib/python2.6/site-packages/pycassa/pool.py", line 577, in execute
    return getattr(conn, f)(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pycassa/pool.py", line 127, in new_f
    result = f(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pycassa/cassandra/Cassandra.py", line 534, in get
    return self.recv_get()
  File "/usr/lib/python2.6/site-packages/pycassa/cassandra/Cassandra.py", line 561, in recv_get
    raise result.nfe
NotFoundException: NotFoundException(_message=None)
10.96.0.155 - - [2017-05-03 15:16:47] "POST /ref-update HTTP/1.1" 500 156 0.019676

Tags: config wpc
Revision history for this message
Sachin Bansal (sbansal) wrote :

On releases older than 3.0, such a crash could cause schema transformer to lose some 'in-flight' messages from ifmap server. As a result, those messages won't be processed resulting in lost functionality until schema transformer is restarted.

This problem is not there in 3.0 or later because schema transformer does not drop any in-flight messages. However, the traceback still needs to be handled in api server.

information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/32643
Submitter: <email address hidden> (<email address hidden>)

Vineet Gupta (vineetrf)
tags: added: wpc
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/32643
Committed: http://github.com/Juniper/contrail-controller/commit/42eb794b6ce51b6d9034e9214b7f097451203709
Submitter: Zuul (<email address hidden>)
Branch: master

commit 42eb794b6ce51b6d9034e9214b7f097451203709
Author: Sahil <email address hidden>
Date: Wed Jun 7 15:47:53 2017 -0700

[api_server]: Fix to call ref_update function.

1. Added a call to ref_update function in _put_common().
We were only doing dbe_update at the time of ref_update_http_post.

2. Modified update_last_modified function for cassandra so that it
doesn't throw a traceback when obj. gets deleted at the time
of obj. update

Change-Id: I36064a8b07503732ed76a4ffaddef9ef02a3c054
Closes-Bug: 1691590

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/32719
Submitter: <email address hidden> (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/32726
Submitter: <email address hidden> (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/32719
Committed: http://github.com/Juniper/contrail-controller/commit/38e3d3e30925ca465dffbdef1bd4b5f9d8f83bb5
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 38e3d3e30925ca465dffbdef1bd4b5f9d8f83bb5
Author: Sahil <email address hidden>
Date: Wed Jun 7 15:47:53 2017 -0700

[api_server]: Fix to call ref_update function.

1. Added a call to ref_update function in _put_common().
We were only doing dbe_update at the time of ref_update_http_post.

2. Modified update_last_modified function for cassandra so that it
doesn't throw a traceback when obj. gets deleted at the time
of obj. update

Change-Id: I36064a8b07503732ed76a4ffaddef9ef02a3c054
Closes-Bug: 1691590
(cherry picked from commit 42eb794b6ce51b6d9034e9214b7f097451203709)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/32726
Committed: http://github.com/Juniper/contrail-controller/commit/2a8ebf97e7f606563fb866b5cc777d50f4d83cb2
Submitter: Zuul (<email address hidden>)
Branch: R2.21.x

commit 2a8ebf97e7f606563fb866b5cc777d50f4d83cb2
Author: Sahil <email address hidden>
Date: Fri Jun 9 14:44:33 2017 -0700

[api_server]: Fix to call ref_update function.

1. Added a call to ref_update function in _put_common().
We were only doing dbe_update at the time of ref_update_http_post.

2. Modified update_last_modified function for cassandra so that it
doesn't throw a traceback when obj. gets deleted at the time
of obj. update

Change-Id: I2b58b4a4974891a5b584ad7848f8747fe5a8f8d8
Closes-Bug: 1691590

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.