Contrail-Analytics-API returns some stale UVEs

Bug #1383896 reported by Anish Mehta
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Committed
High
Anish Mehta
R1.1
Fix Committed
High
Anish Mehta
R2.0
Fix Committed
Undecided
Anish Mehta
OpenContrail
Fix Committed
High
Anish Mehta

Bug Description

We saw an instance on a scale setup where a stale UVE was being shown.
It seemed that the generator was earlier connected to b5s33, and then moved to b5s32
But part of the UVE was still being reported out of b5s33, and that part was stale.

From: Anish Mehta <email address hidden>
Date: Monday, October 20, 2014 at 4:52 PM
To: Harshad Nakil <email address hidden>, Megh Bhatt <email address hidden>
Cc: Raj Reddy <email address hidden>, Sundaresan Rajangam <email address hidden>
Subject: Re: Mismatch in UVE data in introspect and analytics

Looks like a bug in the UVE aggregation logic.
This generator is connected to b5s32. The UVE value in redis in b5s32 is correct.
But it was previously connected to b5s33. The incorrect UVE is being reported out of there.

We should not be getting values for this UVE from b5s33. The generator is not on the NGENERATORS set on b5s33.
I’ll look into it some more.

From: Harshad Nakil <email address hidden>
Date: Monday, October 20, 2014 at 4:08 PM
To: Anish Mehta <email address hidden>, Megh Bhatt <email address hidden>
Cc: Raj Reddy <email address hidden>
Subject: Mismatch in UVE data in introspect and analytics

Down interface count is wrong

http://10.84.25.31:8081/analytics/uves/vrouter/b1s42v2?flat

http://10.84.3.95:8085/Snh_SandeshUVECacheReq?x=VrouterAgent

Regards
-Harshad

Tags: analytics
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/3968
Committed: http://github.org/Juniper/contrail-controller/commit/5331148bd3f6a8dea5faf433d721153d372c004f
Submitter: Zuul
Branch: master

commit 5331148bd3f6a8dea5faf433d721153d372c004f
Author: Anish Mehta <email address hidden>
Date: Tue Oct 21 14:00:17 2014 -0700

contrail-collector does UVE Updates based on messages from generators.
It also deletes UVE values from redis when generators disconnect.
We saw an instance where updates seems to happen just after a disconnect.
This causes contrail-analytics-api to return stale values for some UVE attributes.

Before the UVEUpdate script writes updates to redis, it should check that the generator
for which the update is being done is still connected (via the NGENERATORS set in redis)

Change-Id: If376bca64f5a8ba5be9eaa72a71d47a275b6eb20
Closes-Bug: #1383896

Changed in juniperopenstack:
importance: Undecided → High
assignee: nobody → Anish Mehta (amehta00)
tags: added: analytics
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/3969
Committed: http://github.org/Juniper/contrail-controller/commit/547383b267bc51b59079e2b6b0d32eadf891e373
Submitter: Zuul
Branch: R1.10

commit 547383b267bc51b59079e2b6b0d32eadf891e373
Author: Anish Mehta <email address hidden>
Date: Tue Oct 21 14:00:17 2014 -0700

contrail-collector does UVE Updates based on messages from generators.
It also deletes UVE values from redis when generators disconnect.
We saw an instance where updates seems to happen just after a disconnect.
This causes contrail-analytics-api to return stale values for some UVE attributes.

Before the UVEUpdate script writes updates to redis, it should check that the generator
for which the update is being done is still connected (via the NGENERATORS set in redis)

Change-Id: If376bca64f5a8ba5be9eaa72a71d47a275b6eb20
Closes-Bug: #1383896

Revision history for this message
Anish Mehta (amehta00) wrote :

was committed to master

Changed in juniperopenstack:
status: New → Fix Committed
Changed in opencontrail:
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.