contrail-collector going berserk over "nw_ace_uuid: () INVALID" messages

This error message is logged by the contrail-collector when the vrouter-agent sends invalid uuid string. In this specific case, vrouter-agent seem to have sent empty string for nw_ace_uuid in the flow log. Assigning the bug to Ashok to check if the vrouter-agent sends "" nw_ace_uuid in any case.

As far as the high cpu utilization is concerned, there is not enough information to analyze further. Perhaps, it could be due to high rate of messages sent by the agent/other generators. Any information regarding the number of generators connected to the contrail-collector and the rate at which the contrail-collector received messages during period of high cpu utilization would be helpful.

Similar issue was reported sometime back.
https://bugs.launchpad.net/juniperopenstack/+bug/1344716
As part of the fix, the boost::uuid::sting_generator (throws exception when invalid uuid string is received) was replaced with stream operator.

We still see this and contrail-collector still eats massive amounts of CPU and writes this stuff into its logfile all the time. Now it's happening with Juno and Contrail 2.0. Any help will be appreciated.

The "" uuid is being sent from agent because "" uuid is being received by agent as part of acl rule configuration. On checking configuration in API server it was found that null UUIDs were present for some ACL rules. I have attached a tar file which has ACL entries and rules for one of the ACL (default-domain:admin:float24:float24) which had null UUIDs

Martin, could you please tell us what version of contrail you are running? And if you had upgraded from an earlier version, please let us know that too.

This is Contrail R2.0 with a few patches applied on top of it (mostly taken from your git repos). We updated this installation from 1.20 to 2.0 a week ago, but as we have pointed out earlier, we saw the same messages from contrail-collector with 1.20 as well.

Review in progress for https://review.opencontrail.org/17051
Submitter: Raj Reddy (<email address hidden>)

Reviewed: https://review.opencontrail.org/17051
Committed: http://github.org/Juniper/contrail-controller/commit/f32f0c9921f1d3809181e5bc092be8fbaa05ac80
Submitter: Zuul
Branch: master

commit f32f0c9921f1d3809181e5bc092be8fbaa05ac80
Author: Raj Reddy <email address hidden>
Date: Tue Feb 9 13:39:05 2016 -0800

There are some corner cases where the ace uuid comes as null, we
can just ignore in the collector than printing and error message
as that causes log file to be overwhelmed.
Closes-Bug: #1400617

Change-Id: I4943f0d0621330ccb105182d9e6f8f29c177561a

Review in progress for https://review.opencontrail.org/19216
Submitter: Jean-Philippe Braun (<email address hidden>)

Review in progress for https://review.opencontrail.org/19744
Submitter: Raj Reddy (<email address hidden>)

Review in progress for https://review.opencontrail.org/19745
Submitter: Raj Reddy (<email address hidden>)

Reviewed: https://review.opencontrail.org/19216
Committed: http://github.org/Juniper/contrail-controller/commit/a1a9ab44a4814b6f4ce7a9bf622d2459ca4fda28
Submitter: Zuul
Branch: R2.21.x

commit a1a9ab44a4814b6f4ce7a9bf622d2459ca4fda28
Author: Raj Reddy <email address hidden>
Date: Tue Feb 9 13:39:05 2016 -0800

There are some corner cases where the ace uuid comes as null, we
can just ignore in the collector than printing and error message
as that causes log file to be overwhelmed.
Closes-Bug: #1400617

Change-Id: I4943f0d0621330ccb105182d9e6f8f29c177561a
(cherry picked from commit f32f0c9921f1d3809181e5bc092be8fbaa05ac80)

Reviewed: https://review.opencontrail.org/19745
Committed: http://github.org/Juniper/contrail-controller/commit/9bc4ff65e9bcc3d906256c7ac7b3b0e46275d502
Submitter: Zuul
Branch: R2.22.x

commit 9bc4ff65e9bcc3d906256c7ac7b3b0e46275d502
Author: Raj Reddy <email address hidden>
Date: Tue Feb 9 13:39:05 2016 -0800

There are some corner cases where the ace uuid comes as null, we
can just ignore in the collector than printing and error message
as that causes log file to be overwhelmed.
Closes-Bug: #1400617

Change-Id: I4943f0d0621330ccb105182d9e6f8f29c177561a
(cherry picked from commit f32f0c9921f1d3809181e5bc092be8fbaa05ac80)

Reviewed: https://review.opencontrail.org/19744
Committed: http://github.org/Juniper/contrail-controller/commit/31b42b19ffae918d0b633aff5d44e5a9f078a560
Submitter: Zuul
Branch: R2.20

commit 31b42b19ffae918d0b633aff5d44e5a9f078a560
Author: Raj Reddy <email address hidden>
Date: Tue Feb 9 13:39:05 2016 -0800

There are some corner cases where the ace uuid comes as null, we
can just ignore in the collector than printing and error message
as that causes log file to be overwhelmed.
Closes-Bug: #1400617

Change-Id: I4943f0d0621330ccb105182d9e6f8f29c177561a
(cherry picked from commit f32f0c9921f1d3809181e5bc092be8fbaa05ac80)