contrail-schema failing to start when policy count threshold crossed
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R2.21.x |
Fix Committed
|
High
|
Sahil Sabharwal | |||
R3.0 |
Fix Committed
|
High
|
Sahil Sabharwal | |||
R3.1 |
Fix Committed
|
High
|
Sahil Sabharwal | |||
R3.2 |
Fix Committed
|
High
|
Sahil Sabharwal | |||
Trunk |
Fix Committed
|
High
|
Sahil Sabharwal |
Bug Description
On November 9th contrail-schema failed to start in one of our clusters (contrail 2.21.3-47.el6). The problem is due to the number of network policies in the cluster.
To work around the issue we applied the following patch to /usr/lib/
def _zk_listener(self, state):
if state == KazooState.
if self._election:
# Update connection info
elif state == KazooState.LOST:
# Lost the session with ZooKeeper Server
# Best of option we have is to exit the process and restart all
# over again
if self._lost_cb:
else:
- os._exit(2)
+ pass
elif state == KazooState.
# Update connection info
This allows contrail-schema to start but needless to say it's a very short term fix.
The problem can be recreated as follows,
1. Create 2 networks.
2. Create 2 policies with 100 rules each and attach to the networks created in step 1. (see note 1 below)
3. Stop and start the contrail-schema service. I started it interactively rather than using the service as it gave me a little more control.
/usr/bin/python /usr/bin/
4. Note the number of rules returned from /access-
5. Repeat these steps until contrail-schema fails to start
In my testing contrail-schema failed to start when these numbers were reached,
- 263 virtual networks
- 717 network policies
- 98412 network policy rules as returned from /access-
- 2:40 time it takes contrail-schema to fail
*Note 1: Policy Details*
Each policy has 100 rules. Each rule looked like this,
source: local
source port: any
dest: 10.0.0.$i/32
dest_port: any
protocol: tcp
direction <>
action pass
$i runs from 1 to 100.
*Note 2: Counting rules*
curl -s -o acls.json -H "X-Auth-Token: $TOKEN" "http://[CONTRAIL CONTROLLER IP]:9100/
grep -o rule_uuid acls.json | wc -l
tags: | added: config |
tags: | added: wpc |
information type: | Proprietary → Public |
tags: | added: att-aic-contrail |
Review in progress for https:/ /review. opencontrail. org/26678
Submitter: <email address hidden> (<email address hidden>)