High CPU load on compute node caused by contrail-vrouter-agent process

Bug #1488780 reported by Xiang Hui
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
New
Undecided
Unassigned

Bug Description

[Description]
CPU load spikes occurred in three episodes on a single day, but have not reoccurred since.

We received a notification from out monitoring system of high CPU load on one of our compute nodes, compute02. On investigation we found that 1 particular process was causing this, namely contrail-vrouter-agent. While investigating the problem, the CPU utilization suddenly recovered, only to spike up again at around 14:10 local time, but has since dropped down again. This second episode was also due to contrail-vrouter-agent.

Attached is a screen shot showing the load, the output from top during one of these events, the output from df -h, and the output from a ps ax|grep to check the process causing the problem.

We haven't been able to figure out why this is happening and the only error in the contrail-vrouter-agent log file is about a device running out of space, yet none of the storage devices on the compute node is even close to running out of space.

Could anyone please assist in figuring out what's going on here?

Tags: vrouter
Revision history for this message
Xiang Hui (xianghui) wrote :
Xiang Hui (xianghui)
information type: Proprietary → Public
Revision history for this message
Dominique Poulain (dominique-poulain) wrote :

Seems related to <https://bugs.launchpad.net/juniperopenstack/+bug/1400633> (contrail-vrouter-agent reporting "No space left on device").

While in the present case the high-load spikes went away, the concern is that a similar occurrence affecting all compute instances simultaneously would bring the environment to its knees, hence a keen desire to get to the bottom of this :-)

tags: added: vrouter
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.