neutron-sriov-agent consumes much resources by redundand calls to ip link show

Bug #1653131 reported by Mikhail Chernik
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Fix Committed
Medium
MOS Neutron
9.x
Fix Released
Medium
Kevin Benton

Bug Description

On a node with SR-IOV interfaces neutron-sriov-agent seems to call 'ip link show' every 2 second for each interface. This leads to CPU utilization up to 60+% (I've seen 32% for neutron-rootwrap and 32% for neutron-sriov-nic-agent)

Since output of `ip link show' is the same for all interfaces, it is enough to call it only once per 2 sec interval

root@node-7:~# grep '15:10:' /var/log/neutron/neutron-sriov-agent.log | grep "'ip', 'link', 'show'" | wc -l
5783

# all interfaces
root@node-7:~# ip link show | grep '^[0-9].*' | wc -l
274

# SR-IOV VF interfaces
root@node-7:~# ip link show | grep '^[0-9].*' | grep eth | wc -l
252

 9224 root 10 -10 5148300 144148 9940 S 299.9 0.1 8871:03 ovs-vswitchd
36425 root 20 0 199672 16764 7156 S 27.2 0.0 682:36.73 neutron-rootwra
36398 neutron 20 0 196936 60620 10416 S 20.3 0.0 522:57.02 neutron-sriov-n

Revision history for this message
Mikhail Chernik (mchernik) wrote :

snippets are taken on snapshot #688

Changed in mos:
milestone: none → 10.0
status: New → Confirmed
Revision history for this message
Kevin Benton (kevinbenton) wrote :

This should be significantly improved by the following upstream patch: https://review.openstack.org/416400

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Can someone cherry-pick this change and confirm the fix? https://review.openstack.org/#/c/416523/1

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Grab the latest version from https://review.openstack.org/#/c/416523 , not the first one in the series.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Proposed cherry-pick here so we don't have to wait for upstream: https://review.fuel-infra.org/29676

tags: added: wait-for-stable
Revision history for this message
Alexander Ignatov (aignatov) wrote :

Fix https://review.openstack.org/#/c/416524/ was merged in upstream for stable/mitaka. Waiting for regular sync.

Revision history for this message
Alexander Ignatov (aignatov) wrote :

stable/mitaka sync in downstream https://review.fuel-infra.org/#/c/29701/

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/neutron (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/29701
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 38f448751c553098539b787e755b6688dcdf59bf
Author: Jenkins <email address hidden>
Date: Tue Jan 10 12:12:06 2017

Merge the tip of origin/stable/mitaka into origin/9.0/mitaka

67e334b Reduce IP link show calls for SR-IOV scan loop

Closes-Bug: #1653131
Change-Id: I9c8ca8fb807836de944bf845ef9928135f82400a

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/neutron (9.0/mitaka)

Change abandoned by Alexander Ignatov <email address hidden> on branch: 9.0/mitaka
Review: https://review.fuel-infra.org/29676
Reason: Abandoning in favor of https://review.fuel-infra.org/#/c/29701/

Revision history for this message
Alexander Ignatov (aignatov) wrote :
tags: added: on-verification
Revision history for this message
Kristina Berezovskaia (kkuznetsova) wrote :

Verified on 9.2 snap 822

root@node-3:~# grep '12:07:' /var/log/neutron/neutron-sriov-agent.log | grep "'ip', 'link', 'show'" | wc -l
30
root@node-3:~# vim /var/log/neutron/neutron-sriov-agent.log
root@node-3:~# grep '11:03:' /var/log/neutron/neutron-sriov-agent.log | grep "'ip', 'link', 'show'" | wc -l
15
root@node-3:~# ip link show | grep '^[0-9].*' | wc -l
25
root@node-3:~# ip link show | grep '^[0-9].*' | wc ^C
root@node-3:~# ip link show | grep '^[0-9].*' | grep eth | wc -l
8
root@node-3:~# grep '11:04:' /var/log/neutron/neutron-sriov-agent.log | grep "'ip', 'link', 'show'" | wc -l
30

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.