Mirantis OpenStack

"OSError: [Errno 24] Too many open files" during cleanup 2000 dhcp namespaces

Bug #1499729 reported by Leontiy Istomin
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
High
Sergey Kolekonov
Mirantis OpenStack 8.0
7.0.x
Won't Fix
Medium
Ilya Shakhat
Mirantis OpenStack 7.0-updates
8.0.x
Fix Released
High
Sergey Kolekonov
Mirantis OpenStack 8.0

Bug Description

I've tried to reproduce this bug https://bugs.launchpad.net/fuel/+bug/1436414 using the following steps:
root@node-6:~# ip netns | grep -c qdhcp
3
 * Create 2k namespaces:
   for i in {1000..2999}; do ip netns add "qdhcp-6a6302b1-37a7-4142-a11d-5be2778f$i" ; done
 * Run q-agent-cleanup:
   neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/dhcp_agent.ini
after command execution:
root@node-6:~# ip netns | grep -c qdhcp
2003
from /var/log/neutron/neutron-netns-cleanup.log on node-6: http://paste.openstack.org/show/474018/

Cluster Configuration:
Baremetal,Ubuntu,IBP,HA,Neutron-VxLAN,Сeph-all,Nova-debug,Nova-quotas,7.0-301
Controllers:3 Computes+Ceph:15 LMA:2

api: '1.0'
astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
auth_required: true
build_id: '301'
build_number: '301'
feature_groups:
- mirantis
fuel-agent_sha: 50e90af6e3d560e9085ff71d2950cfbcca91af67
fuel-library_sha: 5d50055aeca1dd0dc53b43825dc4c8f7780be9dd
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c
fuelmain_sha: a65d453215edb0284a2e4761be7a156bb5627677
nailgun_sha: 4162b0c15adb425b37608c787944d1983f543aa8
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 486bde57cda1badb68f915f66c61b544108606f3
release: '7.0'

Diagnostic Snapshot will be soon.

See original description
Leontiy Istomin (listomin)
affects: fuel → mos
Changed in mos:
assignee: nobody → MOS Neutron (mos-neutron)
milestone: none → 7.0
Ilya Shakhat (shakhat)
Changed in mos:
status: New → Triaged
importance: Undecided → High
milestone: 7.0 → 7.0-updates
Leontiy Istomin (listomin)
description: updated
Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Won't Fix for 7.0-updates because of Medium importance

Revision history for this message
Ilya Shakhat (shakhat) wrote :

Upstream pyroute2 bug: https://github.com/svinota/pyroute2/issues/194
The fix is in version 0.3.15

Revision history for this message
Ilya Shakhat (shakhat) wrote :

The current version fails on ~160 namespaces. Raising the priority to High

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/trusty/pyroute2 (8.0)

Fix proposed to branch: 8.0
Change author: Sergey Kolekonov <email address hidden>
Review: https://review.fuel-infra.org/14526

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to packages/centos7/python-pyroute2 (8.0)

Related fix proposed to branch: 8.0
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/14737

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to packages/trusty/pyroute2 (8.0)

Reviewed: https://review.fuel-infra.org/14526
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: b36afc3aee21a7c4250f08373483f61ffa3fdb41
Author: Sergey Kolekonov <email address hidden>
Date: Thu Dec 17 09:33:57 2015

Add pyroute2 0.3.15 for MOS 8.0

Due to bug in pyroute 0.3.10 netns leaked open unix sockets until the open file
limit is reached, it led to neutron-ns-cleanup utility failures on large
number of namespaces.
pyroute2 bug: https://github.com/svinota/pyroute2/issues/194

Change-Id: I2456cfed5cbf7ae5ae57254d6c0fcca349dff9a4
Closes-bug: #1499729

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to packages/centos7/python-pyroute2 (8.0)

Related fix proposed to branch: 8.0
Change author: Sergey Kolekonov <email address hidden>
Review: https://review.fuel-infra.org/15936

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to packages/centos7/python-pyroute2 (8.0)

Reviewed: https://review.fuel-infra.org/15936
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: a9e4c58dcb31fbd7d85c263673520e03c7e911f7
Author: Sergey Kolekonov <email address hidden>
Date: Thu Dec 31 09:10:55 2015

Add pyroute2 0.3.15 for CentOS 7

Due to bug in pyroute 0.3.10 netns leaked open unix sockets until the open file
limit is reached, it led to neutron-ns-cleanup utility failures on large
number of namespaces.
pyroute2 bug: https://github.com/svinota/pyroute2/issues/194

Change-Id: I6fc1443a6210e354d88c012e30f25c5759546f4a
Related-bug: #1499729

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to packages/centos7/python-pyroute2 (master)

Related fix proposed to branch: master
Change author: Sergey Kolekonov <email address hidden>
Review: https://review.fuel-infra.org/15937

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to packages/centos7/python-pyroute2 (master)

Reviewed: https://review.fuel-infra.org/15937
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: 53504660755a66aacf8ea813db88d02261b39b3b
Author: Sergey Kolekonov <email address hidden>
Date: Thu Dec 31 10:17:53 2015

Add pyroute2 0.3.15 for CentOS 7

Due to bug in pyroute 0.3.10 netns leaked open unix sockets until the open file
limit is reached, it led to neutron-ns-cleanup utility failures on large
number of namespaces.
pyroute2 bug: https://github.com/svinota/pyroute2/issues/194

Change-Id: I6fc1443a6210e354d88c012e30f25c5759546f4a
Related-bug: #1499729
(cherry picked from commit a9e4c58dcb31fbd7d85c263673520e03c7e911f7)

Anastasia Kuznetsova (akuznetsova)
tags: added: neutron
Anastasia Kuznetsova (akuznetsova)
tags: added: area-neutron
removed: neutron
Egor Kotko (ykotko)
tags: added: on-verification
Revision history for this message
Ilya Shakhat (shakhat) wrote :

It's observed that after step #2 empty namespaces are not removed. This behaviour is wrong, however it does not impact working system. Everywhere in OCF scripts neutron-netns-cleanup is always called with "--force" parameter, and it works as expected.

The case without --force breaks on line #330 -- the condition doesn't take into account that namespace always has loopback interface. @ykotko - please file a bug regarding this.

Revision history for this message
Egor Kotko (ykotko) wrote :

Done. The link: https://bugs.launchpad.net/mos/+bug/1541470

Revision history for this message
Sergey Arkhipov (sarkhipov) wrote :

Verified on build #496 on scale env 14. Problem with namespace cleanups is https://bugs.launchpad.net/mos/+bug/1541470 as mentioned above.

Olga Gusarenko (ogusarenko)
tags: added: 8.0 release-notes-done
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.