Mirantis OpenStack

[ceilometer] Zombie processes of ceilometer-api on all controllers

Bug #1530128 reported by Timur Nurlygayanov on 2015-12-30

20

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	Mirantis OpenStack	Won't Fix	Medium	Igor Degtiarov	Mirantis OpenStack 8.0-updates
	6.0.x	Won't Fix	Medium	Igor Degtiarov	Mirantis OpenStack 6.0-updates
	8.0.x	Won't Fix	Medium	Igor Degtiarov	Mirantis OpenStack 8.0-updates
	9.x	Won't Fix	Medium	Igor Degtiarov	Mirantis OpenStack 9.0

Bug Description

Steps To Reproduce:
1. Download the latest MOS 8.0 build (#361 in my case)
2. Deploy environment with 3 controllers and 1 compute node, Ceilometer enabled, mongo nodes on controller nodes.
3. Login to controller node
4. Check count of ceilometer processes:

ps aux | grep -v grep | grep ceilometer-api

Observed Result:
We can see zombie processes on all controllers:

root@node-2:~# ps aux | grep -v grep | grep ceilometer-api
ceilome+ 18849 0.0 0.0 0 0 ? Z 13:40 0:00 [ceilometer-api] <defunct>
ceilome+ 30852 2.0 1.1 831828 58052 ? Ssl Dec29 31:40 /usr/bin/python /usr/bin/ceilometer-api --config-file=/etc/ceilometer/ceilometer.conf --log-file=/var/log/ceilometer/ceilometer-api.log

Tags:

Timur Nurlygayanov (tnurlygayanov) on 2015-12-30

Changed in mos:
assignee:	nobody → MOS Ceilometer (mos-ceilometer)
milestone:	none → 8.0
tags:	added: ceilometer

Roman Podoliaka (rpodolyaka) on 2016-01-14

tags:

added: area-ceilometer
removed: ceilometer

Revision history for this message

Igor Degtiarov (idegtiarov) wrote on 2016-01-18:

#1

We found out that that bug is coming from werkzeug, which is creates zombie process if multiple workers run --- link https://github.com/mitsuhiko/werkzeug/issues/810

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-01-18:

#2

So, then we probably can run only one worker for Ceilometer in 8.0 as a workaround?

Changed in mos:
milestone:	8.0 → none
milestone:	none → 8.0

Revision history for this message

Igor Degtiarov (idegtiarov) wrote on 2016-01-18:

#3

We can run only one api-worker and do not create any zombie process but it will crucial decrease performance of api service.

Igor Degtiarov (idegtiarov) on 2016-01-21

Changed in mos:
status:	Confirmed → Won't Fix

Revision history for this message

Igor Degtiarov (idegtiarov) wrote on 2016-01-21:

#4

Zombie process appear after child process have done some GET request and closed. New GET request lead to run new child process which successful closing zombie one. That is werkzeug issue, not Celiometer one.

Andrii Petrenko (aplsms) on 2016-02-09

tags:

added: customer-found support

Revision history for this message

Stanislav Kolenkin (skolenkin) wrote on 2016-02-09:

#5

We have found the similar problem in MOS 6.0
108 1069 0.4 0.0 146656 44928 ? Ss Feb08 4:48 /usr/bin/python /usr/bin/ceilometer-api --log-dir=/var/log/ceilometer
108 20032 0.6 0.0 105448 34516 ? Rs Feb08 7:41 /usr/bin/python /usr/bin/ceilometer-agent-central
108 21159 0.0 0.0 119192 33684 ? Ss Feb08 0:47 /usr/bin/python /usr/bin/ceilometer-agent-compute
108 22176 1.2 0.0 76216 19576 ? Ss Feb08 14:43 /usr/bin/python /usr/bin/ceilometer-agent-notification
108 22217 2.8 0.0 92056 23848 ? S Feb08 33:03 /usr/bin/python /usr/bin/ceilometer-agent-notification
108 24710 0.2 0.0 101092 30252 ? Ss Feb08 3:30 /usr/bin/python /usr/bin/ceilometer-alarm-evaluator --log-dir=/var/log/ceilometer
108 27534 2.0 0.0 99360 28312 ? Ss Feb08 24:15 /usr/bin/python /usr/bin/ceilometer-alarm-notifier --log-dir=/var/log/ceilometer
108 34069 1.2 0.0 73396 16992 ? Ss Feb08 14:21 /usr/bin/python /usr/bin/ceilometer-collector
108 34078 10.1 0.0 98548 26276 ? S Feb08 118:10 /usr/bin/python /usr/bin/ceilometer-collector
108 34887 1.0 0.0 0 0 ? Z 01:36 0:00 [ceilometer-api] <defunct>
108 35076 1.0 0.0 0 0 ? Z 01:36 0:00 [ceilometer-api] <defunct>

ii ceilometer-agent-central 2014.2-fuel6.0~mira25 ceilometer central agent
ii ceilometer-agent-compute 2014.2-fuel6.0~mira25 ceilometer compute agent
ii ceilometer-agent-notification 2014.2-fuel6.0~mira25 ceilometer notification agent
ii ceilometer-alarm-evaluator 2014.2-fuel6.0~mira25 Ceilometer alarm evaluator
ii ceilometer-alarm-notifier 2014.2-fuel6.0~mira25 Ceilometer notification agent
ii ceilometer-api 2014.2-fuel6.0~mira25 ceilometer api service
ii ceilometer-collector 2014.2-fuel6.0~mira25 ceilometer collector service
ii ceilometer-common 2014.2-fuel6.0~mira25 ceilometer common files
ii python-ceilometer 2014.2-fuel6.0~mira25 ceilometer python libraries
ii python-ceilometerclient 1.0.12-fuel6.1~miracustom1 Client library for Openstack ceilometer server.

We have found the similar problem in MOS 6.0
108       1069  0.4  0.0 146656 44928 ?        Ss   Feb08   4:48 /usr/bin/python /usr/bin/ceilometer-api --log-dir=/var/log/ceilometer
108      20032  0.6  0.0 105448 34516 ?        Rs   Feb08   7:41 /usr/bin/python /usr/bin/ceilometer-agent-central
108      21159  0.0  0.0 119192 33684 ?        Ss   Feb08   0:47 /usr/bin/python /usr/bin/ceilometer-agent-compute
108      22176  1.2  0.0  76216 19576 ?        Ss   Feb08  14:43 /usr/bin/python /usr/bin/ceilometer-agent-notification
108      22217  2.8  0.0  92056 23848 ?        S    Feb08  33:03 /usr/bin/python /usr/bin/ceilometer-agent-notification
108      24710  0.2  0.0 101092 30252 ?        Ss   Feb08   3:30 /usr/bin/python /usr/bin/ceilometer-alarm-evaluator --log-dir=/var/log/ceilometer
108      27534  2.0  0.0  99360 28312 ?        Ss   Feb08  24:15 /usr/bin/python /usr/bin/ceilometer-alarm-notifier --log-dir=/var/log/ceilometer
108      34069  1.2  0.0  73396 16992 ?        Ss   Feb08  14:21 /usr/bin/python /usr/bin/ceilometer-collector
108      34078 10.1  0.0  98548 26276 ?        S    Feb08 118:10 /usr/bin/python /usr/bin/ceilometer-collector
108      34887  1.0  0.0      0     0 ?        Z    01:36   0:00 [ceilometer-api] <defunct>
108      35076  1.0  0.0      0     0 ?        Z    01:36   0:00 [ceilometer-api] <defunct>

ii  ceilometer-agent-central            2014.2-fuel6.0~mira25                               ceilometer central agent
ii  ceilometer-agent-compute            2014.2-fuel6.0~mira25                               ceilometer compute agent
ii  ceilometer-agent-notification       2014.2-fuel6.0~mira25                               ceilometer notification agent
ii  ceilometer-alarm-evaluator          2014.2-fuel6.0~mira25                               Ceilometer alarm evaluator
ii  ceilometer-alarm-notifier           2014.2-fuel6.0~mira25                               Ceilometer notification agent
ii  ceilometer-api                      2014.2-fuel6.0~mira25                               ceilometer api service
ii  ceilometer-collector                2014.2-fuel6.0~mira25                               ceilometer collector service
ii  ceilometer-common                   2014.2-fuel6.0~mira25                               ceilometer common files
ii  python-ceilometer                   2014.2-fuel6.0~mira25                               ceilometer python libraries
ii  python-ceilometerclient             1.0.12-fuel6.1~miracustom1                          Client library for Openstack ceilometer server.

Revision history for this message

Eugene Nikanorov (enikanorov) wrote on 2016-02-23:

#6

Why is it won't fix for 8.0 and 9.0?!

Revision history for this message

Eugene Nikanorov (enikanorov) wrote on 2016-02-23:

#7

If it's a problem of underlying library - we need to make a workaround - in the downstream repo and in the upstream preferably.

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-02-25:

#8

Medium bug, not fixed in MOS 8.0, moved to MOS 8.0 updates.

Revision history for this message

Nadya Privalova (nprivalova) wrote on 2016-03-30:

#9

Moved to won't fixed because the problem is in werkzeug. We will get rid of this server in MOS 10

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.