Keystone crash with coredump during apache stop

Michał, could you please clarify which version of MOS you are using?

Btw, is Keystone (apache) restarted automatically after core dump?

hey,
yeah its working fine after restart + coredump generated.

On Wed, May 4, 2016 at 3:42 PM, Roman Podoliaka <
<email address hidden>> wrote:

> Btw, is Keystone (apache) restarted automatically after core dump?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1575106
>
> Title:
> Keystone crash with coredump during apache stop
>
> Status in Mirantis OpenStack:
> Incomplete
>
> Bug description:
> Hey,
> i have lots of coredump from keystone during apache stop/restart
>
> ~# keystone --version
> 1.3.1
>
> (gdb) where
> #0 0x00007f32631c7730 in _PyTrash_thread_destroy_chain () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #1 0x00007f326317ff4e in PyEval_EvalFrameEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #2 0x00007f326318154d in PyEval_EvalCodeEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #3 0x00007f326317fdd8 in PyEval_EvalFrameEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #4 0x00007f326318154d in PyEval_EvalCodeEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #5 0x00007f326317fdd8 in PyEval_EvalFrameEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #6 0x00007f326318154d in PyEval_EvalCodeEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #7 0x00007f32631b67a5 in ?? () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #8 0x00007f3263122d43 in PyObject_Call () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #9 0x00007f326317c3b1 in PyEval_EvalFrameEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #10 0x00007f3263180059 in PyEval_EvalFrameEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #11 0x00007f3263180059 in PyEval_EvalFrameEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #12 0x00007f326318154d in PyEval_EvalCodeEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #13 0x00007f32631b66d0 in ?? () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #14 0x00007f3263122d43 in PyObject_Call () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #15 0x00007f32630ae7bd in ?? () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #16 0x00007f3263122d43 in PyObject_Call () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #17 0x00007f326319b577 in PyEval_CallObjectWithKeywords () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #18 0x00007f3263103f92 in ?? () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> #19 0x00007f3269a8f182 in start_thread (arg=0x7f32414e3700) at
> pthread_create.c:312
> #20 0x00007f32697bc47d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
>
> (gdb) bt full
> #0 0x00007f32631c7730 in _PyTrash_thread_destroy_chain () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> No symbol table info available.
> #1 0x00007f326317ff4e in PyEval_EvalFrameEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> No symbol table info available.
> #2 0x00007f326318154d in PyEval_EvalCodeEx () from
> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
> No symbol table info available.
> #3 ...

This was reported by client on MOS 7.0

What mod_wsgi version they use?

If apache (mod_wsgi) restarts Keystone automatically after core dump and it works correctly, i.e. no actions are needed from the operator side, I suggest we keep this as Medium.

It's not clear if MOS versions 8.0+ are affected - we might have upgraded mod_wsgi since 7.0. MOS Keystone, please check that.

stable/7.0 iso:
http://jenkins-product.srt.mirantis.net:8080/job/custom_7.0_iso/1345/artifact/artifacts/listing.txt
libapache2-mod-wsgi_3.4-4ubuntu2.1.14.04.2~u14.04+mos1_amd64.deb

stable/8.0 iso:
http://jenkins-product.srt.mirantis.net:8080/job/custom_8.0_iso/1836/artifact/artifacts/listing.txt
libapache2-mod-wsgi_4.4.15-0.1_amd64.deb

We had similar issue(modwsgi) https://bugs.launchpad.net/fuel/+bug/1493353 for 7.0 release. But, it seems that in 8.0 release we have modwsgi's version what was proposed by Graham Dumpleton in https://github.com/GrahamDumpleton/mod_wsgi/issues/81 . It looks like the simplest way to resolve an issue is to upgrade modwsgi version, but, I'm not sure that it's possible during the maintenance cycle.

Please update mod_wsgi: till 4.1.25(or smth. like that) it had graceful restart bug.

Fix proposed to branch: 7.0
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/20341

I didn't have the detailed steps for reproducing this bug, so i have tried to perform graceful restart for apache and I didn't see any coredumps. Keystone is restarted and keeps working without any problem.

MOS 10.0 Build #187

Please check https://github.com/GrahamDumpleton/mod_wsgi/issues/81

4.4.15 didn't solve issue.

But setting WSGIApplicationGroup %{GLOBAL} mitigates how often we can see problem.

To reproduce issue you can use:
# for i in `seq 10`; do /etc/init.d/apache2 graceful; done
 * Reloading web server apache2
 *
 * Reloading web server apache2
 *
 * Reloading web server apache2
 *
 * Apache2 is not running
 * Reloading web server apache2
 *
 * Apache2 is not running
 * Reloading web server apache2

After apache is down, in logs you can find:
# tail /var/log/apache2/error.log
[Wed May 11 11:30:55.605894 2016] [core:notice] [pid 2608] AH00060: seg fault or similar nasty error detected in the parent process

Can't find reason why this bug is on CI team..
MOS Keystone team, are there any required actions from CI related to 7.0.x series?

mos-keystone team, please clarify if any actions needed to work at this bug

There is no existing fix for it. In the ISO we have implemented a set of retries and sleeps in order to mitigate the issue. Apache wiki describes this problem: https://wiki.apache.org/httpd/CouldNotBindToAddress .

From the bugreport it is not clear to me why you think that keystone is involved. What makes you think it does?

Regardless, there is little keystone team can do here. The issue does not happen during the operation of keystone. It doesn't affect HA or scalability. The operator running apache restarts should implement retries and waits.

I am moving it to "wishlist".