Just saw this on stable/newton and having looked at the logs I think this is a logging issue in libvirtd when terminating an instance created by the ImagesTestJSON tests.
- Attempt to terminate the ImagesTestJSON instance prior to libvirtd dying :
http://logs.openstack.org/61/407961/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/3dc2b54/logs/screen-n-cpu.txt.gz#_2016-12-12_07_48_37_547
- Call to qemuDomainLogContextRead that I think is triggering this issue :
http://logs.openstack.org/61/407961/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/3dc2b54/logs/libvirt/libvirtd.txt.gz#_2016-12-12_07_48_37_835
- Further call logged to virLogProbablyLogMessage in syslog :
http://logs.openstack.org/61/407961/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/3dc2b54/logs/syslog.txt.gz#_Dec_12_07_48_37
Kashyap, could you raise this with the libvirt folks for their take on this?
~~~ Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: *** Error in `/usr/sbin/libvirtd': malloc(): memory corruption: 0x000055a98c7c6810 *** Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: ======= Backtrace: ========= Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fc916b557e5] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /lib/x86_64-linux-gnu/libc.so.6(+0x8181e)[0x7fc916b5f81e] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fc916b615d4] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /lib/x86_64-linux-gnu/libc.so.6(realloc+0x358)[0x7fc916b61e68] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /lib/x86_64-linux-gnu/libc.so.6(+0xe53a2)[0x7fc916bc33a2] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /lib/x86_64-linux-gnu/libc.so.6(regexec+0xb3)[0x7fc916bc59c3] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /usr/lib/x86_64-linux-gnu/libvirt.so.0(virLogProbablyLogMessage+0x1f)[0x7fc917563a5f] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so(+0x7d080)[0x7fc8c7ddf080] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so(+0x933dc)[0x7fc8c7df53dc] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /usr/lib/x86_64-linux-gnu/libvirt.so.0(virEventPollRunOnce+0x7b3)[0x7fc91754d393] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /usr/lib/x86_64-linux-gnu/libvirt.so.0(virEventRunDefaultImpl+0x41)[0x7fc91754bc01] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /usr/lib/x86_64-linux-gnu/libvirt.so.0(virNetDaemonRun+0x14d)[0x7fc917697b4d] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /usr/sbin/libvirtd(+0x14407)[0x55a98c480407] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fc916afe830] Dec 12 07:48:37 ubuntu-xenial-osic-cloud1-disk-6107639 libvirtd[29343]: /usr/sbin/libvirtd(+0x14bd9)[0x55a98c480bd9] ~~~
Just saw this on stable/newton and having looked at the logs I think this is a logging issue in libvirtd when terminating an instance created by the ImagesTestJSON tests.
- Attempt to terminate the ImagesTestJSON instance prior to libvirtd dying :
http:// logs.openstack. org/61/ 407961/ 1/gate/ gate-tempest- dsvm-neutron- full-ubuntu- xenial/ 3dc2b54/ logs/screen- n-cpu.txt. gz#_2016- 12-12_07_ 48_37_547
- Call to qemuDomainLogCo ntextRead that I think is triggering this issue :
http:// logs.openstack. org/61/ 407961/ 1/gate/ gate-tempest- dsvm-neutron- full-ubuntu- xenial/ 3dc2b54/ logs/libvirt/ libvirtd. txt.gz# _2016-12- 12_07_48_ 37_835
- Further call logged to virLogProbablyL ogMessage in syslog :
http:// logs.openstack. org/61/ 407961/ 1/gate/ gate-tempest- dsvm-neutron- full-ubuntu- xenial/ 3dc2b54/ logs/syslog. txt.gz# _Dec_12_ 07_48_37
Kashyap, could you raise this with the libvirt folks for their take on this?
~~~ xenial- osic-cloud1- disk-6107639 libvirtd[29343]: *** Error in `/usr/sbin/ libvirtd' : malloc(): memory corruption: 0x000055a98c7c6810 *** xenial- osic-cloud1- disk-6107639 libvirtd[29343]: ======= Backtrace: ========= xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /lib/x86_ 64-linux- gnu/libc. so.6(+0x777e5) [0x7fc916b557e5 ] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /lib/x86_ 64-linux- gnu/libc. so.6(+0x8181e) [0x7fc916b5f81e ] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /lib/x86_ 64-linux- gnu/libc. so.6(__ libc_malloc+ 0x54)[0x7fc916b 615d4] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /lib/x86_ 64-linux- gnu/libc. so.6(realloc+ 0x358)[ 0x7fc916b61e68] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /lib/x86_ 64-linux- gnu/libc. so.6(+0xe53a2) [0x7fc916bc33a2 ] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /lib/x86_ 64-linux- gnu/libc. so.6(regexec+ 0xb3)[0x7fc916b c59c3] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /usr/lib/ x86_64- linux-gnu/ libvirt. so.0(virLogProb ablyLogMessage+ 0x1f)[0x7fc9175 63a5f] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /usr/lib/ libvirt/ connection- driver/ libvirt_ driver_ qemu.so( +0x7d080) [0x7fc8c7ddf080 ] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /usr/lib/ libvirt/ connection- driver/ libvirt_ driver_ qemu.so( +0x933dc) [0x7fc8c7df53dc ] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /usr/lib/ x86_64- linux-gnu/ libvirt. so.0(virEventPo llRunOnce+ 0x7b3)[ 0x7fc91754d393] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /usr/lib/ x86_64- linux-gnu/ libvirt. so.0(virEventRu nDefaultImpl+ 0x41)[0x7fc9175 4bc01] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /usr/lib/ x86_64- linux-gnu/ libvirt. so.0(virNetDaem onRun+0x14d) [0x7fc917697b4d ] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /usr/sbin/ libvirtd( +0x14407) [0x55a98c480407 ] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /lib/x86_ 64-linux- gnu/libc. so.6(__ libc_start_ main+0xf0) [0x7fc916afe830 ] xenial- osic-cloud1- disk-6107639 libvirtd[29343]: /usr/sbin/ libvirtd( +0x14bd9) [0x55a98c480bd9 ]
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
Dec 12 07:48:37 ubuntu-
~~~