nailgun agent: segfault due to ruby-sigar bug

Fix proposed to branch: master
Review: https://review.openstack.org/190225

Looks like this version of Ruby p484 has many problem with SIG-FAULT. Workaround with GC.disable has not helped.

Another similar bugs:

* https://bugs.ruby-lang.org/issues/9679
* https://communities.vmware.com/message/2349694

Looks like we have 2 possible solution:

* we can update ohai gem to 6.16 instead of 6.14;
* we can update it to Ruby 1.9.3-p545 and check it again.

After ohai update it start works fine. I rollback it to 6.14 and it is still works fine. But SIG-FAULT has very random nature.

Additional info:

Obtained service url from config file: 'http://10.20.0.2:8000/api'
/usr/lib/ruby/vendor_ruby/ohai/mash.rb:192: [BUG] rb_gc_mark(): unknown data type 0x0(0x1fbf7e0) non object
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

I prefer to limit changes at this moment and update of gem looks better.

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: master
Review: https://review.openstack.org/190225

> * we can update it to Ruby 1.9.3-p545 and check it again

No, we can't. Rebuilding mcollective, puppet, and all other ruby software in Ubuntu is not an option.

> Another similar bugs:

> * https://bugs.ruby-lang.org/issues/9679

Could you elaborate, please:

a) What is exactly similar here (other than being terminated by SIGSEGV, which is pretty common consequence of memory management errors)
b) Are you sure this is ruby runtime bug (rather than bug of chef itself)?
c) Are you sure Ubuntu version of ruby 1.9.1 is the same/older?

Which extension, exactly?
How do I reproduce the error (a snippet of ruby code, please)?

I've started packaging ohai 6.16, but I don't think the update can possibly solve the problem. ohai itself is purely ruby code, so I doubt it can segfault.

As far as I understand the only C extensions which used by ohai are ruby-yajl, ruby-sigar.

The problem might be caused by https://github.com/hyperic/sigar/issues/28 (which is not solved yet)

I guess we need to wrap every sigar usage in ohai like this: https://github.com/m1093782566/bosh-cloudstack-cpi/commit/7adcca3a43fd5a63ec2297531f580f5eb32b762b

The following code segfaults for me:

require 'sigar'
require 'logger'

GC.disable

sigars = []

100.times do
  sigar = Sigar.new
  #sigar.logger = Logger.new(STDOUT)
  sigars << sigar
end

puts "Go"
GC.enable
GC.start

sleep 10

sigarbug.rb:17: [BUG] rb_gc_mark(): unknown data type 0x10(0x7f811c5817c8) non object
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0004 p:---- s:0010 b:0010 l:000009 d:000009 CFUNC :start
c:0003 p:0101 s:0007 b:0007 l:0019b8 d:002550 EVAL sigarbug.rb:17
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:0019b8 d:0019b8 TOP

-- Ruby level backtrace information ----------------------------------------
sigarbug.rb:17:in `<main>'
sigarbug.rb:17:in `start'

-- C level backtrace information -------------------------------------------
/usr/lib/libruby-1.9.1.so.1.9(+0x155bb9) [0x7f811c6ddbb9] ../sysdeps/x86_64/multiarch/../strcmp.S:1007
/usr/lib/libruby-1.9.1.so.1.9(+0x5966d) [0x7f811c5e166d] vfscanf.c:2350
/usr/lib/libruby-1.9.1.so.1.9(rb_bug+0xb3) [0x7f811c5e1de3] vfscanf.c:1602
/usr/lib/libruby-1.9.1.so.1.9(+0x6ef90) [0x7f811c5f6f90] iogetdelim.c:56
/usr/lib/libruby-1.9.1.so.1.9(+0x6f0e1) [0x7f811c5f70e1] iogetdelim.c:90
/usr/lib/libruby-1.9.1.so.1.9(rb_gc+0x1a) [0x7f811c5f830a] ioseekpos.c:34
/usr/lib/libruby-1.9.1.so.1.9(rb_gc_start+0x9) [0x7f811c5f8349] ioseekpos.c:49
/usr/lib/libruby-1.9.1.so.1.9(+0x14c1d3) [0x7f811c6d41d3] ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1374
/usr/lib/libruby-1.9.1.so.1.9(+0x146a95) [0x7f811c6cea95] ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:1444
/usr/lib/libruby-1.9.1.so.1.9(+0x149d8e) [0x7f811c6d1d8e] ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2078
/usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0x9f) [0x7f811c6da85f] ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:2561
/usr/lib/libruby-1.9.1.so.1.9(+0x5ca1a) [0x7f811c5e4a1a] vfscanf.c:1956
/usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0x1d) [0x7f811c5e585d] ../sysdeps/posix/tempname.c:203
/usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x1e) [0x7f811c5e730e] vfprintf.c:1422
ruby1.9.1() [0x40086b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f811c1e4ec5] libc-start.c:287
ruby1.9.1() [0x400899]

-- Other runtime information -----------------------------------------------

* Loaded script: sigarbug.rb

* Loaded features:

    0 enumerator.so
    1 /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
    2 /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
    3 /usr/lib/ruby/1.9.1/rubygems/defaults.rb
    4 /usr/lib/ruby/1.9.1/x86_64-linux/rbconfig.rb
    5 /usr/lib/ruby/1.9.1/rubygems/deprecate.rb
    6 /usr/lib/ruby/1.9.1/rubygems/exceptions.rb
    7 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb
    8 /usr/lib/ruby/1.9.1/rubygems/custom_require.rb
    9 /usr/lib/ruby/1.9.1/rubygems.rb
   10 /usr/lib/ruby/vendor_ruby/1.9.1/x86_64-linux/sigar.so
   11 /usr/lib/ruby/1.9.1/thread.rb
   12 /usr/lib/ruby/1.9.1/monitor.rb
   13 /usr/lib/ruby/1.9.1/logger.rb

* Process memory map:

00400000-00401000 r-xp 00000000 fc:00 30718 /usr/bin/ruby1.9.1
00600000-0060100...

The segfault can be reproduced with ruby 2.1.5 and 2.0.0.484, so upgrading ruby won't help anyway (provided that I've correctly identified the root cause).

Guys, there's no doubts that we do have the issue. Still, the issue is rare and we didn't see it on CI, so why do we want to fix it ASAP in 6.1? It looks to me as a good candidate to be moved to 7.0, because I really don't see a *safe* fix right now.

The attached patch should solve the problem. However it breaks ohai unit tests (and might be totally wrong)

Alexei Sheplyakov (asheplyakov), i try to use GC.disable in main script on problem node and it has not helped. I still got this error.

In other hands after installation new version of ohai or reinstall old version of ohai problem disappear.

Hmm, let try to use patch. +1 for me. Also i suggest to merge Dmitry Ilyin changes.

I talk about this changes: https://review.openstack.org/#/c/190225/

GC.disable is not necessary (and it won't help since other modules can turn it on).

Reproduced again: https://bugs.launchpad.net/fuel/+bug/1475290

After discussion with Alexei Sheplyakov we need fix tests which broken if we apply patch. Assigned to myself.

Change abandoned by Dmitry Pyzhov (<email address hidden>) on branch: master
Review: https://review.openstack.org/190225
Reason: No activity for more than a month

Guys, the same error was reproduced on scale lab 24 July 2015, so it is still actual. Is there anything preventing us from applying this patch with SigarBox suggested by Alexey?

Another suggestion is to disable network_listeners plugin since it gathers the information about network connections and we don't need this in nailgun-agent

https://github.com/stackforge/fuel-nailgun-agent/blob/master/agent#L163

I've checked Vladimir Kozhukalov. Sorry, but this will not work, because Sugar is used in many places including CPU, Networks, Disks and in many others.

Fix tests. Now can safely apply this patch and include it in our ohai package.

Fix tests. Now can safely apply this patch and include it in our ohai package.

Related fix proposed to branch: master
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/10374

Reviewed: https://review.fuel-infra.org/10374
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 17a594c444a57087ccdf863b253bcdf8d3b51ffa
Author: Alexei Sheplyakov <email address hidden>
Date: Wed Aug 12 14:19:17 2015

Add ohai custom package for Ubuntu

The original ohai package suffers from a segfault in a C extension
(ruby-sigar). This segfault causes false alarms (the node goes offline
from the nailgun point of the view) and sporadic tests failures.
Apparently fixing ruby-sigar itself is rather complicated, it's much
easier to work around the problem in ohai itself. Hence a custom
package.

Change-Id: Idab24a61df562c312ea47234aa02012223b7b0a0
Related-Bug: #1463835

Fix proposed to branch: 7.0
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/10377

Reviewed: https://review.fuel-infra.org/10377
Submitter: Michael Semenov <email address hidden>
Branch: 7.0

Commit: 7219d5622e11e8a49f5e4713d7405986ea63c176
Author: Alexei Sheplyakov <email address hidden>
Date: Wed Aug 12 15:34:32 2015

Work around the segfault in ruby-sigar

ruby-sigar attempts to GC a pointer which is not ruby object which yields
a segfault [1]. This segfault causes false alarms (the node goes offline
from the nailgun point of the view) and sporadic tests failures. Apparently
fixing ruby-sigar itself is rather complicated, it's much easier to work
around the problem in ohai itself, hence a custom package.

[1] https://github.com/hyperic/sigar/issues/28

The original source has been downloaded from
http://archive.ubuntu.com/ubuntu/pool/universe/o/ohai/ohai_6.14.0.orig.tar.gz
http://archive.ubuntu.com/ubuntu/pool/universe/o/ohai/ohai_6.14.0-2.debian.tar.gz

Closes-Bug: #1463835
Change-Id: Icf0da3a829863530b95ba88cd671bdc82fe08f51

Reassigning bug MOS 6.1 to MOS sustaining team (they are the right guys to decide if the fix needs to be backported to MOS 6.1)

Verified on ISO #219 availability of the needed package.
I also did not see this error on latest ISOs

"build_id": "2015-08-23_15-01-12", "build_number": "219", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-23_15-01-12", "build_number": "219", "api": "1.0", "fuel-library_sha": "3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51", "nailgun_sha": "7790ce872512ecdf21689e6a5f970dd7119febdb", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "16839cbf471b7142b04c0d2c2d94786bc486fefe", "release": "7.0", "fuelmain_sha": "a494e6628319abfef57e1754f6453cf8f1a4bc65"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51", "nailgun_sha": "7790ce872512ecdf21689e6a5f970dd7119febdb", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "16839cbf471b7142b04c0d2c2d94786bc486fefe", "release": "7.0", "fuelmain_sha": "a494e6628319abfef57e1754f6453cf8f1a4bc65"

Reproduced on ISO #228

"build_id": "2015-08-25_03-34-40", "build_number": "228", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-25_03-34-40", "build_number": "228", "api": "1.0", "fuel-library_sha": "1d23fa825a73564bb6c715993ea9df8a66249a93", "nailgun_sha": "9f227f2cc0ca276715cb098c73b2bd231b103eae", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "3ad03d076c46347691cc3480dd19d34e37b73df4", "release": "7.0", "fuelmain_sha": "28d4bfcff7a0fb1b37504dbcac4998789df17935"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "1d23fa825a73564bb6c715993ea9df8a66249a93", "nailgun_sha": "9f227f2cc0ca276715cb098c73b2bd231b103eae", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "3ad03d076c46347691cc3480dd19d34e37b73df4", "release": "7.0", "fuelmain_sha": "28d4bfcff7a0fb1b37504dbcac4998789df17935"

Test case:
1. Create new environment
2. Choose Neutron, VLAN
3. Choose Ceph for volumes
4. Choose Murano
5. Add 3 controller, 1 compute3 ceph
6. Start deployment
7. Stop deployment during provisioning
8. Wait until nodes become 'Pending addition'
9. Deploy the environment again. It was successful
10. But controller (node-3) has gone away and back online many times

Issue is fixed but we have another problem https://bugs.launchpad.net/fuel/+bug/1488844

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/13248

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/13248
Reason: MOS 6.1 is based on Ubuntu 14.04 (trusty).

Fix proposed to branch: 6.1
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/13513

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/13609

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/13513

Reviewed: https://review.fuel-infra.org/13609
Submitter: Vitaly Sedelnik <email address hidden>
Branch: 6.1

Commit: 54bcb1dac1c9e35ea1fbf7b88fcef96220c57841
Author: Denis Puchkin <email address hidden>
Date: Tue Nov 3 11:23:34 2015

Work around the segfault in ruby-sigar

ruby-sigar attempts to GC a pointer which is not ruby object which yields
a segfault [1]. This segfault causes false alarms (the node goes offline
from the nailgun point of the view) and sporadic tests failures. Apparently
fixing ruby-sigar itself is rather complicated, it's much easier to work
around the problem in ohai itself, hence a custom package.

[1] https://github.com/hyperic/sigar/issues/28

The original source has been downloaded from
http://archive.ubuntu.com/ubuntu/pool/universe/o/ohai/ohai_6.14.0.orig.tar.gz
http://archive.ubuntu.com/ubuntu/pool/universe/o/ohai/ohai_6.14.0-2.debian.tar.gz

This is a backport of 7219d5622e11e8a49f5e4713d7405986ea63c176 for mos6.1

Closes-Bug: #1463835
Change-Id: I8e05d0125b790c619140b4c982d3333806a4e4e2

The ohai package can't be build for mos 6.1, because OBS/6.1 CI can't use 'trusty/universe' repo.
in this repo located necessary packages to meet the build dependencies

To resolve unmet dependencies issue we should re-build 7 packages from 'trusty/universe' for mos 6.1, it is :
ruby-mixlib-config
ruby-mixlib-log
ruby-mixlib-cli
ruby-systemu
ruby-yajl
ruby-sigar
ruby-ipaddress

> The ohai package can't be build for mos 6.1, because OBS/6.1 CI can't use 'trusty/universe' repo.

Please fix the CI to lift this silly restriction.

> To resolve unmet dependencies issue we should re-build 7 packages from 'trusty/universe' for mos 6.1

ruby-mixlib-config
ruby-mixlib-log
ruby-mixlib-cli
ruby-systemu
ruby-yajl
ruby-sigar
ruby-ipaddress

Most of ruby gems are in the `universe' section, rebuilding them to make CI happy is plain wrong.

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14441

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14444

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14448

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14450

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14451

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14465

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14441

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14445

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14446

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14450

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14444

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14448

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14451

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14465