Fuel for OpenStack

ssh failed to run that caused master node unaccessability by 22 port

  1. Series 8.0.x
  2. Bug #1539511
Bug #1539511 reported by Tatyanka
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Sergey Kulanov
Fuel for OpenStack 9.0
8.0.x
Confirmed
High
Fuel Sustaining
Fuel for OpenStack 8.0-updates

Bug Description

https://product-ci.infra.mirantis.net/view/9.0-liberty/job/9.0-liberty.ubuntu.smoke_neutron/108/console

Test failed with:
ERROR: Create environment and set up master node
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/helpers/decorators.py", line 83, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/tests/base_test_case.py", line 167, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/models/environment.py", line 444, in setup_environment
    self.wait_for_provisioning()
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/models/environment.py", line 519, in wait_for_provisioning
    (self.d_env.admin_net), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 108, in _wait
    return raising_predicate()
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/models/environment.py", line 519, in <lambda>
    (self.d_env.admin_net), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 56, in _tcp_ping
    s.connect((str(host), int(port)))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

after connect to vm over vncviewer I see that there is no ssh service running, after it restart node became accessible over ssh

Looks like we do not start it at all due to errr in bootsrap_master.log:
http://paste.openstack.org/show/485382/

See original description
Tags: area-python
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
assignee: nobody → MOS Linux (mos-linux)
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

looks like caused https://github.com/openstack/fuel-menu/commit/bf7baeb2be4cbb5bfe7b2ac0ecc168b9793438ac

description: updated
Revision history for this message
Sergey Kulanov (skulanov) wrote :
Download full text (4.0 KiB)

yep, need to revert

[root@fuel ~]# fuelmenu

Traceback (most recent call last):
  File "/usr/bin/fuelmenu", line 9, in <module>
    load_entry_point('fuelmenu==9.0.0', 'console_scripts', 'fuelmenu')()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 458, in main
    setup()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 314, in setup
    FuelSetup()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 93, in __init__
    self.main()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 254, in main
    self.mainloop.run()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 271, in run
    self.screen.run_wrapper(self._run)
  File "/usr/lib64/python2.7/site-packages/urwid/raw_display.py", line 241, in run_wrapper
    return fn()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 336, in _run
    self.event_loop.run()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 707, in run
    self._loop()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 786, in _loop
    self._watch_files[fd]()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 387, in _update
    self.process_input(keys)
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 487, in process_input
    k = self._topmost_widget.keypress(self.screen_size, k)
  File "/usr/lib64/python2.7/site-packages/urwid/container.py", line 1102, in keypress
    return self.body.keypress( (maxcol, remaining), key )
  File "/usr/lib64/python2.7/site-packages/urwid/listbox.py", line 986, in keypress
    key = focus_widget.keypress((maxcol,),key)
  File "/usr/lib64/python2.7/site-packages/urwid/container.py", line 2240, in keypress
    key = w.keypress((mc,) + size[1:], key)
  File "/usr/lib64/python2.7/site-packages/urwid/container.py", line 1559, in keypress
    key = self.focus.keypress(tsize, key)
  File "/usr/lib64/python2.7/site-packages/urwid/decoration.py", line 384, in keypress
    return self._original_widget.keypress((maxcol, self.height), key)
  File "/usr/lib64/python2.7/site-packages/urwid/decoration.py", line 836, in keypress
    return self._original_widget.keypress((maxcol,maxrow-top-bottom), key)
  File "/usr/lib64/python2.7/site-packages/urwid/listbox.py", line 986, in keypress
    key = focus_widget.keypress((maxcol,),key)
  File "/usr/lib64/python2.7/site-packages/urwid/wimp.py", line 534, in keypress
    self._emit('click')
  File "/usr/lib64/python2.7/site-packages/urwid/widget.py", line 463, in _emit
    signals.emit_signal(self, name, self, *args)
  File "/usr/lib64/python2.7/site-packages/urwid/signals.py", line 120, in emit
    result |= bool(callback(*args_copy))
  File "/usr/lib/python2.7/site-packages/fuelmenu/modules/saveandquit.py", line 50, in save_and_quit
    if self.save():
  File "/usr/lib/python2.7/site-packages/fuelmenu/modules/saveandquit.py", line 56, in save
    results, modulename = self.parent.global_save()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 299, in global_save
    if module.apply(None):
  File "/usr/lib/python2.7/site-packages/fuelmenu/modules/fueluser....

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-menu (master)

Fix proposed to branch: master
Review: https://review.openstack.org/273976

Changed in fuel:
assignee: MOS Linux (mos-linux) → Sergey Kulanov (skulanov)
status: New → In Progress
Roman Vyalov (r0mikiam)
tags: added: area-python
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-menu (master)

Change abandoned by Sergey Kulanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/273976
Reason: fixed by https://review.openstack.org/#/c/273978/

Revision history for this message
Sergey Kulanov (skulanov) wrote :

fixed in https://review.openstack.org/#/c/273978/

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Alexandr Kostrikov (akostrikov-mirantis) wrote :

Seems, it is reproduced on ci[0]:
cat /var/log/fuelmenu.log
2016-03-15 21:12:24,482 DEBUG Executing command: docker ps -a
2016-03-15 21:12:24,532 DEBUG Command executed with exit code: 1
2016-03-15 21:12:24,536 DEBUG Executing command: dhcpcheck discover --timeout 5 -f json --ifaces enp0s3
2016-03-15 21:12:35,897 DEBUG Command executed with exit code: 0
2016-03-15 21:12:35,898 DEBUG No DHCP servers found
2016-03-15 21:12:35,898 DEBUG Executing command: arping -D -c3 -w1 -I enp0s3 -s 0.0.0.0 10.109.5.2
2016-03-15 21:12:37,916 DEBUG Command executed with exit code: 0
2016-03-15 21:12:37,944 ERROR Unable to read YAML: /etc/fuel/astute.yaml

I assume that by lines in third comment: "IOError: [Errno 2] No such file or directory: '/etc/fuel/astute.yaml'"

[0] https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_neutron_public/47/console

Revision history for this message
Alexandr Kostrikov (akostrikov-mirantis) wrote :

It has not been reproduced. Setting to 'fix released'

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Vadim Rovachev (vrovachev) wrote :

Reproduced on 8.0.
https://patching-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.fuel_master_migrate/15/console

Revision history for this message
Sergey Kulanov (skulanov) wrote :

@Vadim,

I'd rather vote for creating another bug with "migration flow" cause this bug should not be reproduced during provisioning/deployment master node

Also could you please also add logs from the reverted environment, why do you think it's related to fuel-menu?

Revision history for this message
Vadim Rovachev (vrovachev) wrote :

@Sergey,

sorry for my wrong 0 comment.

In test result we see that test fails on check migrate.
https://patching-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.fuel_master_migrate/30/testReport/(root)/Case_FuelMasterMigrate__Config_1ctrl_1comp_neutronTUN/

Log from master node:
[root@nailgun ~]# cat /var/log/fuelmenu.log
2016-07-18 11:58:12,222 DEBUG No DHCP servers found
2016-07-18 11:58:14,246 ERROR Unable to read YAML: /etc/fuel/astute.yaml
2016-07-18 11:58:14,263 ERROR Unable to read YAML: /etc/fuel/astute.yaml

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.