Deployment process of openstack on the second controller of nsx's machine always fails due to different reasons.

Bug #1368728 reported by Tatyana Dubyk
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Stepan Rogov
5.1.x
Fix Committed
Critical
Stepan Rogov
6.0.x
Fix Released
Critical
Andrey Danin

Bug Description

I've checked this bug on different configurations of env in ha mode.
After each fallen deploy I've reseted snapshot on nsx machine and cleared previous envs.

==============nsx settings===========================
Fuel UI-> Settings -> VMware NSX

NSX_USERNAME='admin'
NSX_PASSWORD= 'admin'
NSX_TRANSPORT_ZONE='0f12c1f7-5605-46b6-aa3f-4bf50bde47e5'
L3_SERVICE_UUID='aa153851-bd8e-4852-9772-d98a431849e0'
NSX_CONTROLLER_ENDPOINT='172.16.1.253'
URL_TO_NSX_BITS='http://10.20.204.1'

NSX network -> management -> untag and set 172.16.1.0/24 on eth2
=====================================================
Configuration 1:
===================================================
steps to reproduce:
1.set up lab on nsx machine
2.create env and start deploy:
   OS: Ubuntu (HA mode)
   hypervisor: QEMU
   roles: 3 controllers, 1 compute, 1 cinder

3. check that deployment of openstack on 1st controller is finished successfully
4. check that deployment of openstack on 2nd controller is failed

Expected result: Deployment process of openstack on each of nodes will be finished successfully
Actual result: deployment of openstack on 2nd controller is failed due to reason: Could not find init script or upstart conf file for 'glance-registry'

------------Logs------------------------------------

root@node-12:~# less /var/log/puppet.log

Thu Sep 11 19:45:49 +0000 2014 /Stage[main]/Glance::Registry/Service[glance-registry] (notice): Dependency Service[mysql-service] has failure
s: true
Thu Sep 11 19:45:49 +0000 2014 /Stage[main]/Glance::Registry/Service[glance-registry] (warning): Skipping because of failed dependencies
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry.conf in /etc/init
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry.conf in /etc/init.d
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry in /etc/init
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry in /etc/init.d
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry.sh in /etc/init
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry.sh in /etc/init.d
Thu Sep 11 19:45:49 +0000 2014 /Stage[main]/Glance::Registry/Service[glance-registry] (err): Failed to call refresh: Could not find init script or upstart conf file for 'glance-registry'
Thu Sep 11 19:45:49 +0000 2014 /Stage[main]/Glance::Registry/Service[glance-registry] (err): Could not find init script or upstart conf file for 'glance-registry'

Glance package is not installed due to some reason.

root@node-12:~# aptitude show glance-registry
Package: glance-registry
State: not installed
Version: 1:2014.1.1-fuel5.1~mira10
Priority: extra
Section: python
Maintainer: Ubuntu OpenStack <email address hidden>
Architecture: all
Uncompressed Size: 21.5 k
Depends: glance-common (= 1:2014.1.1-fuel5.1~mira10), upstart-job, python
Recommends: python-glanceclient
Replaces: glance (< 2012.1~e1~20110919.1021-0ubuntu2), glance (< 2012.1~e1~20110919.1021-0ubuntu2)
Description: OpenStack Image Registry and Delivery Service - Registry
 The Glance project provides an image registration, discovery and delivery service. These services may be used as stand-along services, and
 they may also be used by Nova to deliver images from object stores, such as OpenStack's Swift service, to Nova's compute nodes.

 This package contains the glance registry server.
Homepage: http://launchpad.net/glance

=================================================
Configuration 2:
=================================================
steps to reproduce:
1.set up lab on nsx machine
2.create env and start deploy:
   OS: Ubuntu (HA mode)
   hypervisor: QEMU
   roles: 2 controllers, 1 compute

3. check that deployment of openstack on 1st controller is finished successfully
4. check that deployment of openstack on 2nd controller is failed

Expected result: Deployment process of openstack on each of nodes will be finished successfully
Actual result:deployment of openstack on 2nd controller is failed due to reason: MySQL is not running

-----------------Logs------------------------------------
Trace from HC logs on node-7 (controller) (/Stage[main]/Galera/Service[mysql-service]/ensure) change from stopped to running failed: execution expired

Trace from node-7 : less /var/log less mysqld.log
<30>Sep 11 14:25:17 node-7 mysql-wss[2680]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 0 retries left
<30>Sep 11 14:25:22 node-7 mysql-wss[2680]: INFO: MySQL is not running
<27>Sep 11 14:25:22 node-7 mysql-wss[2680]: ERROR: GTID have wrong format: :
<27>Sep 11 14:25:22 node-7 mysql-wss[2680]: ERROR: Wrong GTID, not updating gtid attribute
<30>Sep 11 14:25:22 node-7 mysql-wss[2680]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 2 retries left
<30>Sep 11 14:25:27 node-7 mysql-wss[2680]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 1 retries left
<30>Sep 11 14:25:32 node-7 mysql-wss[2680]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 0 retries left
<30>Sep 11 14:25:37 node-7 mysql-wss[2680]: INFO: MySQL is not running
<30>Sep 11 14:25:37 node-7 mysql-wss[2913]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 0 retries left
<30>Sep 11 14:25:42 node-7 mysql-wss[2913]: INFO: MySQL is not running
<27>Sep 11 14:25:42 node-7 mysql-wss[2913]: ERROR: Directory /var/run/mysqld is not writable by mysql
<30>Sep 11 14:25:42 node-7 mysql-wss[2976]: INFO: MySQL is not running

no one service not available on node-7

and after manually restart and stop of mysql service too.

Tags: nsx
Revision history for this message
Tatyana Dubyk (tdubyk) wrote :
Tatyana Dubyk (tdubyk)
description: updated
Changed in fuel:
importance: Undecided → High
Revision history for this message
Tatyana Dubyk (tdubyk) wrote :

api: '1.0'
astute_sha: b622d9b36dbdd1e03b282b9ee5b7435ba649e711
auth_required: true
build_id: 2014-09-10_00-01-12
build_number: '506'
feature_groups:
- mirantis
fuellib_sha: e3f947cc4142210499a282bc5f183c333552fa23
fuelmain_sha: ca1b4839a70a10041f8eaf8b9ac995c8b0d4521a
nailgun_sha: 82091e0d61f252619a0842d0f8debb6b602a61fe
ostf_sha: 1de6ed1c0b72f6687ffb4bebc2c939b135a88e34
production: docker
release: '5.1'
release_versions:
  2014.1.1-5.1:
    VERSION:
      api: '1.0'
      astute_sha: b622d9b36dbdd1e03b282b9ee5b7435ba649e711
      build_id: 2014-09-10_00-01-12
      build_number: '506'
      feature_groups:
      - mirantis
      fuellib_sha: e3f947cc4142210499a282bc5f183c333552fa23
      fuelmain_sha: ca1b4839a70a10041f8eaf8b9ac995c8b0d4521a
      nailgun_sha: 82091e0d61f252619a0842d0f8debb6b602a61fe
      ostf_sha: 1de6ed1c0b72f6687ffb4bebc2c939b135a88e34
      production: docker
      release: '5.1'

Revision history for this message
Tatyana Dubyk (tdubyk) wrote :

nsx connector type: GRE

Changed in fuel:
assignee: nobody → Fuel Partner Integration Team (fuel-partner)
Stepan Rogov (srogov)
Changed in fuel:
assignee: Fuel Partner Integration Team (fuel-partner) → Stepan Rogov (srogov)
Stepan Rogov (srogov)
Changed in fuel:
status: New → Confirmed
Revision history for this message
Stepan Rogov (srogov) wrote :

The bug occurs because corosync can not run mysql.
Broke check resources before starting ("insufficient privileges").
This is because this test is carried out from the shell ("/bin/sh"). In ubuntu is a symlink to dash.
But in the file /etc/profile.d/nsx-alias.sh from the package nicira-ovs-hypervisor-node uses bash-syntax, though there specified interpreter /bin/sh.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/121915

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/121997

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/121915
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5cd540218f53b933d8d5dd3c1aa347d60876f12d
Submitter: Jenkins
Branch: master

commit 5cd540218f53b933d8d5dd3c1aa347d60876f12d
Author: Stepan Rogov <email address hidden>
Date: Tue Sep 16 20:00:30 2014 +0400

    Added file /etc/profile.d/nsx-alias.sh deletion in Ubuntu HA + NSX deploy

    Delete this file, because it uses bash-syntax for functions defenitions,
    but specified interpreter is /bin/sh and in Ubuntu /bin/sh->/bin/dash
    When corosync checks it resources, it uses /bin/sh which is equivalent
    /bin/dash in Ubuntu.
    Accordingly, corosync can not launch mysql.

    Change-Id: I950ae6ad40c2c43c410cc0af98ff3b08b52a0260
    Closes-bug: #1368728

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/121997
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2dac97bdb43629167e1774e2c7cc441a3146ef52
Submitter: Jenkins
Branch: stable/5.1

commit 2dac97bdb43629167e1774e2c7cc441a3146ef52
Author: Stepan Rogov <email address hidden>
Date: Tue Sep 16 20:00:30 2014 +0400

    Added file /etc/profile.d/nsx-alias.sh deletion in Ubuntu HA + NSX deploy

    Delete this file, because it uses bash-syntax for functions defenitions,
    but specified interpreter is /bin/sh and in Ubuntu /bin/sh->/bin/dash
    When corosync checks it resources, it uses /bin/sh which is equivalent
    /bin/dash in Ubuntu.
    Accordingly, corosync can not launch mysql.

    Change-Id: I950ae6ad40c2c43c410cc0af98ff3b08b52a0260
    Closes-bug: #1368728

Revision history for this message
okosse (okosse) wrote :

I checked it on fuel-6.0-56-2014-12-18_01-32-01.iso

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.