Juniper Openstack

Storage provisioning is failing in 2.20 build 57 Juno

  1. Series trunk
  2. Bug #1466958
Bug #1466958 reported by Vinod Nair
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
Critical
Sanju Abraham
Juniper Openstack r2.20-fcs
Trunk
Fix Committed
Critical
Sanju Abraham
Juniper Openstack r3.0-fcs

Bug Description

On a 2.20 Build 57 Juno based HA setup , storage provisioning (setup_storage) is failing
Can't connect to MySQL server on '127.0.0.1'

do openstack-config --set /etc/cinder/cinder.conf DEFAULT rabbit_port 5673
2015-06-19 10:41:43:420271: [root@10.87.141.1] out: [13.1.0.1] out: [localhost] local: sudo cinder-manage db sync
2015-06-19 10:41:43:484215: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 CRITICAL cinder [-] OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '127.0.0.1' (111)") None None
2015-06-19 10:41:44:048577: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder Traceback (most recent call last):
2015-06-19 10:41:44:048853: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/bin/cinder-manage", line 552, in <module>
2015-06-19 10:41:44:048987: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder main()
2015-06-19 10:41:44:049108: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/bin/cinder-manage", line 549, in main
2015-06-19 10:41:44:049225: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder fn(*fn_args)
2015-06-19 10:41:44:049341: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/bin/cinder-manage", line 229, in sync
2015-06-19 10:41:44:049456: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return migration.db_sync(version)
2015-06-19 10:41:44:049574: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/migration.py", line 28, in db_sync
2015-06-19 10:41:44:049695: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return IMPL.db_sync(version=version)
2015-06-19 10:41:44:049814: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.py", line 40, in db_sync
2015-06-19 10:41:44:049934: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder current_version = db_version()
2015-06-19 10:41:44:050054: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.py", line 52, in db_version
2015-06-19 10:41:44:050208: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return versioning_api.db_version(get_engine(), repository)
2015-06-19 10:41:44:050330: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 80, in get_engine
2015-06-19 10:41:44:050455: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder facade = _create_facade_lazily()
2015-06-19 10:41:44:050668: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 67, in _create_facade_lazily
2015-06-19 10:41:44:050736: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder **dict(CONF.database.iteritems())
2015-06-19 10:41:44:050795: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 711, in __init__
2015-06-19 10:41:44:050852: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder **engine_kwargs)
2015-06-19 10:41:44:050910: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 386, in create_engine
2015-06-19 10:41:44:050966: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder connection_trace=connection_trace
2015-06-19 10:41:44:051023: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 890, in __call__
2015-06-19 10:41:44:051079: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder self._url_from_target(target), target, arg, kw)
2015-06-19 10:41:44:051136: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 927, in _dispatch_on
2015-06-19 10:41:44:051192: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return self._dispatch_on_db_driver(dbname, driver, arg, kw)
2015-06-19 10:41:44:051249: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 981, in _dispatch_on_db_driver
2015-06-19 10:41:44:051305: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder if self._invoke_fn(fn, arg, kw) is not None:
2015-06-19 10:41:44:051361: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 930, in _invoke_fn
2015-06-19 10:41:44:051418: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return fn(*arg, **kw)
2015-06-19 10:41:44:051476: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 456, in _init_events
2015-06-19 10:41:44:051533: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder realmode = engine.execute("SHOW VARIABLES LIKE 'sql_mode'").fetchone()
2015-06-19 10:41:44:051591: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1751, in execute
2015-06-19 10:41:44:052917: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder connection = self.contextual_connect(close_with_result=True)
2015-06-19 10:41:44:052988: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1799, in contextual_connect
2015-06-19 10:41:44:053056: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder self.pool.connect(),
2015-06-19 10:41:44:053114: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 338, in connect
2015-06-19 10:41:44:053171: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return _ConnectionFairy._checkout(self)
2015-06-19 10:41:44:053229: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 641, in _checkout
2015-06-19 10:41:44:053286: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder fairy = _ConnectionRecord.checkout(pool)
2015-06-19 10:41:44:053431: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 440, in checkout
2015-06-19 10:41:44:053649: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder rec = pool._do_get()
2015-06-19 10:41:44:053776: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 961, in _do_get
2015-06-19 10:41:44:053894: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return self._create_connection()
2015-06-19 10:41:44:054013: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 285, in _create_connection
2015-06-19 10:41:44:054130: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return _ConnectionRecord(self)
2015-06-19 10:41:44:054247: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 411, in __init__
2015-06-19 10:41:44:054363: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder self.connection = self.__connect()
2015-06-19 10:41:44:054481: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 537, in __connect
2015-06-19 10:41:44:054596: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder connection = self.__pool._creator()
2015-06-19 10:41:44:054712: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 96, in connect
2015-06-19 10:41:44:054827: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder connection_invalidated=invalidated
2015-06-19 10:41:44:054944: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-06-19 10:41:44:055059: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder reraise(type(exception), exception, tb=exc_tb)
2015-06-19 10:41:44:055176: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 90, in connect
2015-06-19 10:41:44:055294: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return dialect.connect(*cargs, **cparams)
2015-06-19 10:41:44:055412: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 377, in connect
2015-06-19 10:41:44:055527: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return self.dbapi.connect(*cargs, **cparams)
2015-06-19 10:41:44:055660: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
2015-06-19 10:41:44:055782: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder return Connection(*args, **kwargs)
2015-06-19 10:41:44:055898: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
2015-06-19 10:41:44:056038: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder super(Connection, self).__init__(*args, **kwargs2)
2015-06-19 10:41:44:056162: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '127.0.0.1' (111)") None None
2015-06-19 10:41:44:056277: [root@10.87.141.1] out: [13.1.0.1] out: 2015-06-19 10:41:44.016 22718 TRACE cinder
2015-06-19 10:41:44:056392: [root@10.87.141.1] out: [13.1.0.1] out:
2015-06-19 10:41:44:090986: [root@10.87.141.1] out: [13.1.0.1] out: Fatal error: local() encountered an error (return code 1) while executing 'sudo cinder-manage db sync'
2015-06-19 10:41:44:091092: [root@10.87.141.1] out: [13.1.0.1] out:
2015-06-19 10:41:44:091153: [root@10.87.141.1] out: [13.1.0.1] out: Aborting.
2015-06-19 10:41:44:091208: [root@10.87.141.1] out: [13.1.0.1] out:
2015-06-19 10:41:44:492211: [root@10.87.141.1] out:
2015-06-19 10:41:44:508151: [root@10.87.141.1] out:
2015-06-19 10:41:44:508409: [root@10.87.141.1] out: Fatal error: run() received nonzero return code 1 while executing!
2015-06-19 10:41:44:508565: [root@10.87.141.1] out:
2015-06-19 10:41:44:508724: [root@10.87.141.1] out: Requested: sudo storage-fs-setup --storage-master 13.1.0.1 --storage-setup-mode setup --storage-hostnames cs-scale-1 cs-scale-2 cs-scale-3 cs-scale-4 cs-scale-5 cs-scale-6 cs-scale-7 cs-scale-8 --storage-compute-hostnames cs-scale-4 cs-scale-5 cs-scale-6 cs-scale-7 cs-scale-8 --storage-hosts 13.1.0.1 13.1.0.2 13.1.0.3 13.1.0.4 13.1.0.5 13.1.0.6 13.1.0.7 13.1.0.8 --storage-host-tokens n1keenA n1keenA n1keenA n1keenA n1keenA n1keenA n1keenA n1keenA --storage-disk-config cs-scale-4:/dev/sdb:/dev/sdf cs-scale-4:/dev/sdc:/dev/sdf cs-scale-4:/dev/sdd:/dev/sdf cs-scale-5:/dev/sdb:/dev/sdf cs-scale-5:/dev/sdc:/dev/sdf cs-scale-5:/dev/sdd:/dev/sdf cs-scale-6:/dev/sdb:/dev/sdf cs-scale-6:/dev/sdc:/dev/sdf cs-scale-6:/dev/sdd:/dev/sdf cs-scale-7:/dev/sdb:/dev/sdf cs-scale-7:/dev/sdc:/dev/sdf cs-scale-7:/dev/sdd:/dev/sdf cs-scale-8:/dev/sdb:/dev/sdf cs-scale-8:/dev/sdc:/dev/sdf cs-scale-8:/dev/sdd:/dev/sdf --storage-ssd-disk-config cs-scale-4:/dev/sde cs-scale-5:/dev/sde cs-scale-6:/dev/sde cs-scale-7:/dev/sde cs-scale-8:/dev/sde --storage-journal-config none --storage-local-disk-config none --storage-local-ssd-disk-config none --storage-nfs-disk-config none --storage-directory-config none --storage-chassis-config cs-scale-4:Chassis1 cs-scale-5:Chassis2 cs-scale-6:Chassis3 cs-scale-7:Chassis4 cs-scale-8:Chassis5 --storage-mon-hosts none --collector-hosts 13.1.0.1 13.1.0.2 13.1.0.3 --collector-host-tokens n1keenA n1keenA n1keenA --cfg-host 13.1.0.1 --cinder-vip 13.1.0.10 --config-hosts 13.1.0.2 13.1.0.3 --storage-os-hosts 13.1.0.2 13.1.0.3 --storage-os-host-tokens n1keenA n1keenA --cfg-vip none --storage-replica-size None
2015-06-19 10:41:44:508899: [root@10.87.141.1] out: Executed: /bin/bash -l -c "sudo storage-fs-setup --storage-master 13.1.0.1 --storage-setup-mode setup --storage-hostnames cs-scale-1 cs-scale-2 cs-scale-3 cs-scale-4 cs-scale-5 cs-scale-6 cs-scale-7 cs-scale-8 --storage-compute-hostnames cs-scale-4 cs-scale-5 cs-scale-6 cs-scale-7 cs-scale-8 --storage-hosts 13.1.0.1 13.1.0.2 13.1.0.3 13.1.0.4 13.1.0.5 13.1.0.6 13.1.0.7 13.1.0.8 --storage-host-tokens n1keenA n1keenA n1keenA n1keenA n1keenA n1keenA n1keenA n1keenA --storage-disk-config cs-scale-4:/dev/sdb:/dev/sdf cs-scale-4:/dev/sdc:/dev/sdf cs-scale-4:/dev/sdd:/dev/sdf cs-scale-5:/dev/sdb:/dev/sdf cs-scale-5:/dev/sdc:/dev/sdf cs-scale-5:/dev/sdd:/dev/sdf cs-scale-6:/dev/sdb:/dev/sdf cs-scale-6:/dev/sdc:/dev/sdf cs-scale-6:/dev/sdd:/dev/sdf cs-scale-7:/dev/sdb:/dev/sdf cs-scale-7:/dev/sdc:/dev/sdf cs-scale-7:/dev/sdd:/dev/sdf cs-scale-8:/dev/sdb:/dev/sdf cs-scale-8:/dev/sdc:/dev/sdf cs-scale-8:/dev/sdd:/dev/sdf --storage-ssd-disk-config cs-scale-4:/dev/sde cs-scale-5:/dev/sde cs-scale-6:/dev/sde cs-scale-7:/dev/sde cs-scale-8:/dev/sde --storage-journal-config none --storage-local-disk-config none --storage-local-ssd-disk-config none --storage-nfs-disk-config none --storage-directory-config none --storage-chassis-config cs-scale-4:Chassis1 cs-scale-5:Chassis2 cs-scale-6:Chassis3 cs-scale-7:Chassis4 cs-scale-8:Chassis5 --storage-mon-hosts none --collector-hosts 13.1.0.1 13.1.0.2 13.1.0.3 --collector-host-tokens n1keenA n1keenA n1keenA --cfg-host 13.1.0.1 --cinder-vip 13.1.0.10 --config-hosts 13.1.0.2 13.1.0.3 --storage-os-hosts 13.1.0.2 13.1.0.3 --storage-os-host-tokens n1keenA n1keenA --cfg-vip none --storage-replica-size None"
2015-06-19 10:41:44:509189: [root@10.87.141.1] out:
2015-06-19 10:41:44:509486: [root@10.87.141.1] out: Aborting.
2015-06-19 10:41:44:509638: [root@10.87.141.1] out:
2015-06-19 10:41:44:610824:

Tags: storage
Revision history for this message
Sanju Abraham (asanju) wrote :

Request submitter to lower the important from Critical to Medium. Retry was not done to mark this as a Critical issue.

Retry should have been successful.

Commits - (https://review.opencontrail.org/#/c/11827/) was applied on the same setup and storage provisioning completed without any issue.

There are other enhancement suggested for cinder provisioning on a HA setup, which is to connect only on the VIP address for provisioning DB.

Submitter to add more details and closing comments

Revision history for this message
Vinod Nair (vinodnair) wrote :

Tried with the Sanju's patch and storage provisioning went through fine in 2.20 Build 59 Juno .. Did not see any issues with mysql going down

OpenContrail Admin (ci-admin-f)
information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.