tripleo

FAILED - RETRYING: waiting for the monitor(s) to form the quorum

Bug #1858123 reported by wes hayutin on 2020-01-02

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Giulio Fidente	tripleo ussuri-1

Bug Description

http://logs.rdoproject.org/42/23642/7/check/tripleo-ceph-integration-rhel-8-scenario001-standalone/306007c/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

ExecStartPre: ''{ path=/bin/sh ; argv[]=/bin/sh -c "$(command -v mkdir)" -p /etc/ceph /var/lib/ceph/mon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }'''
2020-01-02 17:37:57 | - ' ExecStop: ''{ path=/usr/bin/podman ; argv[]=/usr/bin/podman stop ceph-mon-standalone ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }'''
2020-01-02 17:37:57 | - ' ExecStopPost: ''{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/ceph/ceph-mon.standalone.asok ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }'''
2020-01-02 17:37:57 | - ' FragmentPath: /etc/systemd/system/ceph-mon@.service'
2020-01-02 17:37:57 | - ' Id: <email address hidden>'
2020-01-02 17:37:57 | - ' Names: <email address hidden>'
2020-01-02 17:37:57 | - ' PrivateTmp: ''no'''
2020-01-02 17:37:57 | - ' ProtectHome: ''no'''
2020-01-02 17:37:57 | - ' ProtectSystem: ''no'''
2020-01-02 17:37:57 | - ' Requires: sysinit.target system-ceph\x2dmon.slice'
2020-01-02 17:37:57 | - ' Restart: always'
2020-01-02 17:37:57 | - ' RestartUSec: 10s'
2020-01-02 17:37:57 | - ' Slice: system-ceph\x2dmon.slice'
2020-01-02 17:37:57 | - ' TimeoutStartUSec: 2min'
2020-01-02 17:37:57 | - ' TimeoutStopUSec: 15s'
2020-01-02 17:37:57 | - ' Type: simple'
2020-01-02 17:37:57 | - ' UnitFilePreset: disabled'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjYwMTMwMmM5LWJjODItNGY1Yy1hMWU1LTBmMzlkNzk1ZmNmMSJ9\e[64D\e[K"
2020-01-02 17:37:57 | - 'TASK [ceph-mon : include_tasks ceph_keys.yml] **********************************'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjg1ZTdkYTY2LTliN2EtNDY4OC1hODY2LTBmYjkzZmNiMmMxYyJ9\e[64D\e[KThursday 02 January 2020 17:36:09 +0000 (0:00:00.773) 0:01:17.733 ****** "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjJkYzIzZWIyLTUyYjgtNDYyYi04NzFmLTg3ZDZiNjAyYjUwNSJ9\e[64D\e[Kincluded: /usr/share/ceph-ansible/roles/ceph-mon/tasks/ceph_keys.yml for standalone"
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjJjNTI1MWZkLTA4NzgtNDM1YS05NjY5LTZhYTFjNTM2MDY1MiJ9\e[64D\e[K"
2020-01-02 17:37:57 | - 'TASK [ceph-mon : waiting for the monitor(s) to form the quorum...] *************'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImM2M2I0M2M4LTQ2N2EtNDJjYi1iNjI5LTVlMzU2M2Q2ZmM0MyJ9\e[64D\e[KThursday 02 January 2020 17:36:09 +0000 (0:00:00.172) 0:01:17.905 ****** "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImZlN2I1MDA3LTNhMGEtNGQ4Yi1hZTA4LTgwN2FkNTc1MjJmNCJ9\e[64D\e[KFAILED - RETRYING: waiting for the monitor(s) to form the quorum... (10 retries left)."
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjBiNDFhNjY1LTY1MTYtNDMzZC1hZDc1LTI0MjJjMzBkYTk4NCJ9\e[64D\e[Kok: [standalone] => changed=false "
2020-01-02 17:37:57 | - ' attempts: 2'
2020-01-02 17:37:57 | - ' - daemon'
2020-01-02 17:37:57 | - ' - mon.standalone'
2020-01-02 17:37:57 | - ' - mon_status'
2020-01-02 17:37:57 | - ' - --format'
2020-01-02 17:37:57 | - ' delta: ''0:00:00.381976'''
2020-01-02 17:37:57 | - ' end: ''2020-01-02 17:36:30.498019'''
2020-01-02 17:37:57 | - ' start: ''2020-01-02 17:36:30.116043'''
2020-01-02 17:37:57 | - ' stdout: ''{"name":"standalone","rank":0,"state":"leader","election_epoch":3,"quorum":[0],"quorum_age":13,"features":{"required_con":"2449958747315912708","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus"],"quorum_con":"4611087854031667199","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus"]},"outside_quorum":[],"extra_probe_peers":[{"addrvec":[{"type":"v2","addr":"192.168.24.1:3300","nonce":0}]},{"addrvec":[{"type":"v2","addr":"192.168.24.1:3300","nonce":0},{"type":"v1","addr":"192.168.24.1:6789","nonce":0}]}],"sync_provider":[],"monmap":{"epoch":1,"fsid":"4b5c8c0a-ff60-454b-a1b4-9747aa737d19","modified":"2020-01-02 17:36:07.036897","created":"2020-01-02 17:36:07.036897","min_mon_release":14,"min_mon_release_name":"nautilus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus"],"optional":[]},"mons":[{"rank":0,"name":"standalone","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.24.1:3300","nonce":0},{"type":"v1","addr":"192.168.24.1:6789","nonce":0}]},"addr":"192.168.24.1:6789/0","public_addr":"192.168.24.1:6789/0"}]},"feature_map":{"mon":[{"features":"0x3ffddff8ffacffff","release":"luminous","num":1}]}}'''
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjlkY2RjNjUxLTMyYjItNGUwZi04ZGM2LTI1NzQ2ZmVmYzFkYyJ9\e[64D\e[K"
2020-01-02 17:37:57 | - 'TASK [ceph-mon : fetch ceph initial keys] **************************************'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImQ4ZWEzNTM3LTIzNTgtNGE4Ny1iYTRhLWNmNDY4NWMwNDU5YiJ9\e[64D\e[KThursday 02 January 2020 17:36:30 +0000 (0:00:21.277) 0:01:39.182 ****** "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjZlZDdhMzg0LWZiMTUtNGQ5NC04OTRjLWM1NGI4MzRlODhmZCJ9\e[64D\e[Kchanged: [standalone] => changed=true "
2020-01-02 17:37:57 | - ' - --entrypoint=ceph'
2020-01-02 17:37:57 | - ' - -n'
2020-01-02 17:37:57 | - ' - -k'
2020-01-02 17:37:57 | - ' - /var/lib/ceph/mon/ceph-standalone/keyring'
2020-01-02 17:37:57 | - ' - auth'
2020-01-02 17:37:57 | - ' - get'
2020-01-02 17:37:57 | - ' - client.bootstrap-rgw'
2020-01-02 17:37:57 | - ' - plain'
2020-01-02 17:37:57 | - ' - -o'
2020-01-02 17:37:57 | - ' - /var/lib/ceph/bootstrap-rgw/ceph.keyring'
2020-01-02 17:37:57 | - ' delta: ''0:00:06.711774'''
2020-01-02 17:37:57 | - ' end: ''2020-01-02 17:36:37.489395'''
2020-01-02 17:37:57 | - ' start: ''2020-01-02 17:36:30.777621'''
2020-01-02 17:37:57 | - ' stderr: exported keyring for client.bootstrap-rgw'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjM5YTA3NTEyLWVmYzEtNDUzZC05ZGI5LTQ5MTI1YTI2ODU4ZSJ9\e[64D\e[K"
2020-01-02 17:37:57 | - 'TASK [ceph-mon : include secure_cluster.yml] ***********************************'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImZiNWQ3NTYxLTA5ZGYtNGI2Yi1hODBmLTgyZjc3YmVkOWMzMCJ9\e[64D\e[KThursday 02 January 2020 17:36:37 +0000 (0:00:06.995) 0:01:46.178 ****** "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjk0NDYyYjMzLTc3ZGUtNDRlYS05MWUzLTVhMzUwOGRhNTkwOSJ9\e[64D\e[Kskipping: [standalone] => changed=false "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImUyZjhiMGFiLTI3NjItNDIwZS04ZmZhLTRhMTEyMzdhZjA0YSJ9\e[64D\e[K"

Found the same error in an old upgrade bug FYI
https://bugs.launchpad.net/tripleo/+bug/1832597

Tags:

wes hayutin (weshayutin) on 2020-01-09

tags:	added: depcheck removed: alert
tags:	added: promotion-blocker

Revision history for this message

Giulio Fidente (gfidente) wrote on 2020-01-10:

from the logs it looks like we're installing a pretty old version of ceph-ansible

ceph-ansible-4.0.0-0.1.rc16.1.el8.noarch

in cbs, for centos7, we have 4.0.6 final already and that's the one we should be using

from the undercloud config it looks like we're using a temporary location [1] to override the ceph-ansible location

I'll help refreshing that build to 4.0.6 then we can try this again

http://logs.rdoproject.org/42/23642/7/check/tripleo-ceph-integration-rhel-8-scenario001-standalone/306007c/logs/undercloud/etc/yum.repos.d/ceph-ansible-override.repo.txt.gz

Revision history for this message

Giulio Fidente (gfidente) wrote on 2020-01-10:

there is ceph-ansible from master in http://logs.rdoproject.org/42/23642/7/check/tripleo-ceph-integration-master/036b3a0/buildset/ but it doesn't seem to be getting installed

OpenStack Infra (hudson-openstack) on 2020-01-10

Changed in tripleo:
assignee:	nobody → Giulio Fidente (gfidente)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-01-10: Fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/701960
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=0df6618148cf19689e191053d2a0a75aaf15144f
Submitter: Zuul
Branch: master

commit 0df6618148cf19689e191053d2a0a75aaf15144f
Author: Giulio Fidente <email address hidden>
Date: Fri Jan 10 14:18:56 2020 +0100

Revert "Use custom ceph-ansible until c8 storage sig ready"

We can build RPMs on the fly now, from the stable-4 branch,
see https://review.rdoproject.org/r/#/c/24461/

This reverts commit d3d57757ae02a709aa2eac935d964b9d602e9128.

Closes-Bug: 1858123
Change-Id: I69d4d82c3874b2a66bcd8c666b5d5d85f4f0dd3c

Changed in tripleo:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.