failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Triaged
|
Low
|
Unassigned |
Bug Description
juju 2.8.9 bootstrap failed:
2021-02-24-14:05:15 root ERROR [localhost] Command failed: juju bootstrap --bootstrap-
2021-02-24-14:05:15 root ERROR [localhost] STDOUT follows:
Creating Juju controller "foundations-maas" on maas_cloud/default
Looking for packaged Juju agent version 2.8.9 for amd64
Launching controller instance(s) on maas_cloud/
- hcyhbs (arch=amd64 mem=4G cores=2)
Installing Juju agent on bootstrap instance
Fetching Juju Dashboard 0.3.0
Waiting for address
Attempting to connect to 10.244.40.100:22
Connected to 10.244.40.100
Running machine configuration script...
Cloud-init v. 20.4.1-
2021-02-24 13:57:30,187 - handlers.
2021-02-24 13:57:30,189 - handlers.
2021-02-24 13:57:30,507 - handlers.
Cloud-init v. 20.4.1-
ci-info: +++++++
ci-info: +------
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +------
ci-info: | ens4 | True | 10.244.40.100 | 255.255.248.0 | global | 52:54:00:c7:31:b1 |
ci-info: | ens4 | True | fe80::5054:
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +------
ci-info: +++++++
ci-info: +------
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +------
ci-info: | 0 | 0.0.0.0 | 10.244.40.1 | 0.0.0.0 | ens4 | UG |
ci-info: | 1 | 10.244.40.0 | 0.0.0.0 | 255.255.248.0 | ens4 | U |
ci-info: +------
ci-info: +++++++
ci-info: +------
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +------
ci-info: | 1 | fe80::/64 | :: | ens4 | U |
ci-info: | 3 | local | :: | ens4 | U |
ci-info: | 4 | ff00::/8 | :: | ens4 | U |
ci-info: +------
Cloud-init v. 20.4.1-
Hit:1 http://
Hit:2 http://
Hit:3 http://
Hit:4 http://
Get:5 http://
Get:6 http://
Get:7 http://
Get:8 http://
Get:9 http://
Get:10 http://
Get:11 http://
Get:12 http://
Get:13 http://
Get:14 http://
Get:15 http://
Get:16 http://
Get:17 http://
Fetched 6336 kB in 5s (1246 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
ifupdown
The following NEW packages will be installed:
bridge-utils
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 30.5 kB of archives.
After this operation, 112 kB of additional disk space will be used.
Get:1 http://
Fetched 30.5 kB in 0s (72.0 kB/s)
Selecting previously unselected package bridge-utils.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71214 files and directories currently installed.)
Preparing to unpack .../bridge-
Unpacking bridge-utils (1.6-2ubuntu1) ...
Setting up bridge-utils (1.6-2ubuntu1) ...
Processing triggers for man-db (2.9.1-1) ...
+ mkdir -p /var/lib/juju
+ cat
+ chmod 0755 /var/lib/
+ set -xe
+ install -D -m 644 /dev/null /var/lib/
+ printf %s\n user-admin:
Cloud-init v. 20.4.1-
Cloud-init v. 20.4.1-
Logging to /var/log/
Regenerating SSH RSA host key
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/
Your public key has been saved in /etc/ssh/
The key fingerprint is:
SHA256:
The key's randomart image is:
+---[RSA 3072]----+
| ..... o. .+oo |
|... . . ..+oo |
| o. . . +E+. |
|. .. o . o =. |
| o + + S . + . |
|. + + o . .o.o|
| = . .+.=|
| . =.+o=|
| + o.o+|
+----[SHA256]-----+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/
Your public key has been saved in /etc/ssh/
The key fingerprint is:
SHA256:
The key's randomart image is:
+---[DSA 1024]----+
|. o.. .o===o.+++|
|.o . + +....+o*++|
|o.. + . * .o + .o|
|o+ + o . Bo .|
|..+ . o S. .|
|. . .|
| . . |
| E .|
| .. |
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/
Your public key has been saved in /etc/ssh/
The key fingerprint is:
SHA256:
The key's randomart image is:
+---[ECDSA 256]---+
| . +.++o+ |
| . . ..E.o=.o |
| + . +=+B. |
|o.= . +++.. |
|o+.+. o S.. |
|..Bo.+ |
|.=.*+ |
|o.+o |
| .o |
+----[SHA256]-----+
Running apt-get update
Hit:1 http://
Hit:2 http://
Hit:3 http://
Hit:4 http://
Reading package lists...
Running apt-get upgrade
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
bind9-dnsutils bind9-host bind9-libs libldap-2.4-2 libldap-common libssl1.1
open-vm-tools openssl pollinate
9 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 3987 kB of archives.
After this operation, 114 kB of additional disk space will be used.
Get:1 http://
Get:2 http://
Get:3 http://
Get:4 http://
Get:5 http://
Get:6 http://
Get:7 http://
Get:8 http://
Get:9 http://
Preconfiguring packages ...
Fetched 3987 kB in 5s (800 kB/s)
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71242 files and directories currently installed.)
Preparing to unpack .../0-libssl1.
Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.2) over (1.1.1f-1ubuntu2.1) ...
Preparing to unpack .../1-openssl_
Unpacking openssl (1.1.1f-1ubuntu2.2) over (1.1.1f-1ubuntu2.1) ...
Preparing to unpack .../2-bind9-
Unpacking bind9-dnsutils (1:9.16.
Preparing to unpack .../3-bind9-
Unpacking bind9-libs:amd64 (1:9.16.
Preparing to unpack .../4-bind9-
Unpacking bind9-host (1:9.16.
Preparing to unpack .../5-libldap-
Unpacking libldap-common (2.4.49+
Preparing to unpack .../6-libldap-
Unpacking libldap-2.4-2:amd64 (2.4.49+
Preparing to unpack .../7-pollinate
Unpacking pollinate (4.33-3ubuntu1.
Preparing to unpack .../8-open-
Unpacking open-vm-tools (2:11.1.
Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.2) ...
Setting up libldap-common (2.4.49+
Setting up pollinate (4.33-3ubuntu1.
Setting up open-vm-tools (2:11.1.
Installing new version of config file /etc/vmware-
Setting up openssl (1.1.1f-1ubuntu2.2) ...
Setting up bind9-libs:amd64 (1:9.16.
Setting up libldap-2.4-2:amd64 (2.4.49+
Setting up bind9-host (1:9.16.
Setting up bind9-dnsutils (1:9.16.
Processing triggers for systemd (245.4-4ubuntu3.4) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Installing curl, cpu-checker, bridge-utils, tmux, ubuntu-fan
Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.68.0-
curl set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
msr-tools
The following NEW packages will be installed:
cpu-checker msr-tools
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 16.9 kB of archives.
After this operation, 66.6 kB of additional disk space will be used.
Get:1 http://
Get:2 http://
Fetched 16.9 kB in 0s (53.0 kB/s)
Selecting previously unselected package msr-tools.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71255 files and directories currently installed.)
Preparing to unpack .../msr-
Unpacking msr-tools (1.3-3) ...
Selecting previously unselected package cpu-checker.
Preparing to unpack .../cpu-
Unpacking cpu-checker (0.7-1.1) ...
Setting up msr-tools (1.3-3) ...
Setting up cpu-checker (0.7-1.1) ...
Processing triggers for man-db (2.9.1-1) ...
Reading package lists...
Building dependency tree...
Reading state information...
bridge-utils is already the newest version (1.6-2ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
tmux is already the newest version (3.0a-2ubuntu0.2).
tmux set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
dns-root-data dnsmasq-base libidn11
The following NEW packages will be installed:
dns-root-data dnsmasq-base libidn11 ubuntu-fan
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 400 kB of archives.
After this operation, 1229 kB of additional disk space will be used.
Get:1 http://
Get:2 http://
Get:3 http://
Get:4 http://
Fetched 400 kB in 1s (594 kB/s)
Selecting previously unselected package dns-root-data.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71269 files and directories currently installed.)
Preparing to unpack .../dns-
Unpacking dns-root-data (2019052802) ...
Selecting previously unselected package libidn11:amd64.
Preparing to unpack .../libidn11_
Unpacking libidn11:amd64 (1.33-2.2ubuntu2) ...
Selecting previously unselected package dnsmasq-base.
Preparing to unpack .../dnsmasq-
Unpacking dnsmasq-base (2.80-1.1ubuntu1.2) ...
Selecting previously unselected package ubuntu-fan.
Preparing to unpack .../ubuntu-
Unpacking ubuntu-fan (0.12.13) ...
Setting up dns-root-data (2019052802) ...
Setting up libidn11:amd64 (1.33-2.2ubuntu2) ...
Setting up dnsmasq-base (2.80-1.1ubuntu1.2) ...
Setting up ubuntu-fan (0.12.13) ...
Created symlink /etc/systemd/
Processing triggers for systemd (245.4-4ubuntu3.4) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Fetching Juju agent version 2.8.9 for amd64
Attempt 1 to download agent binaries from https:/
agent binaries from https:/
4e5d20bdc3b7650
1091694ab6178e7
Installing Juju machine agent
2021-02-24 13:59:39 INFO juju.cmd supercommand.go:54 running jujud [2.8.9 0 a48bedf0ae2096d
2021-02-24 13:59:41 INFO juju.agent identity.go:22 writing system identity file
2021-02-24 13:59:41 WARNING juju.mongo mongo.go:479 overwriting args.dataDir (set to /var/lib/juju) to /var/snap/
2021-02-24 13:59:41 INFO juju.mongo mongo.go:484 Ensuring mongo server is running; data directory /var/snap/
2021-02-24 13:59:41 WARNING juju.mongo service.go:326 configuring mongod with --noauth flag enabled
2021-02-24 13:59:41 INFO juju.packaging manager.go:103 installing "core" via "snap"
2021-02-24 13:59:41 INFO juju.packaging.
2021-02-24 13:59:55 INFO juju.packaging manager.go:103 installing "juju-db" via "snap"
2021-02-24 13:59:55 INFO juju.packaging.
2021-02-24 14:05:14 INFO juju.worker.
ERROR failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers
ERROR failed to bootstrap model: subprocess encountered error code 1
Example run:
https:/
It looks like after 5 minutes, (after snap install) we were unable to connect to mongo, which caused us to give up trying. I'm curious if there are logs around mongo's initialization itself if it would report a reason why it was unable to start.