Cannot bootstrap with --bootstrap-series jammy on focal machine

Bug #2008020 reported by Bartosz Woronicz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
High
Unassigned

Bug Description

Deploying juju on Focal machine with --bootstrap-series jammy fails
as it cannot connect over SSH to the instance

juju 2.9.38 (stable 2023-01-17)
mastier@foauld:~$ snap info juju|grep installed
installed: 2.9.38 (21790) 97MB classic

mastier@foauld:~$ lsb_release -r
Release: 20.04
mastier@foauld:~$ juju --debug bootstrap localhost testjammyonfocal --bootstrap-series jammy
20:09:38 INFO juju.cmd supercommand.go:56 running juju [2.9.38 6d211be0d72d6f4d625c61c7c4ddb4e9325226c8 gc go1.18.9]
20:09:38 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/21790/bin/juju", "--debug", "bootstrap", "localhost", "testjammyonfocal", "--bootstrap-series", "jammy"}
20:09:38 DEBUG juju.container.lxd connection.go:180 using LXD socket at path: "/var/snap/lxd/common/lxd/unix.socket"
20:09:51 DEBUG juju.container.lxd network.go:181 created new nic device "eth0" in profile "default"
20:09:51 DEBUG juju.container.lxd connection.go:180 using LXD socket at path: "/var/snap/lxd/common/lxd/unix.socket"
20:09:53 DEBUG juju.container.lxd storage.go:85 created new disk device "root" in profile "default"
20:09:53 INFO cmd provider.go:356 Resolved LXD host address on bridge lxdbr0: https://10.111.175.1:8443
20:09:53 INFO cmd cloudcredential.go:47 updating credential store
20:09:53 DEBUG juju.cmd.juju.commands bootstrap.go:1313 authenticating with region "" and credential "localhost" ()
20:09:53 DEBUG juju.cmd.juju.commands bootstrap.go:1461 provider attrs: map[project:default]
20:09:54 INFO cmd authkeys.go:114 Adding contents of "/home/mastier/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
20:09:54 DEBUG juju.cmd.juju.commands bootstrap.go:1536 preparing controller with config: map[agent-metadata-url: agent-stream:released apt-ftp-proxy: apt-http-proxy: apt-https-proxy: apt-mirror: apt-no-proxy: authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDb4aaUWW0eQGnI56Dpe/s1QAi9l7xNvbNlNP/81oNzChEkjBf/0EGD5XfqBLqgTQlZFcNyjfb7pSVLCy9QSsnqW3ARX4lF+Mv77zv+qVtWi+Y1mKe+skh2u37JnvkY6zlWDpWC9ww32AtX9TCV6bd5zTEwT2OumTBeF7FXSIDBtMuWAfP50UGf2ncJH4L/Lfp7s0VI6EcWbLBs63+ZGyuAtie5x6cpSE8jiu+M7kXzqqlWyDosyyNiPdz/L+1xkTkq1iA5bPQ1e86N3zCfMjsCaUGs5nuRVGsVKGDmD9uRrSgzzc1E63jk1l1mUniA4bFghFbV+jsQCIAz1g25y9fr juju-client-key
 automatically-retry-hooks:true backup-dir: charmhub-url:https://api.charmhub.io cloudinit-userdata: container-image-metadata-url: container-image-stream:released container-inherit-properties: container-networking-method: default-series: default-space: development:false disable-network-management:false disable-telemetry:false egress-subnets: enable-os-refresh-update:true enable-os-upgrade:true fan-config: firewall-mode:instance ftp-proxy: http-proxy: https-proxy: ignore-machine-addresses:false image-metadata-url: image-stream:released juju-ftp-proxy: juju-http-proxy: juju-https-proxy: juju-no-proxy:127.0.0.1,localhost,::1 logforward-enabled:false logging-config: logging-output: lxd-snap-channel:5.0/stable max-action-results-age:336h max-action-results-size:5G max-status-history-age:336h max-status-history-size:5G name:controller net-bond-reconfigure-delay:17 no-proxy:127.0.0.1,localhost,::1 num-container-provision-workers:4 num-provision-workers:16 project:default provisioner-harvest-mode:destroyed proxy-ssh:false resource-tags: snap-http-proxy: snap-https-proxy: snap-store-assertions: snap-store-proxy: snap-store-proxy-url: ssl-hostname-verification:true test-mode:false transmit-vendor-metrics:true type:lxd update-status-hook-interval:5m uuid:5da5b4f5-1ed9-481d-8ee3-9d286a2dd13f]
20:09:54 INFO cmd bootstrap.go:856 Creating Juju controller "testjammyonfocal" on localhost/localhost
20:09:54 INFO juju.cmd.juju.commands bootstrap.go:927 combined bootstrap constraints:
20:09:54 DEBUG juju.environs.bootstrap bootstrap.go:320 model "controller" supports application/machine networks: true
20:09:54 DEBUG juju.environs.bootstrap bootstrap.go:322 network management by juju enabled: true
20:09:54 INFO cmd bootstrap.go:397 Loading image metadata
20:09:54 INFO cmd bootstrap.go:470 Looking for packaged Juju agent version 2.9.38 for amd64
20:09:54 INFO juju.environs.bootstrap tools.go:82 looking for bootstrap agent binaries: version=2.9.38
20:09:54 DEBUG juju.environs.tools tools.go:87 finding agent binaries in stream: "released"
20:09:54 DEBUG juju.environs.tools tools.go:89 reading agent binaries with major.minor version 2.9
20:09:54 DEBUG juju.environs.tools tools.go:98 filtering agent binaries by version: 2.9.38
20:09:54 DEBUG juju.environs.tools tools.go:101 filtering agent binaries by os type: ubuntu
20:09:54 DEBUG juju.environs.tools tools.go:104 filtering agent binaries by architecture: amd64
20:09:54 DEBUG juju.environs.tools urls.go:133 trying datasource "keystone catalog"
20:09:54 DEBUG juju.environs.simplestreams simplestreams.go:417 searching for signed metadata in datasource "default simplestreams"
20:09:54 DEBUG juju.environs.simplestreams simplestreams.go:452 looking for data index using path streams/v1/index2.sjson
20:09:56 DEBUG juju.environs.simplestreams simplestreams.go:748 using default candidate for content id "com.ubuntu.juju:released:agents" are {20210329 mirrors:1.0 content-download streams/v1/cpc-mirrors-agents.sjson []}
20:09:56 DEBUG juju.environs.simplestreams simplestreams.go:464 looking for data index using URL https://streams.canonical.com/juju/tools/streams/v1/index2.sjson
20:09:56 DEBUG juju.environs.simplestreams simplestreams.go:487 read metadata index at "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson"
20:09:56 DEBUG juju.environs.simplestreams simplestreams.go:1019 finding products at path "streams/v1/com.ubuntu.juju-released-agents.sjson"
20:09:57 INFO juju.environs.bootstrap tools.go:84 found 1 packaged agent binaries
20:09:57 INFO cmd bootstrap.go:483 Located Juju agent version 2.9.38-ubuntu-amd64 at https://streams.canonical.com/juju/tools/agent/2.9.38/juju-2.9.38-linux-amd64.tgz
20:09:57 INFO cmd bootstrap.go:581 Starting new instance for initial controller
20:09:57 INFO cmd environ.go:212 To configure your system to better support LXD containers, please see: https://linuxcontainers.org/lxd/docs/master/explanation/performance_tuning/
20:09:58 INFO cmd bootstrap.go:171 Launching controller instance(s) on localhost/localhost...
20:09:58 DEBUG juju.provider.lxd environ_broker.go:33 StartInstance: "0", jammy
20:09:58 DEBUG juju.cloudconfig.instancecfg instancecfg.go:923 Setting numa ctl preference to false
20:09:58 DEBUG juju.environs imagemetadata.go:119 obtained image datasource "default ubuntu cloud images"
20:09:58 DEBUG juju.container.lxd connection.go:197 LXD requires https://, using: https://cloud-images.ubuntu.com/releases/
20:10:03 DEBUG juju.container.lxd image.go:95 Found image remotely - "default ubuntu cloud images" "ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz" "6de3bbd44fa9fdaeef6e5a96b03bb0c09b59b6be7d594bcd3beb92c1956250b6"
20:10:03 DEBUG juju.container.lxd image.go:128 Copying image from remote server
20:11:24 DEBUG juju.service discovery.go:67 discovered init system "systemd" from series "jammy"
20:11:24 DEBUG juju.provider.lxd environ_broker.go:223 LXD user data; 3854 bytes
20:11:24 INFO juju.container.lxd container.go:256 starting new container "juju-2dd13f-0" (image "ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz")
20:11:24 DEBUG juju.container.lxd container.go:257 new container has profiles [default juju-controller]
20:12:05 DEBUG juju.container.lxd container.go:286 created container "juju-2dd13f-0", waiting for start...
20:12:10 INFO juju.provider.lxd environ_broker.go:48 started instance "juju-2dd13f-0"
20:12:10 INFO cmd bootstrap.go:309 - juju-2dd13f-0 (arch=amd64)
20:12:10 INFO juju.environs.bootstrap bootstrap.go:993 newest version: 2.9.38
20:12:10 INFO juju.environs.bootstrap bootstrap.go:1008 picked bootstrap agent binary version: 2.9.38
20:12:10 INFO cmd bootstrap.go:629 Installing Juju agent on bootstrap instance
20:12:13 DEBUG juju.environs.simplestreams simplestreams.go:417 searching for signed metadata in datasource "gui simplestreams"
20:12:13 DEBUG juju.environs.simplestreams simplestreams.go:452 looking for data index using path streams/v1/index2.sjson
20:12:15 DEBUG juju.environs.simplestreams simplestreams.go:464 looking for data index using URL https://streams.canonical.com/juju/gui/streams/v1/index2.sjson
20:12:15 DEBUG juju.environs.simplestreams simplestreams.go:467 streams/v1/index2.sjson not accessed, actual error: [{github.com/juju/juju/environs/simplestreams.(*urlDataSource).Fetch:192: "https://streams.canonical.com/juju/gui/streams/v1/index2.sjson" not found}]
20:12:15 DEBUG juju.environs.simplestreams simplestreams.go:468 streams/v1/index2.sjson not accessed, trying legacy index path: streams/v1/index.sjson
20:12:16 DEBUG juju.environs.simplestreams simplestreams.go:487 read metadata index at "https://streams.canonical.com/juju/gui/streams/v1/index.sjson"
20:12:16 DEBUG juju.environs.simplestreams simplestreams.go:1019 finding products at path "streams/v1/com.canonical.streams-released-dashboard.sjson"
20:12:17 INFO cmd bootstrap.go:786 Fetching Juju Dashboard 0.8.1
20:12:17 DEBUG juju.cloudconfig.instancecfg instancecfg.go:923 Setting numa ctl preference to false
Waiting for address
Attempting to connect to 10.111.175.125:22
20:12:18 DEBUG juju.provider.common bootstrap.go:651 connection attempt for 10.111.175.125 failed: ssh: connect to host 10.111.175.125 port 22: Connection refused
20:12:23 DEBUG juju.provider.common bootstrap.go:651 connection attempt for 10.111.175.125 failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:DDuq70+plaCnZt1EUbPphlgFnmdHO/Cl2t9Ar8m9jwI.
Please contact your system administrator.
Add correct host key in /tmp/juju-known-hosts2859455908 to get rid of this message.
Offending ED25519 key in /tmp/juju-known-hosts2859455908:3
  remove with:
  ssh-keygen -f "/tmp/juju-known-hosts2859455908" -R "10.111.175.125"
ECDSA host key for 10.111.175.125 has changed and you have requested strict checking.
Host key verification failed.
20:12:28 DEBUG juju.provider.common bootstrap.go:651 connection attempt for 10.111.175.125 failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:DDuq70+plaCnZt1EUbPphlgFnmdHO/Cl2t9Ar8m9jwI.
Please contact your system administrator.
Add correct host key in /tmp/juju-known-hosts2859455908 to get rid of this message.
Offending ED25519 key in /tmp/juju-known-hosts2859455908:3
  remove with:
  ssh-keygen -f "/tmp/juju-known-hosts2859455908" -R "10.111.175.125"
ECDSA host key for 10.111.175.125 has changed and you have requested strict checking.
Host key verification failed.
20:12:33 DEBUG juju.provider.common bootstrap.go:651 connection attempt for 10.111.175.125 failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:DDuq70+plaCnZt1EUbPphlgFnmdHO/Cl2t9Ar8m9jwI.
Please contact your system administrator.
Add correct host key in /tmp/juju-known-hosts2859455908 to get rid of this message.
Offending ED25519 key in /tmp/juju-known-hosts2859455908:3
  remove with:
  ssh-keygen -f "/tmp/juju-known-hosts2859455908" -R "10.111.175.125"
ECDSA host key for 10.111.175.125 has changed and you have requested strict checking.
Host key verification failed.

...

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:DDuq70+plaCnZt1EUbPphlgFnmdHO/Cl2t9Ar8m9jwI.
Please contact your system administrator.
Add correct host key in /tmp/juju-known-hosts2859455908 to get rid of this message.
Offending ED25519 key in /tmp/juju-known-hosts2859455908:3
  remove with:
  ssh-keygen -f "/tmp/juju-known-hosts2859455908" -R "10.111.175.125"
ECDSA host key for 10.111.175.125 has changed and you have requested strict checking.
Host key verification failed.
20:24:08 ERROR juju.provider.lxd environ_instance.go:35 failed to get instances from LXD: failed to begin transaction: sql: database is closed
20:24:08 ERROR juju.cmd.juju.commands bootstrap.go:885 failed to bootstrap model: refreshing addresses: failed to begin transaction: sql: database is closed
20:24:08 DEBUG juju.cmd.juju.commands bootstrap.go:886 (error details: [{github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:990: failed to bootstrap model} {github.com/juju/juju/environs/bootstrap.Bootstrap:713: } {github.com/juju/juju/environs/bootstrap.bootstrapIAAS:664: } {refreshing addresses: failed to begin transaction: sql: database is closed}])
20:24:08 DEBUG juju.cmd.juju.commands bootstrap.go:1647 cleaning up after failed bootstrap
20:24:08 INFO juju.provider.common destroy.go:21 destroying model "controller"
20:24:08 INFO juju.provider.common destroy.go:32 destroying instances
20:24:08 ERROR juju.cmd.juju.commands bootstrap.go:1649 error cleaning up: destroying instances: Get "https://10.111.175.1:8443/1.0/instances?instance-type=container&project=default&recursion=1": Unable to connect to: 10.111.175.1:8443 ([dial tcp 10.111.175.1:8443: connect: connection refused])
20:24:08 INFO cmd supercommand.go:544 command finished

Revision history for this message
Bartosz Woronicz (mastier1) wrote :

Full log attached. repetition of connection trials removed.

Revision history for this message
Bartosz Woronicz (mastier1) wrote :

Container is still runing, copy of tar gzipped /var/log

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 2008020] Re: Cannot bootstrap with --bootstrap-series jammy on focal machine

Looking at the initial log looks like we are expecting a key fingerprint of
ED25519
   Offending ED25519 key in /tmp/juju-known-hosts2859455908:3
But the new host is actually giving us a ECDSA back:
  ECDSA host key for 10.111.175.125 has changed and you have requested
strict checking.

I don't know why we would have expected one but gotten the other.

On Tue, Feb 21, 2023 at 3:55 PM mastier1 <email address hidden> wrote:

> Container is still runing, copy of tar gzipped /var/log
>
> ** Attachment added: "testjujujammyonfocal_varlog.tgz"
>
> https://bugs.launchpad.net/juju/+bug/2008020/+attachment/5648990/+files/testjujujammyonfocal_varlog.tgz
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/2008020
>
> Title:
> Cannot bootstrap with --bootstrap-series jammy on focal machine
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/2008020/+subscriptions
>
>

Ian Booth (wallyworld)
Changed in juju:
milestone: none → 2.9.41
status: New → Triaged
importance: Undecided → High
Changed in juju:
milestone: 2.9.41 → 2.9.42
Changed in juju:
milestone: 2.9.42 → 2.9.43
Changed in juju:
milestone: 2.9.43 → 2.9.44
Changed in juju:
milestone: 2.9.44 → 2.9.45
Changed in juju:
milestone: 2.9.45 → 2.9.46
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.