juju ssh fails to connect when host keys have been changed (until the machine agent is restarted)

Bug #1940956 reported by Loïc Gomez
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Low
Unassigned

Bug Description

When trying to connect to a machine with juju ssh, it might fail with this error:

$ juju ssh 6
ERROR cannot connect to any address: [10.0.5.26:22 my.host.name:22]

Enabling debug gives a lot more details, including:

12:26:57 DEBUG juju.network.ssh reachable.go:110 host key for 10.0.5.26:22 not in our accepted set: log at TRACE to see raw keys
12:26:57 DEBUG juju.network.ssh reachable.go:110 host key for my.host.name:22 at 10.0.5.26:22 not in our accepted set: log at TRACE to see raw keys
12:26:57 DEBUG juju.network.ssh reachable.go:159 dial 10.0.5.26:22 failed with: dial tcp 10.0.5.26:22: i/o timeout
12:26:57 DEBUG juju.cmd.juju.commands ssh_machine.go:516 getting target "6" address(es) failed: cannot connect to any address: [10.0.5.26:22 my.host.name:22] (retrying)

We should be able to tell juju to either update or remove SSH host keys from its internal database in case there's been a voluntary change of these on a machine.

Also, the error message should be clearer and tell the user this is a host keys trust issue.

As a workaround, it's possible to juju ssh --no-host-key-checks, but security-wise I wouldn't recommend it.

Thanks

Loïc Gomez (kotodama)
description: updated
Haw Loeung (hloeung)
Changed in juju:
status: New → Confirmed
Revision history for this message
John A Meinel (jameinel) wrote :

Digging internally, it looks like the host keys are checked on machine agent start. So one workaround should be to just bounce the machine agent (juju run --machine 10 systemctl restart jujud-machine-10).

I think we could treat host key checking (where we got valid keys but they didn't match our records) at something higher than DEBUG.

John A Meinel (jameinel)
Changed in juju:
importance: Undecided → Medium
milestone: 2.9.1 → 2.9.13
status: Confirmed → Triaged
tags: added: bitesize
Revision history for this message
James Simpson (jsimpso) wrote :

Thanks John - can confirm that restarting the machine agent got this working for us. Really appreciate the quick response there!

FWIW we figured out that the root cause of our particular issue was actually an openssh package upgrade - we had originally removed our ECDSA host key and the package upgrade very helpfully created a new one.

Changed in juju:
milestone: 2.9.13 → 2.9.14
Changed in juju:
milestone: 2.9.14 → 2.9.15
John A Meinel (jameinel)
summary: - juju ssh fails to connect when host keys have been changed
+ juju ssh fails to connect when host keys have been changed (until the
+ machine agent is restarted)
Changed in juju:
importance: Medium → Low
Changed in juju:
milestone: 2.9.15 → 2.9.16
Changed in juju:
milestone: 2.9.16 → 2.9.17
Changed in juju:
milestone: 2.9.17 → 2.9.18
Changed in juju:
milestone: 2.9.18 → 2.9.19
Changed in juju:
milestone: 2.9.19 → 2.9.20
Changed in juju:
milestone: 2.9.20 → 2.9.21
Changed in juju:
milestone: 2.9.21 → 2.9.22
Changed in juju:
milestone: 2.9.22 → 2.9.23
Changed in juju:
milestone: 2.9.23 → 2.9.24
Changed in juju:
milestone: 2.9.24 → 2.9.25
Changed in juju:
milestone: 2.9.25 → 2.9.26
Changed in juju:
milestone: 2.9.26 → 2.9.27
Changed in juju:
milestone: 2.9.27 → 2.9.28
Changed in juju:
milestone: 2.9.28 → 2.9.29
Revision history for this message
John A Meinel (jameinel) wrote :

The one thing we could do better here is:
a) better docs around how to handle if your SSH server keys have changed
b) better error messages if trying to SSH is failing because the host keys the client is seeing don't match the expected host keys (as it is functionally a sign of a person-in-the-middle attack).
However, as the workaround works (restarting the machine agent), we probably won't schedule this work in the next 6 months.

Changed in juju:
milestone: 2.9.29 → none
Loïc Gomez (kotodama)
tags: added: canonical-is
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.