juju ssh/scp/run commands cause spurious key errors

Excerpts from Clint Byrum's message of Sun Jun 26 08:51:09 UTC 2011:
> Public bug reported:
>
> With lots of instances, one occasionally finds themselves using the same
> twice.
>
> Ensemble ssh should use a unique known_hosts file per environment, so as
> not to pollute the user's main known_hosts file.
>
> ** Affects: ensemble
> Importance: Undecided
> Status: New
>

ideally ensemble should also seed and store the ssh host fingerprints to prevent the (imo) spurious ack question from ssh, likely in ~/.ensemble/ssh_hosts

I have some working shell code that addresses at least part of this, which you can tap for ideas. It generates SSH keys, adds fingerprints to a separate known hosts file, and prunes them when done with the instance.

It's not mergeable into Juju as is, but the functionality is quite nice. It's in a shell script called 'cloud-sandbox' in lp:bikeshed.

Yes, we should do this.

A less obvious problem is that an invalid host key causes SSH tunneling to be disabled.

Amulet tests are spuriously failing due to this bug. IP addresses get recycled as the leases expire, and the 'juju run' commands Amulet makes fail due to the old host key being in the root users known_hosts file.

======================================================================
ERROR: test suite for <class 'tests.test_integration.Test3UnitDeployment'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stub/charms/cassandra/spike/.venv3/lib/python3.4/site-packages/nose/suite.py", line 210, in run
    self.setUp()
  File "/home/stub/charms/cassandra/spike/.venv3/lib/python3.4/site-packages/nose/suite.py", line 293, in setUp
    self.setupContext(ancestor)
  File "/home/stub/charms/cassandra/spike/.venv3/lib/python3.4/site-packages/nose/suite.py", line 316, in setupContext
    try_run(context, names)
  File "/home/stub/charms/cassandra/spike/.venv3/lib/python3.4/site-packages/nose/util.py", line 470, in try_run
    return func()
  File "/home/stub/charms/cassandra/spike/tests/test_integration.py", line 79, in setUpClass
    deployment.deploy(timeout=WAIT_TIMEOUT)
  File "/home/stub/charms/cassandra/spike/testing/amuletfixture.py", line 74, in deploy
    self.sentry.wait(timeout=timeout)
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 259, in wait
    status = self.unit[unit].juju_agent()
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 117, in juju_agent
    return self._run_unit_script("juju_agent.py")
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 114, in _run_unit_script
    raise IOError(output)
OSError: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
04:1f:98:dc:d5:aa:3a:4d:aa:d8:f6:a2:15:e5:fa:29.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:3
  remove with: ssh-keygen -f "/root/.ssh/known_hosts" -R 10.0.3.223
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
ERROR command timed out

FWIW - We saw this too in our automated OpenStack charm testing (UOSCI). Our work around is to overwrite known_hosts with our base known_hosts file on every build, on every jenkins slave. A bit of a hack, but it does the trick.

The root users .ssh/known_hosts is also getting polluted by 'juju run', which is causing subsequent 'juju run' commands to fail when IP addresses are recycled.

I'm also seeing IP addresses recycled much more often, especially with the local provider which now reuses them immediately.

I think this is dependent on security bug #892552

This was fixed a long time ago as part of bug 892552. Bug 1579593 still remains.