bzrlib.tests.test_transport.TestSSHConnections.test_bzr_connect_to_bzr_ssh

I've landed a little bit more logging in stub_sftp, so hopefully the next occurrence of this failure will provide some more hints.

I just hit this running selftest parallel=fork:

ERROR: bzrlib.tests.test_transport.TestSSHConnections.test_bzr_connect_to_bzr_ssh
----------------------------------------------------------------------
_StringException: Text attachment: log
------------
359.965 failed to load system host keys: [Errno 2] No such file or directory: '/tmp/testbzr-sIvONc.tmp/bzrlib.tests.test_transport.TestSSHConnections.test_bzr_connect_to_bzr_ssh/home/.ssh/known_hosts'
359.965 failed to load bzr host keys: [Errno 2] No such file or directory: '/tmp/testbzr-sIvONc.tmp/bzrlib.tests.test_transport.TestSSHConnections.test_bzr_connect_to_bzr_ssh/home/.bazaar/ssh_host_keys'
359.965 creating config directory: '/tmp/testbzr-sIvONc.tmp/bzrlib.tests.test_transport.TestSSHConnections.test_bzr_connect_to_bzr_ssh/home/.bazaar'
359.970 opening working tree '/tmp/testbzr-sIvONc.tmp'
------------
Text attachment: traceback
------------
Traceback (most recent call last):
  File "/home/mbp/lib/python/testtools/runtest.py", line 144, in _run_user
    return fn(*args)
  File "/home/mbp/lib/python/testtools/testcase.py", line 442, in _run_test_method
    testMethod()
  File "/home/mbp/bzr/work/bzrlib/tests/test_transport.py", line 981, in test_bzr_connect_to_bzr_ssh
    t.mkdir('foo')
  File "/home/mbp/bzr/work/bzrlib/transport/remote.py", line 242, in mkdir
    self._serialise_optional_mode(mode))
  File "/home/mbp/bzr/work/bzrlib/transport/remote.py", line 182, in _call2
    return self._client.call(method, *args)
  File "/home/mbp/bzr/work/bzrlib/smart/client.py", line 132, in call
    result, protocol = self.call_expecting_body(method, *args)
  File "/home/mbp/bzr/work/bzrlib/smart/client.py", line 145, in call_expecting_body
    method, args, expect_response_body=True)
  File "/home/mbp/bzr/work/bzrlib/smart/client.py", line 90, in _call_and_read_response
    readv_body=readv_body, body_stream=body_stream)
  File "/home/mbp/bzr/work/bzrlib/smart/client.py", line 63, in _send_request
    encoder.call(method, *args)
  File "/home/mbp/bzr/work/bzrlib/smart/protocol.py", line 1309, in call
    self._write_end()
  File "/home/mbp/bzr/work/bzrlib/smart/protocol.py", line 1133, in _write_end
    self.flush()
  File "/home/mbp/bzr/work/bzrlib/smart/protocol.py", line 1099, in flush
    self._real_write_func(''.join(self._buf))
  File "/home/mbp/bzr/work/bzrlib/smart/medium.py", line 396, in accept_bytes
    self._accept_bytes(bytes)
  File "/home/mbp/bzr/work/bzrlib/smart/medium.py", line 978, in _accept_bytes
    self._medium._accept_bytes(bytes)
  File "/home/mbp/bzr/work/bzrlib/smart/medium.py", line 794, in _accept_bytes
    self._ensure_connection()
  File "/home/mbp/bzr/work/bzrlib/smart/medium.py", line 818, in _ensure_connection
    '--directory=/', '--allow-writes'])
  File "/home/mbp/bzr/work/bzrlib/transport/ssh.py", line 332, in connect_ssh
    t = self._connect(username, password, host, port)
  File "/home/mbp/bzr/work/bzrlib/transport/ssh.py", line 287, in _connect
    self._raise_connection_error(host, port=port, orig_error=e)
  File "/home/mbp/bzr/work/bzrlib/transport/ssh.py", line 257, in _raise_connection_error
    orig_error=orig_error)
SocketConnectionError: ...

Could it be that we're just not synchronizing with the server before trying to connect to it, therefore there's a timing and load dependent case where it has not yet listened on the socket?

Jam's last fix was precisely about this point, so this shouldn't be the case anymore.

How many times should we hit issues with this test before we get rid of it ?

Sorry, this sounds far more rude than intended, ctrl-atl-del:

I think this test has failed far too may times for wrong reasons so we should fix it by either:
- adding more tests for specific aspects that will fail and be easier to fix than this one,
- since it is already known to be slow, let spawn the server in a subprocess avoiding some intra-process issues,
- add a babune specific run via test-local-server to cover even more cases regarding a true ssh connection to a smart server