qemu+ssh connections to a remote libvirt fail (from o to n)

Forgot -- this was working as recently as about one month ago.

Looks like a libvirt issue, reassigning.

@hggdh,

thanks for submitting this bug. I tried to reproduce it with a new uptodate natty and a new uptodate oneiric host. But from the oneiric host,

   virsh -c qemu+ssh://ubuntu@10.55.60.90/session list --all

worked, and I was able to start and watch and destroy a VM.

Is there anything else unusual about these hosts? Firewalling or custom libvirt configuration?

Hi Serge,

These hosts are in the QA lab, and only accessible via a SSH proxy. Firewalls are certainly in place, but no custom libvirt customisation to my knowledge. We can get you access if you need.

Interestingly, via the proxy I get this:

[cerdea-aws]cerdea@xango3:~$ virsh -c qemu+ssh://aldebaran/system list --all
error: packet 1550146349 bytes received from server too large, want 262144
error: failed to connect to the hypervisor
[cerdea-aws]cerdea@xango3:~$

And, if I am local at aldebaran:

cerdea@aldebaran~:$ virsh -c qemu+ssh://aldebaran/system list --all
cerdea@aldebaran's password:
 Id Name State
----------------------------------
1058 8913cfbf-4757-491f-b8c5-02df75d18f6e running

cerdea@aldebaran:~$

What is the arch of both? is one 32-bit and one 64?

My laptop is a 64-bit core-i7; aldebaran is a 64-bit, probably Xeon-based.

Also, running visrh -c qemu+ssh//localhost/system works on my laptop.

Quoting C de-Avillez (<email address hidden>):
> My laptop is a 64-bit core-i7; aldebaran is a 64-bit, probably Xeon-
> based.
>
> Also, running visrh -c qemu+ssh//localhost/system works on my laptop.

Hi,

I've failed to reproduce this. Is there any chance you would have time
to take these two exact same systems, re-install uptodate natty and
oneiric on them, and re-test? Or, could I get access to them (pls
feel free to ping me in irc) if they are still set up? If the oneiric
one is your laptop, I can try setting up a new oneiric client in the
lab.

thanks,
-serge

I just installed an Oneiric (Unity) on a local VM. Connecting to the Lab machines results in the same error. There is no option of reinstalling Natty on the Lab machines, these are production machines.

hggdh2, can you confirm the ssh path that is being undertaken.. is ssh being tunnelled over nc?

Thanks.

Yes, it is. Before trying it I have to start a SSH session to batuan (the gateway to the machines). Batuan itself is proxied thru yet another proxy. All of them run via nc.

The issue turned out to be due to how libvirt uses nc through ssh . hggdh had a line in his .ssh/config entry for the host containing

 LocalCommand printf "whatever"

The text being printed out was interpreted by the virsh client as coming from libvirt, so the first set of characters were read as the (far too large) length of the message.

Note that LocalCommand is a valid ssh_config option; if virsh cannot deal with it, it should be documented.

The virsh manpage doesn't mention ssh, so it sounds like the file /usr/share/doc/libvirt-doc/remote.html shipped with libvirt-doc could use a patch mentioning this.

hi,
I have a similiar issue but no special config for my ssh:

the ubuntu clients (one is maverick, one is natty) connect via ssh key authentication to libvirtd server.
All servers are fedora, except of one ubuntu (oneiric). The connection to the feodora are all ok for the ubuntu i get:

as normal user:
virsh -c qemu+ssh://<email address hidden>/system list --all
-> after entering the root pass: connected, the list appears

as root:
virsh -c qemu+ssh://<email address hidden>/system list --all
-> error:
error: Connection reset by peer
error: failed to connect to the hypervisor

debug shows:
15:36:04.233: debug : virInitialize:340 : register drivers
15:36:04.233: debug : virRegisterDriver:928 : registering Test as driver 0
15:36:04.233: debug : virRegisterNetworkDriver:734 : registering Test as network driver 0
15:36:04.233: debug : virRegisterInterfaceDriver:765 : registering Test as interface driver 0
15:36:04.233: debug : virRegisterStorageDriver:796 : registering Test as storage driver 0
15:36:04.233: debug : virRegisterDeviceMonitor:827 : registering Test as device driver 0
15:36:04.233: debug : virRegisterSecretDriver:858 : registering Test as secret driver 0
15:36:04.233: debug : virRegisterNWFilterDriver:889 : registering Test as network filter driver 0
15:36:04.233: debug : virRegisterDriver:928 : registering Xen as driver 1
15:36:04.233: debug : virRegisterDriver:928 : registering OPENVZ as driver 2
15:36:04.233: debug : virRegisterDriver:928 : registering remote as driver 3
15:36:04.233: debug : virRegisterNetworkDriver:734 : registering remote as network driver 1
15:36:04.233: debug : virRegisterInterfaceDriver:765 : registering remote as interface driver 1
15:36:04.233: debug : virRegisterStorageDriver:796 : registering remote as storage driver 1
15:36:04.233: debug : virRegisterDeviceMonitor:827 : registering remote as device driver 1
15:36:04.233: debug : virRegisterSecretDriver:858 : registering remote as secret driver 1
15:36:04.233: debug : virRegisterNWFilterDriver:889 : registering remote as network filter driver 1
15:36:04.233: debug : virConnectOpenAuth:1499 : name=qemu+ssh://<email address hidden>/system, auth=0xe7d778, flags=0
15:36:04.233: debug : do_open:1205 : name "qemu+ssh://<email address hidden>/system" to URI components:
  scheme qemu+ssh
  opaque (null)
  authority (null)
  server kavasir.XXXXXXXXXXX
  user rsgadmin
  port 0
  path /system

15:36:04.233: debug : do_open:1244 : trying driver 0 (Test) ...
15:36:04.233: debug : do_open:1250 : driver 0 Test returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 1 (Xen) ...
15:36:04.233: debug : do_open:1250 : driver 1 Xen returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 2 (OPENVZ) ...
15:36:04.233: debug : do_open:1250 : driver 2 OPENVZ returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 3 (remote) ...
15:36:04.233: debug : doRemoteOpen:565 : proceeding with name = qemu:///system
15:36:04.234: debug : virExecWithHook:712 : ssh -l rsgadmin kavasir.XXXXXX sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then CMD="nc -q 0 -U /var/run/libvirt/libvirt-sock";else CMD="nc -U /var/run/libvi...

@Tim,

please open a new bug for your problem.

Just wanted to let you know that this bug not only appears when you have the special SSH Option set, but also if you have any form of message inside your .bashrc or similar files. I got a small banner in my .bashrc, and connections via virt-manager / virsh reproducably will only work if I comment it out or rename the .bashrc so it will not get executed.

Thanks Max that solved this issue for me as well.