Restarting libvirtd breaks Eucalyptus NC
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
eucalyptus (Ubuntu) |
Fix Released
|
Medium
|
Dustin Kirkland | ||
Lucid |
Fix Released
|
Medium
|
Dustin Kirkland |
Bug Description
Steps to reproduce:
- make sure Eucalyptus NC is up and running
- run /etc/init.
- try to start a new Instance. If all other NC's are busy and the new request lands in the current NC, then
/
will show:
[EUCAINFO ] doRunInstance() invoked (id=i-4912092F cores=2 disk=10 memory=512)
...
[EUCAERROR ] libvirt: cannot send data: Broken pipe (code=38)
...
[EUCAINFO ] currently running/booting: i-4912092F
[EUCAERROR ] libvirt: cannot send data: Broken pipe (code=38)
[EUCAFATAL ] hypervisor failed to start domain
...
[EUCAINFO ] doTerminateInst
[EUCAERROR ] libvirt: cannot send data: Broken pipe (code=38)
[EUCAWARN ] warning: domain i-4912092F to be terminated not running on hypervisor
Apparently the NC keeps trying to use the old libvirtd socket and doesn't notice the daemon was restarted.
Another VERY nasty consequence is that Eucalyptus looses track of the previously running instances in that node! (even if they stay running in KVM)
In CC.log:
[EUCAINFO ] TerminateInstan
[EUCAERROR ] ERROR: TerminateInstance() could not be invoked (check NC host, port, and credentials)
while in the NC we still have:
# virsh list
Connecting to uri: qemu:///system
Id Name State
-------
107 i-42A10746 running
At the very least the libvirt-bin/Euca-NC upstart dependencies should be such that restarting libvirtd restarts NC, or ideally and much better, Eucalyptus NC should be fixed to handle a restarted libvirtd.
Description: Ubuntu 9.10
Release: 9.10
ii euca2ools 1.0+bzr20091007
ii eucalyptus-common 1.6~bzr931-
ii eucalyptus-gl 1.6~bzr931-
ii eucalyptus-nc 1.6~bzr931-
Changed in eucalyptus (Ubuntu Lucid): | |
status: | In Progress → Fix Committed |
Needs reproduction on lucid / 1.6.2