Ubuntu
nova package

qemu-nbd processes left behind after instance deletion

Bug #1287666 reported by Chris Stratford
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
nova (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

We have a couple of nova compute nodes which have ended up with a load of qemu-nbd processes:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24365 root 20 0 364m 2304 988 R 86 0.0 445:23.55 qemu-nbd
10512 root 20 0 363m 2044 924 R 70 0.0 512:55.08 qemu-nbd
28618 root 20 0 294m 2292 980 R 66 0.0 255:53.59 qemu-nbd
31353 root 20 0 294m 2300 984 R 65 0.0 254:18.38 qemu-nbd
12326 root 20 0 289m 2028 920 R 65 0.0 451:57.23 qemu-nbd
 1031 root 20 0 361m 2040 920 R 64 0.0 253:19.43 qemu-nbd
28248 root 20 0 291m 2032 924 R 64 0.0 442:52.39 qemu-nbd
 5999 root 20 0 294m 2292 984 R 64 0.0 198:28.80 qemu-nbd
28179 root 20 0 291m 2028 920 R 64 0.0 256:06.30 qemu-nbd
13010 root 20 0 291m 2036 924 R 62 0.0 512:27.49 qemu-nbd
18671 root 20 0 300m 2292 988 R 61 0.0 449:27.23 qemu-nbd

Looking at these processes, most of them refer to instances that no longer exist:

# ps -ef | fgrep 10512
root 10512 1 80 00:35 ? 08:34:07 /usr/bin/qemu-nbd -c /dev/nbd9 /srv/nova/instances/baf8f430-2a6d-4fee-82d3-12d2a0db32e5/disk

# ls -l /srv/nova/instances/baf8f430-2a6d-4fee-82d3-12d2a0db32e5/disk
ls: cannot access /srv/nova/instances/baf8f430-2a6d-4fee-82d3-12d2a0db32e5/disk: No such file or directory

There are no obvious errors in the logs to suggest anything went wrong.

This is version 1:2013.2.1-0ubuntu1~cloud0 running on precise.

Bug #1250231 appears to be describing something very similar if it helps in any way.

James Troup (elmo)
affects: nova → nova (Ubuntu)
Revision history for this message
James Page (james-page) wrote :

Chris

I've seen this happen when the server can't resolve its own hostname, resulting in extra output in the sudo commands that nova uses to create nbd devices; is that potentially happening in your case as well?

Changed in nova (Ubuntu):
status: New → Incomplete
Revision history for this message
Chris Stratford (chris-gondolin) wrote : Re: [Bug 1287666] Re: qemu-nbd processes left behind after instance deletion

On 07/03/14 10:03, James Page wrote:

> I've seen this happen when the server can't resolve its own hostname,
> resulting in extra output in the sudo commands that nova uses to create
> nbd devices; is that potentially happening in your case as well?

Unless there was a temporary DNS glitch, this doesn't seem to be the
case. The server can look up its own name happily and sudo doesn't show
any unexpected output for me.

Since it happened on more than one server at roughly the same time,
while I can't find any evidence in the logs to support it, an external
event (network glitch, busy firewall, etc.) is possible.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for nova (Ubuntu) because there has been no activity for 60 days.]

Changed in nova (Ubuntu):
status: Incomplete → Expired
Magesh GV (magesh-gv)
Changed in nova (Ubuntu):
status: Expired → Confirmed
James Page (james-page)
Changed in nova (Ubuntu):
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.