[hardy] Under long-running load test, KVM guest freeze and host oops

Bug #335097 reported by Etienne Goyer
10
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Invalid
High
Stefan Bader
Hardy
Fix Released
High
Stefan Bader
Intrepid
Fix Released
High
Stefan Bader

Bug Description

SRU justification:

Impact: The function kvm_mmu_remove_write_access() runs under the slots_lock protection but the list it walks can be modified by other codepaths using the mmu_lock. This causes the host to Oops and the guests will hang.

Fix: Patch backported from upstream to add mmu_lock protection around the list walk.

Testcase: Running the validation suite for a longer period of time (24hrs).

When running a specific software validation suite in a KVM guest (both guest and host running hardy) for over 24 hours, the guest will eventually freeze and the host will have the following oops in dmesg:

 75243.174934] Unable to handle kernel paging request at 0000000000100100 RIP:
[75243.174947] [<ffffffff882cc545>] :kvm:kvm_mmu_slot_remove_write_access+0x55/0x70
[75243.174992] PGD 75072d067 PUD 76b738067 PMD 0
[75243.174997] Oops: 0000 [2] SMP
[75243.175001] CPU 4
[75243.175003] Modules linked in: tun bridge af_packet kqemu radeon drm rfcomm l2cap bluetooth kvm_intel kvm ppdev cpufreq_ondemand cpufreq_powersave cpufreq
_conservative cpufreq_userspace cpufreq_stats freq_table sbs sbshc container video output dock battery iptable_filter ip_tables x_tables ac parport_pc lp par
port ipv6 joydev serio_raw evdev psmouse pcspkr i2c_piix4 i2c_core button ext3 jbd mbcache sg sr_mod sd_mod cdrom ata_generic pata_acpi usbhid hid qla2xxx pa
ta_serverworks scsi_transport_fc aacraid ehci_hcd libata scsi_tgt ohci_hcd tg3 scsi_mod usbcore thermal processor fan fbcon tileblit font bitblit softcursor
fuse
[75243.175073] Pid: 7220, comm: kvm Tainted: G D 2.6.24-23-generic #1
[75243.175076] RIP: 0010:[<ffffffff882cc545>] [<ffffffff882cc545>] :kvm:kvm_mmu_slot_remove_write_access+0x55/0x70
[75243.175090] RSP: 0018:ffff81074a48be20 EFLAGS: 00010246
[75243.175092] RAX: 0000000000000000 RBX: ffff81072051c000 RCX: 00007fff48fe3ca0
[75243.175094] RDX: 0000000000100100 RSI: 0000000000000005 RDI: ffff81072051eaf0
[75243.175097] RBP: ffff81074a48be88 R08: 0000000000000000 R09: 0000000000100100
[75243.175099] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
[75243.175101] R13: ffff81072051c020 R14: 000000004010ae42 R15: 0000000000000000
[75243.175104] FS: 00007f5240fd26e0(0000) GS:ffff81081e0d7300(0000) knlGS:0000000000000000
[75243.175107] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[75243.175109] CR2: 0000000000100100 CR3: 000000076b640000 CR4: 00000000000026e0
[75243.175112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[75243.175115] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[75243.175118] Process kvm (pid: 7220, threadinfo ffff81074a48a000, task ffff81081c2087f0)
[75243.175119] Stack: ffffffff882c80f2 0000000000000000 ffff8107fa6177c8 000000011c2087f0
[75243.175126] ffff81072051c000 ffff81074a48be88 000000004010ae42 0000000000000008
[75243.175131] ffffffff882c5771 ffff8106eb3c6168 0000000000000000 0000000000000292
[75243.175135] Call Trace:
[75243.175148] [<ffffffff882c80f2>] :kvm:kvm_vm_ioctl_get_dirty_log+0x82/0xc0
[75243.175174] [<ffffffff882c5771>] :kvm:kvm_vm_ioctl+0xd1/0x200
[75243.175192] [<ffffffff80240920>] do_wait+0x4e0/0xcf0
[75243.175224] [<ffffffff802c0c0f>] do_ioctl+0x2f/0xa0
[75243.175235] [<ffffffff802c0ea0>] vfs_ioctl+0x220/0x2c0
[75243.175254] [<ffffffff802c0fd1>] sys_ioctl+0x91/0xb0
[75243.175274] [<ffffffff8020c39e>] system_call+0x7e/0x83
[75243.175307]
[75243.175308]
[75243.175309] Code: 49 8b 11 49 39 f9 0f 18 0a 75 b9 f3 c3 66 66 66 66 66 2e 0f
[75243.175322] RIP [<ffffffff882cc545>] :kvm:kvm_mmu_slot_remove_write_access+0x55/0x70
[75243.175334] RSP <ffff81074a48be20>
[75243.175336] CR2: 0000000000100100
[75243.175343] ---[ end trace 01e4e553c58023ce ]---

The guest freeze and host oops above is reproducible, with sensibly the same trace in dmesg. Attached two different dmesg output from two different run of the load test on the same machine.

Revision history for this message
Etienne Goyer (etienne-goyer-outlands) wrote :
Revision history for this message
Etienne Goyer (etienne-goyer-outlands) wrote :
Changed in linux:
assignee: nobody → stefan-bader-canonical
importance: Undecided → High
status: New → In Progress
Stefan Bader (smb)
description: updated
Revision history for this message
Stefan Bader (smb) wrote :
Changed in linux:
assignee: nobody → stefan-bader-canonical
importance: Undecided → High
status: New → Fix Committed
Revision history for this message
Stefan Bader (smb) wrote :
Changed in linux:
assignee: nobody → stefan-bader-canonical
importance: Undecided → High
status: New → Fix Committed
Revision history for this message
Stefan Bader (smb) wrote :

Jaunty not affected.

Changed in linux:
status: In Progress → Invalid
Revision history for this message
Martin Pitt (pitti) wrote :

Accepted intrepid into linux-proposed; please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Revision history for this message
Martin Pitt (pitti) wrote :

Accepted linux into hardy-proposed; please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.1 KiB)

This bug was fixed in the package linux - 2.6.24-24.53

---------------
linux (2.6.24-24.53) hardy-proposed; urgency=low

  [Stefan Bader]

  * Rebuild of 2.6.24-24.51 with 2.6.24-23.52 security patches applied.

linux (2.6.24-24.51) hardy-proposed; urgency=low

  [Alessio Igor Bogani]

  * rt: Updated PREEMPT_RT support to rt27
    - LP: #324275

  [Steve Beattie]

  * fix apparmor memory leak on deleted file ops
    - LP: #329489

  [Upstream Kernel Changes]

  * KVM: MMU: Add locking around kvm_mmu_slot_remove_write_access()
    - LP: #335097, #333409
  * serial: 8250: fix shared interrupts issues with SMP and RT kernels
    - LP: #280821
  * 8250.c: port.lock is irq-safe
    - LP: #280821
  * ACPI: Clear WAK_STS on resume
    - LP: #251338

linux (2.6.24-24.50) hardy-proposed; urgency=low

  [Alok Kataria]

  * x86: add X86_FEATURE_HYPERVISOR feature bit
    - LP: #319945
  * x86: add a synthetic TSC_RELIABLE feature bit
    - LP: #319945
  * x86: vmware: look for DMI string in the product serial key
    - LP: #319945
  * x86: Hypervisor detection and get tsc_freq from hypervisor
    - LP: #319945
  * x86: Use the synthetic TSC_RELIABLE bit to workaround virtualization
    anomalies.
    - LP: #319945
  * x86: Skip verification by the watchdog for TSC clocksource.
    - LP: #319945
  * x86: Mark TSC synchronized on VMware.
    - LP: #319945

  [Colin Ian King]

  * SAUCE: Bluetooth USB: fix kernel panic during suspend while streaming
    audio to bluetooth headset
    - LP: #331106

  [James Troup]

  * XEN: Enable architecture specific get_unmapped_area_topdown
    - LP: #237724

  [Stefan Bader]

  * Xen: Fix FTBS after Vmware TSC updates.
    - LP: #319945

  [Upstream Kernel Changes]

  * r8169: fix RxMissed register access
    - LP: #324760
  * r8169: Tx performance tweak helper
    - LP: #326891
  * r8169: use pci_find_capability for the PCI-E features
    - LP: #326891
  * r8169: add 8168/8101 registers description
    - LP: #326891
  * r8169: add hw start helpers for the 8168 and the 8101
    - LP: #326891
  * r8169: additional 8101 and 8102 support
    - LP: #326891
  * Fix memory corruption in console selection
    - LP: #329007

linux (2.6.24-23.52) hardy-security; urgency=low

  [Stefan Bader]
  * rt: Fix FTBS caused by shm changes
    - CVE-2009-0859

  [Steve Beattie]

  * fix apparmor memory leak on deleted file ops
    - LP: #329489

  [Upstream Kernel Changes]

  * NFS: Remove the buggy lock-if-signalled case from do_setlk()
    - CVE-2008-4307
  * sctp: Avoid memory overflow while FWD-TSN chunk is received with bad
    stream ID
    - CVE-2009-0065
  * net: 4 bytes kernel memory disclosure in SO_BSDCOMPAT gsopt try #2
    - CVE-2009-0676
  * sparc: Fix mremap address range validation.
    - CVE-2008-6107
  * copy_process: fix CLONE_PARENT && parent_exec_id interaction
    - CVE-2009-0028
  * security: introduce missing kfree
    - CVE-2009-0031
  * eCryptfs: check readlink result was not an error before using it
    - CVE-2009-0269
  * dell_rbu: use scnprintf() instead of less secure sprintf()
    - CVE-2009-0322
  * drivers/net/skfp: if !capable(CAP_NET_ADMIN): inverted logic
    - CVE-2009-0675
  * Ext4: Fix online res...

Read more...

Changed in linux (Ubuntu Hardy):
status: Fix Committed → Fix Released
Revision history for this message
Fabio Marzocca (thesaltydog) wrote :

When this patch will be available in hardy (not proposed) repositories?

Revision history for this message
Martin Pitt (pitti) wrote :

Fabio, it is in hardy-updates now.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (27.1 KiB)

This bug was fixed in the package linux - 2.6.27-14.33

---------------
linux (2.6.27-14.33) intrepid-proposed; urgency=low

  [Stefan Bader]

  * Fix FTBS due to a mysteriously missing ABI directory.

linux (2.6.27-14.32) intrepid-proposed; urgency=low

  [Stefan Bader]

  * Rebuild of 2.6.27-14.30 with 2.6.27-11.31 security patches applied

linux (2.6.27-14.30) intrepid-proposed; urgency=low

  [ Alexey Starikovskiy ]

  * SAUCE: ACPI: EC: Limit workaround for ASUS notebooks even more
    - LP: #288385

  [ Huaxu Wan ]

  * SAUCE: report rfkill changes event if interface is down
    - LP: #193970

  [ Scott James Remnant ]

  * SAUCE: floppy: Provide a PnP device table in the module.
    - LP: #255651

  [ Steve Beattie ]

  * fix apparmor memory leak on deleted file ops
    - LP: #329489

  [ Stefan Bader ]

  * Revert "ACPI: Fix compiler warnings introduced by 32 to 64 bit acpi
    conversions"
    - LP: #337019
  * Revert "ACPI: Change acpi_evaluate_integer to support 64-bit on 32-bit
    kernels"
    - LP: #337019

  [ Upstream Kernel Changes ]

  * KVM: MMU: Add locking around kvm_mmu_slot_remove_write_access()
    - LP: #335097, #333409
  * ricoh_mmc: Handle newer models of Ricoh controllers
    - LP: #311932

linux (2.6.27-13.29) intrepid-proposed; urgency=low

  [ Colin Ian King ]

  * SAUCE: Bluetooth USB: fix kernel panic during suspend while streaming
    audio to bluetooth headset
    - LP: #331106, #322082

  [ Stefan Bader ]

  * Revert "SAUCE: Work around ACPI corruption upon suspend on some Dell
    machines." (replaced by stable update)
    - LP: #330200
  * Revert "SAUCE: Add back in lost commit for Apple BT Wireless Keyboard"
    (replaced by stable update)
    - LP: #330902

  [ Upstream Kernel Changes ]

  * Revert "vt: fix background color on line feed"
    - LP: #330200
  * ti_usb_3410_5052: support alternate firmware
    - LP: #231276
  * fuse: destroy bdi on umount
    - LP: #324921
  * fuse: fix missing fput on error
    - LP: #324921
  * fuse: fix NULL deref in fuse_file_alloc()
    - LP: #324921
  * inotify: clean up inotify_read and fix locking problems
    - LP: #324921
  * mac80211: decrement ref count to netdev after launching mesh discovery
    - LP: #324921
  * sysfs: fix problems with binary files
    - LP: #324921
  * x86, mm: fix pte_free()
    - LP: #324921
  * alpha: nautilus - fix compile failure with gcc-4.3
    - LP: #324921
  * it821x: Add ultra_mask quirk for Vortex86SX
    - LP: #324921
  * libata: pata_via: support VX855, future chips whose IDE controller use
    0x0571
    - LP: #324921
  * rtl8187: Add termination packet to prevent stall
    - LP: #324921
  * serial_8250: support for Sealevel Systems Model 7803 COMM+8
    - LP: #324921
  * SUNRPC: Fix a memory leak in rpcb_getport_async
    - LP: #324921
  * SUNRPC: Fix autobind on cloned rpc clients
    - LP: #324921
  * USB: fix char-device disconnect handling
    - LP: #324921
  * USB: storage: add unusual devs entry
    - LP: #324921
  * USB: usbmon: Implement compat_ioctl
    - LP: #324921
  * ALSA: hda - add another MacBook Pro 4, 1 subsystem ID
    - LP: #324921
  * ALSA: hda - Add quirk for HP DV6700 laptop
    - LP: #324921
  * ALSA: ...

Changed in linux (Ubuntu Intrepid):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.