Crash in dom0 when accessing clipped RAM

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1111470

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Cannot generate log files since the server does not boot.

Set to confirmed as per automated instructions

The serial log only covers the hypervisor start (crash is about when the dom0 kernel would start). So its not clear which precise kernel version is failing. Also is there a previous precise kernel version that did work (which could be selected through grub)? Or was that the initial install?

I've updated to clarify it was kernel 3.2.0-36 which is failing (sorry - I tried to set this as the package but it switched to the source package and therefore lost the version information). A colleague reports that 3.2.0-15 is successful

3.2.0-15 would be a version even prior Precise release. I cannot even find a tag with that version. And for a 3.5.0 version number it would also be pre-release (but more likely possible).
I got no problems booting my test machine with that kernel, but that is AMD based. So its not quite a proof.

That 00228000 address from the IO space mapping error looks a bit related to the border of the last two E820 ranges. There is also this warning about MTRR ranges not covering all memory and ram being clamped down in some way... Seems all pointing to some odd memory setup issue. Maybe you could try to add "dom0_mem=512M,max:512M" as a test...

I know that 3.5.0-36 is a dev kernel - I followed http://packages.qa.ubuntu.com/qatracker/milestones/223/builds/25321/downloads to test it just to try and debug where the issue was.

Very curiously setting an explicit dom0 memory does allow the machine to boot - that was a surprise to me!

I don't know if my colleague is using an explicit dom0 memory allocation - if he is then it's very possible that it's not even a regression and all current/historic kernels may have required this flag. I'll check this tomorrow when my colleague is back.

I'm a collegue of "the collegue" that Bob mentioned. I just logged into "the collegue's" computer, and he's got dom0_mem=2048M in his Xen commandline:

GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=2048M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all"

He may have set this up before the first time he booted Xen. I've used Xen and Precise before, without setting dom0_mem, and I haven't run into this problem myself. I can't recall the last 3.2 kernel I used that worked, though. Bob, I suppose we could just install the last 10 or so and see if we can track down the fail.

To get a better idea of what is wrong can you run a test for me please? That is to boot the kernel+Xen without the 'dom0_mem' flag so that it crashes. But lets make sure that on the Linux command line you have: "console=hvc0 earlyprintk=xen debug loglevel=8" and for extra measure on the Xen command line pls also append: "sync_console console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all".

That should pinpoint where we fail.

Lastly, Stefan, is the kernel-debuginfo available somewhere easily? I am mighty curious what EIP: ffffffff816415e1 is? I figured it has to be pin_pagetable_pfn or xen_alloc_pte_init. And that the bug is somewhere in the E820 parsing where it includes 228000 as RAM.

Also, it would be interesting to see if 'e820-mtrr-clip e820-verbose ' on the Xen command line resolves this as well?

Bob, I was mentioning the versions not so much because it might be a development version but because I suspect there is some confusion going on between 3.2.0 and 3.5.0 and their abi versions (the number behind the '-'). Sometimes better to be very anal there in asking to avoid more confusion. ;-)

Bob, Mike, yeah I had been changing my test machine from using dom0_mem to allow it to use all memory but it did still work. I somewhat suspect that this could be related how this machine (or class of machines) set up their BIOS tables.

Konrad, usually those would be available at http://ddebs.ubuntu.com/pool/main/l/linux/. Unfortunately we, err, got a slight problem of occasionally loosing them. Since they are big there is a cleanup task at work. And that is sometimes a bit over-eager. I did a compile with 3.2.0-36.57 (seems that while not looking we went ahead and 3.2.0-37.58 is "current"). Here is the relevant snippet from System.map:

ffffffff81641543 t set_page_prot
ffffffff81641581 t xen_remap_domain_mfn_range.part.21
ffffffff8164158c t pin_pagetable_pfn
ffffffff816415e5 t p2m_top_index.part.3

So it seems to be pin_pagetable_pfn. But I will attach the whole map too. I am hesitating a bit for the other files since they are so big, but if you need anything, let me know.

Correction from previous (now hidden) comment - sorry, pasted wrong log.

Added the arguments requested by Konrad. If there are other kernels you'd like us to try to pinpoint which kernel introduced the issue, please let me know.

(XEN) Xen version 4.1.2 (Ubuntu 4.1.2-2ubuntu2.5) (<email address hidden>) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) Tue Jan 8 14:07:12 UTC 2013
(XEN) Console output is synchronous.
(XEN) Bootloader: GRUB 1.99-21ubuntu3.7
(XEN) Command line: placeholder loglvl=all guest_loglvl=all com1=115200,8n1 console=com1,vga noreboot sync_console
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 2 MBR signatures
(XEN) Found 2 EDD information structures
(XEN) WARNING: MTRRs do not cover all of memory.
(XEN) Truncating RAM from 9109504kB to 9043968kB
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000008f000 (usable)
(XEN) 000000000008f000 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000ce674000 (usable)
(XEN) 00000000ce674000 - 00000000ce6cc000 (ACPI NVS)
(XEN) 00000000ce6cc000 - 00000000cf5fb000 (usable)
(XEN) 00000000cf5fb000 - 00000000cf608000 (reserved)
(XEN) 00000000cf608000 - 00000000cf6a6000 (usable)
(XEN) 00000000cf6a6000 - 00000000cf6ab000 (ACPI data)
(XEN) 00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
(XEN) 00000000cf6f2000 - 00000000cf6f3000 (usable)
(XEN) 00000000cf6f3000 - 00000000cf6ff000 (ACPI data)
(XEN) 00000000cf6ff000 - 00000000cf700000 (usable)
(XEN) 00000000cf700000 - 00000000d0000000 (reserved)
(XEN) 00000000fff00000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000228000000 (usable)
(XEN) 0000000228000000 - 000000022c000000 (unusable)
(XEN) ACPI: RSDP 000FE020, 0014 (r0 INTEL )
(XEN) ACPI: RSDT CF6FD038, 004C (r1 INTEL DG965SS 685 1000013)
(XEN) ACPI: FACP CF6FC000, 0074 (r1 INTEL DG965SS 685 MSFT 1000013)
(XEN) ACPI: DSDT CF6F7000, 40E9 (r1 INTEL DG965SS 685 MSFT 1000013)
(XEN) ACPI: FACS CF6AB000, 0040
(XEN) ACPI: APIC CF6F6000, 0078 (r1 INTEL DG965SS 685 MSFT 1000013)
(XEN) ACPI: WDDT CF6F5000, 0040 (r1 INTEL DG965SS 685 MSFT 1000013)
(XEN) ACPI: MCFG CF6F4000, 003C (r1 INTEL DG965SS 685 MSFT 1000013)
(XEN) ACPI: ASF! CF6F3000, 00A6 (r32 INTEL DG965SS 685 MSFT 1000013)
(XEN) ACPI: SSDT CF6AA000, 01BC (r1 INTEL CpuPm 685 MSFT 1000013)
(XEN) ACPI: SSDT CF6A9000, 0175 (r1 INTEL Cpu0Ist 685 MSFT 1000013)
(XEN) ACPI: SSDT CF6A8000, 0175 (r1 INTEL Cpu1Ist 685 MSFT 1000013)
(XEN) ACPI: SSDT CF6A7000, 0175 (r1 INTEL Cpu2Ist 685 MSFT 1000013)
(XEN) ACPI: SSDT CF6A6000, 0175 (r1 INTEL Cpu3Ist 685 MSFT 1000013)
(XEN) System RAM: 8053MB (8247112kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000000228000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fe200
(XEN) DMI 2.4 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[404,0], pm1x_evt[400,0]
(XEN) ACPI: wak...

Oh, right, so it tries to actually map the last range (which is marked unusable in E820):

(XEN) 0000000228000000 - 000000022c000000 (unusable)
...
[ 0.000000] init_memory_mapping: 0000000228000000-000000022c000000
[ 0.000000] 0228000000 - 022c000000 page 4k
[ 0.000000] kernel direct mapping tables up to 22c000000 @ 1e97d8000-1e97fa000

Now why would it do that...

Stefan, so your gut feeling about the 228000 was right.

he E820_UNUSABLE regions are memory that "can" be used (and if you look in 'xen_memory_setup' that is how we set aside the memory for the balloon region - look for the 'type = E820_UNUSABLE). So by that logic, E820_UNUSABLE region should get the same treatment as the rest of the E820_RAM memory.

So this "hack"
diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index 8971a26..e8172bf 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -396,6 +396,7 @@ char * __init xen_memory_setup(void)
      extra_pages);
  i = 0;
  while (i < memmap.nr_entries) {
+ bool fix_unusable = true;
   u64 addr = map[i].addr;
   u64 size = map[i].size;
   u32 type = map[i].type;
@@ -407,9 +408,16 @@ char * __init xen_memory_setup(void)
     size = min(size, (u64)extra_pages * PAGE_SIZE);
     extra_pages -= size / PAGE_SIZE;
     xen_add_extra_mem(addr, size);
- } else
+ } else {
     type = E820_UNUSABLE;
+ fix_unusable = false;
+ }
   }
+ /*
+ * Not sure about this.
+ */
+ if (type == E820_UNUSABLE && fix_unusable)
+ type = E820_RESERVED;

   xen_align_and_add_e820_region(addr, size, type);

Would potentially fix it. But I am not sure what are the other cases where:
 a) It is OK to ignore E820_UNUSABLE altogether as provided by the hypervisor. Are there legitimate reasons for the BIOS to mark those as E820_UNUSABLE? Perhaps memory hotplug? (Jinsong from Intel could help answer that).
 b) Other? Perhaps the fix is in the hypervisor by clipping said memory completely out of the E820? In other words as if it had run with the 'mem=X' and it is oblivious to the non-MTRR region. But what if the MTRR region lies right in smack of other regions (like https://lkml.org/lkml/2012/8/24/474). The choice there would be to remove the E820 completly (but then we would think it is a PCI region, which might be OK or not - but this reminds me of http://lists.xen.org/archives/html/xen-devel/2011-02/msg01238.html where "gaps" are considered as I/O regions and could end up with the intel-agp trying to use it as its "flush" region).
c) Just leave it is as and document users to use 'dom0_mem=max' ? Perhaps we should codify it then? So if we detect the MTRR invalid regions we automatically set the dom0 maxpages as if 'dom0_mem=max:<up to MTRR>' was done? In reality that is what the hypervisor is doing - it will ignore those regions altogether - it is our misfortunate that we treat the region as if it was RAM (which actually is the right *thing*).

Thoughts?

So I found the following definition about the E820 address ranges in the ACPI 3.0a spec (section 14, table 14-1):

AddressRangeUnusable: "This range of address contains memory in which errors have been detected. This range must not be used by the OSPM."

That rather sounds like it may not be good to go there. I must admit, I cannot recall having seen "unusable". Rather "reserved". But who really remembers those things...

Do you want me to try the above "hack" as a patch to confirm that it works? I wasn't sure from the comments if this is a proposed fix or not!

Ok, I found the messages about trunkating memory in the hypervisor code. And when that happens, as Konrad mentioned, the range can be set to unusable by the hypervisor. It would be good to have a glimpse on the dmesg from a non-xen boot with the same kernel on the same machine. Just to see the E820 layout there and that should also show mtrr coverage.

Bob, sorry, seems you commented while I was typing up mine. I would wait with the hack. We are both not sure whether this is the right way to go.

Interesting bit of the normal dmesg boot looks to be:

[ 0.000000] mtrr_cleanup: can not find optimal value
[ 0.000000] please specify mtrr_gran_size/mtrr_chunk_size
[ 0.000000] e820: update [mem 0xcf700000-0xffffffff] usable ==> reserved
[ 0.000000] e820: update [mem 0x228000000-0x22bffffff] usable ==> reserved
[ 0.000000] WARNING: BIOS bug: CPU MTRRs don't cover all of memory, losing 64MB of RAM.
[ 0.000000] ------------[ cut here ]------------
[ 0.000000] WARNING: at /build/buildd/linux-lts-quantal-3.5.0/arch/x86/kernel/cpu/mtrr/cleanup.c:971 mtrr_trim_uncached_memory+0x299/0x2c0()
[ 0.000000] Hardware name:
[ 0.000000] Modules linked in:
[ 0.000000] Pid: 0, comm: swapper Not tainted 3.5.0-22-generic #34~precise1-Ubuntu
[ 0.000000] Call Trace:
[ 0.000000] [<ffffffff81052c9f>] warn_slowpath_common+0x7f/0xc0
[ 0.000000] [<ffffffff81052cfa>] warn_slowpath_null+0x1a/0x20
[ 0.000000] [<ffffffff81d002e7>] mtrr_trim_uncached_memory+0x299/0x2c0
[ 0.000000] [<ffffffff81cfa02b>] setup_arch+0x499/0x821
[ 0.000000] [<ffffffff81685ba6>] ? printk+0x61/0x63
[ 0.000000] [<ffffffff81cf3954>] start_kernel+0xd4/0x3d2
[ 0.000000] [<ffffffff81cf3397>] x86_64_start_reservations+0x131/0x135
[ 0.000000] [<ffffffff81cf3120>] ? early_idt_handlers+0x120/0x120
[ 0.000000] [<ffffffff81cf3468>] x86_64_start_kernel+0xcd/0xdc
[ 0.000000] ---[ end trace 056f32e7b63342ac ]---

I've attached the full dmesg incase there are other things that you want to see.

Oh yes, so this machine has a bit of a complicated usable/reserved pattern. Which cannot be represented with only the 8 variable MTRR registers. The Linux kernel seems to give up and make it reserved, while Xen tries to use it because it gets marked unusable which is actually used as "usable for guest memory" somehow...

Hm Konrad, I wonder, Xen has detected the same and that sets it to unusable, but then unusable seems to be used in a different meaning. Should those cases maybe use reserved, too?

Ok, lets try this (http://people.canonical.com/~smb/lp1111470/) modified hypervisor. For now I was a bit conservative and only modified things in a way that will cause the clipped (not covered by mtrr) area to be removed from e820 instead of changing the type to unusable. If that does work for the affected machine, I take the discussion about it upstream.

Any news?

I can confirm it's bootable with the patch.

Many thanks Stefan, and sorry for the delay in responding to the last message!

Thanks Bob. Now, lets see how proposal goes forth and back.

In upstream discussion this seems to be seen as something that should rather get fixed in the kernel and not the hypervisor. I will see what ideas come up there and get back with an updated change to test. Stay tuned...

One thing for clarification. When using the unpatched xen-hypervisor and a recent kernel (like v3.8) from http://kernel.ubuntu.com/~kernel-ppa/mainline/ and no dom0_mem option on the affected machine. Does that still crash the same?

We've already identified that a quantal development kernel (linux-image-3.5.0-22-generic) works with no dom0_mem option, so I imagine that the latest kernel would also work - is there a particular version that you'd like me to try?

If xen upstream thinks the kernel should be fixed, then is question whether we want to isolate+backport the fix which is already in the quantal development kernel?

The question can clearly answered with yes. The slight confusion was that during all the comments it was not clear to me whether the Quantal kernel was indeed used without or with the dom0_mem option. I took Mike's comment as if it was only *with* this option. Personally I find the usage of "unusable" for used memory slightly confusing but it would then not be the issue.

So if Quantal works even without that option, could you attach the dmesg of a Xen boot on that machine. Probably the difference in output helps to find out which changes make it work.

Okay - I wasn't sure that it was so clear! I thought there was a plan to backport the quantal kernel to precise as the LTS, but anyway!

See attached for the xen dmesg and host dmesg from a successful boot with original Xen and quantal kernel.

And the xen dmesg

The LTS backport kernels are mainly hardware enablement options. But the Precise kernel is still a supported option, so if possible we want bugs fixed (if possible == if not requiring half of the next kernel backported).

OK, so that range appears in the 1-1 mappings but seems not to be freed and the max_pfn later is below that range. Not sure yet what that exactly means but at least gives some better idea where to look.

Hm, one change to think of would be 2e2fb75475c2fc74c98100f1468c8195fee49f3b
  xen/setup: Populate freed MFNs from non-RAM E820 entries and gaps to E820 RAM

That would be in with v3.5-rc3. A final proof usually needs to bisect it. To narrow down the attempts there you could try various kernels from http://kernel.ubuntu.com/~kernel-ppa/mainline/. If it really was the change above, then a v3.5-rc2 would crash and a -rc3 would boot. Though usually one is not so lucky and the fix is in a -rc1, that is most things... :)

Unfortunately -rc1, -rc2 and -rc3 all boot on my machine... Sorry!

So the fix is rather somewhere before... still has v3,3 and v3.4 at least...

3.3 and 3.4 both boot... Am trying some earlier kernels

From http://kernel.ubuntu.com/~kernel-ppa/mainline/ v3-3-precise boots but v3-2-39-precise does not.

Ok, so something between 3.2 and 3.3 which obviously has not been considered as stable material. As said, it usually turns out that 3.3-rc1 is ok, too. I'd love to be wrong there but...

I went through changes between 3.2 and 3.3, but from the descriptions nothing was really jumping out. So I fear the only way to isolate it will be a bisect. It will be taking its time since usually there are about 10-12 iterations if 3.3-rc1 is the first one working. Might be a bit less if it would be a later rc.
Bob, if you have the time to try kernels, let me know (after finding the first good 3.3-rc) and I would prepare further kernels.

quantal has seen the end of its life and is no longer receiving any updates. Marking the quantal task for this ticket as "Won't Fix".

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release