HVM/SSD Xenial AWS AMI doesn't launch on r3.large

This change was made by a bot.

Would it be possible for you to test the 16.04 kernel against other instance types to see if the bug happens with them as well?

The latest 16.04 kernel can be downloaded from:
https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa/+build/9734451

With this kernel, you need to install both the linux-image and linux-image-extra .deb packages.

If the bug is tied to the kernel, we could the test other kernel versions to narrow down the last good version and first bad version. That will allow us to perform a kernel bisect to find the exact commit that caused this.

Thanks in advance!

I know it works on c4 and m4 instance types. Those are the only other ones I've tried. To test this kernel on r3, I'd probably have to launch an instance on m4, upgrade the kernel from above, create a new AMI, and try running that on r3. I'm in major crunch right now, so I can't stop what I'm doing to do that, but I can try to squeeze it in while I'm waiting for other stuff to run, etc. We're upgrading our infrastructure to 16.04 and, for now, I'm just staying away from the r3 instances, but it will be important for us to support them at some point.

I believe I may have misspoken about r3 supporting PV. c3 supports both PV and HVM. r3 only supports HVM.

I wonder how AWS manages to get that odd (literally) number of CPUs. From the dmesg:

smpboot: Allowing 15 CPUs, 13 hotplug CPUs

From that it looks like possible=15 and num_processors=2 in prefill_possible_,map(). And possible is assigned to nr_cpu_ids after printing the line above. Confirmed in dmesg a little down:

setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:15 nr_node_ids:1

The divide error is in smp_init_package_map() which is inlined from smp_store_boot_cpu_info(). This would print "Max logical packages: ..." at some point which is not in dmesg. The only places there that look like they could cause a divide by 0 would be using ncpus. The first instance here:

ncpus = boot_cpu_data.x86_max_cores;
__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);

Have not yet dug down into where x86_max_cores gets set.

OK, looks like we need at least:

commit 56402d63eefe22179f7311a51ff2094731420406
Author: Thomas Gleixner <email address hidden>
Date: Fri May 6 20:48:16 2016 +0200

    x86/topology: Handle CPUID bogosity gracefully

    Joseph reported that a XEN guest dies with a division by 0 in the package
    topology setup code. This happens if cpu_info.x86_max_cores is zero.

That at least will avoid the crash. But the change only sets ncpus to 1 if it is 0. That still might lead to an unexpected number of available CPUs...

That commit also fixes bug 1573231. I have a test kernel already built for that bug if you want to test it. It can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1573231/

I will test it and post results here.

Yes, I am able to boot an r3.large instance using this kernel, and I see the correct number of CPUs. My procedure, in case anyone else wants to reproduce it, was to boot up from the latest test AMI (ami-c1cb23ac), sudo dpkg -i ./linux-headers-4.4.0-21* linux-image-* as downloaded from your link, and manually edit /boot/grub/grub.cfg to put this kernel first since it is an earlier version than 4.4.0-22, which is what was previously installed. I rebooted to make sure I got the new kernel and that the m4.large still worked. Then I stopped the instance, changed its instance type to r3.large, and started it. It came up fine, and /proc/cpuinfo shows the expected number of CPUs.