arm64 xenial maas images don't include u-boot-tools package

May be that the arm64 image we get (the cloud-image) MAAS ends up using doesn't have the package required.

So the thing that happens is that the kernel can't be installed (during curthooks) because flash-kernel is trying to call mkimage (provided by the u-boot-tools package) and that fails. And so the deployment fails.

Log :
Nov 8 12:11:38 swirlix17 cloud-init[2367]: Running command ['chroot', '/tmp/tmp66hzpi9i/target', 'lsb_release', '--all'] with allowed return codes [0] (shell=False, capture=True)
Nov 8 12:11:38 swirlix17 cloud-init[2367]: Running command ['chroot', '/tmp/tmp66hzpi9i/target', 'dpkg', '--print-architecture'] with allowed return codes [0] (shell=False, capture=True)
Nov 8 12:11:38 swirlix17 cloud-init[2367]: got primary mirror: http://ports.ubuntu.com/ubuntu-ports
Nov 8 12:11:38 swirlix17 cloud-init[2367]: got security mirror: http://ports.ubuntu.com/ubuntu-ports
Nov 8 12:11:38 swirlix17 cloud-init[2367]: Apt Mirror info: {'MIRROR': 'http://ports.ubuntu.com/ubuntu-ports', 'PRIMARY': 'http://ports.ubuntu.com/ubuntu-ports', 'SECURITY': 'http://ports.ubuntu.com/ubuntu-ports'}
Nov 8 12:11:38 swirlix17 cloud-init[2367]: Running command ['chroot', '/tmp/tmp66hzpi9i/target', 'debconf-set-selections'] with allowed return codes [0] (shell=False, capture=True)
Nov 8 12:11:38 swirlix17 cloud-init[2367]: Running command ['chroot', '/tmp/tmp66hzpi9i/target', 'dpkg-query', '--list'] with allowed return codes [0] (shell=False, capture=True)
Nov 8 12:11:38 swirlix17 cloud-init[2367]: pkgs_cfgd: {'cloud-init'}
Nov 8 12:11:38 swirlix17 cloud-init[2367]: pkgs_installed: {'less', 'libncurses5', 'diffutils', 'libxdmcp6', 'apport-symptoms', 'libsasl2-modules', 'libprocps4', 'gnupg', 'liberror-perl', 'python3-json-pointer', 'lxd-client', 'xkb-data', 'libusb-0.1-4', 'xml-core', 'libbind9-140', 'dnsutils', 'libparted2', 'software-properties-common', 'libsepol1', 'friendly-recovery', 'zlib1g', 'openssh-client', 'python3-six', 'devio', 'gettext-base', 'python3-dbus', 'libpcap0.8', 'libapparmor1', 'libdevmapper-event1.02.1', 'fuse', 'netbase', 'libblkid1', 'ifenslave', 'krb5-locales', 'libsqlite3-0', 'tar', 'dns-root-data', 'ltrace', 'libsasl2-modules-db', 'libdns-export162', 'python3-pkg-resources', 'ncurses-bin', 'libpython3.5-stdlib', 'openssh-sftp-server', 'sosreport', 'ucf', 'libpci3', 'iputils-tracepath', 'lshw', 'debconf-i18n', 'gzip', 'makedev', 'libisccc140', 'strace', 'libc-bin', 'libhx509-5-heimdal', 'libaccountsservice0', 'libcap-ng0', 'flash-kernel', 'kmod', 'snap-confine', 'lxc-common', 'libdns162', 'libisc160', 'eatmydata', 'libslang2', 'iputils-ping', 'libgssapi-krb5-2', 'libselinux1', 'iproute2', 'libheimntlm0-heimdal', 'keyboard-configuration', 'python3-apt', 'unattended-upgrades', 'libxmuu1', 'lsb-release', 'wget', 'whiptail', 'vim', 'apt-transport-https', 'libmnl0', 'libdrm2', 'systemd', 'python3-prettytable', 'libkmod2', 'python3-cffi-backend', 'libnuma1', 'vim-tiny', 'pciutils', 'libcap2', 'libpcre3', 'libncursesw5', 'libdebconfclient0', 'plymouth-theme-ubuntu-text', 'libheimbase1-heimdal', 'pastebinit', 'libstdc++6', 'liblzma5', 'libsystemd0', 'libasn1-8-heimdal', 'libyaml-0-2', 'initramfs-tools', 'python3-chardet', 'mawk', 'python3-blinker', 'cloud-initramfs-dyn-netconf', 'libpolkit-agent-1-0', 'perl-base', 'update-manager-...

bug 1337538 has more context.
I think we probably need to make curtin know that it needs u-boot-tools on this platform and that it should install it.

The solution for this bug is:
 a.) find the logic for when u-boot-tools should be installed and
     implement that in curtin, and install u-boot-tools only when necessary.
     It is possible that this is "all the time on armhf and arm64".
 b.) remove u-boot-tools from maas-v2 images xenial+ (definitely zesty+)

Read on if you want more details.

If you install u-boot-tools in a system that does not use uboot to boot,
then update-initramfs will fail. update-initramfs is run by kernel
upgrade or other scenario. A workaround to that is setting a environment
variable FLASH_KERNEL_SKIP=true (or something like that) whenever you might run
update-initramfs. I personally think that is an unacceptable solution,
because effectively installation of u-boot-tools breaks your system if you
weren't using u-boot (unless you know the magic rune).

That is why 'flash-kernel' task is left open in bug
 https://bugs.launchpad.net/maas-images/+bug/1337538
u-boot-tools or flash-kernel should do the right thing at runtime instead
of causing kernel upgrade failure.

I've marked the maas-images bug there as Fix-Released as we went ahead and
took the easy way out and installed u-boot-tools into the maas image as a
solution.

With xenial, though, we got rid of the specific "maas" images and made
them just re-packs of the cloud-image with the same content, and the
maas-v3 stream is actually using the squashfs image bit-for-bit.
Note, there are currently 2 differences in the maas-v2 image for xenial:
 a.) maas image on arm has u-boot-tools (I had not realized that was still
the case, and this is true on both xenial and yakkety and zesty. We
should fix that by removing this package).
 b.) in xenial, there is a symlink (LP: #1534205). It is no longer
necessary, but still present as we shipped with it. It can be removed,
but just takes careful verification.

To fix the failed install for bug 1640519 we are not able to install
u-boot-tools into the squashfs image that is used for lxc, as that would
mean users of a container would see failure on upgrade of some packages.

What we need to do is make curtin decide to install u-boot-tools in much
the same way as it decides to install grub-pc or grub-uefi. However, I'm
not sure exactly what logic is used to determine when you should install
that package. it is possible that its "always" on armhf or arm64, as that
seems to have worked before.

Bug 1160360 says it is fix-released, and that flash-kernel will not run inside an lxc container. If that is a complete solution, then we could potentially add u-boot-tools to the squashfs image also.
Dan's comment at https://bugs.launchpad.net/maas-images/+bug/1337538/comments/8 describes this well, and also where we could pull logic to decide if installation is necessary.

I think you're confusing flash-kernel and u-boot-tools. u-boot-tools doesn't provide any kernel hooks AFAIK -- they're all in flash-kernel. And AFAIK flash-kernel will only crash in the way we've seen here if it detects that the platform it's running on requires a uImage, but u-boot-tools isn't installed. So AFAICT it's absolutely safe to install u-boot-tools everywhere, and probably safe to install flash-kernel everywhere. The crash occurs when flash-kernel is installed but some of the things it tries to run on the current platform are not.

Hello Andres, or anyone else affected,

Accepted curtin into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr437-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Lukasz, I tested curtin-common 0.1.0~bzr437-0ubuntu1~16.04.1 in xenial-proposed with MAAS 2.1.3+bzr5573-0ubuntu1

I was able to successfully deploy a xgene host, after updating to the latest curtin version.

Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial

Linux mustang-maas 4.4.0-38-generic #57-Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

curtin-common:
  Installed: 0.1.0~bzr437-0ubuntu1~16.04.1
  Candidate: 0.1.0~bzr437-0ubuntu1~16.04.1
  Version table:
 *** 0.1.0~bzr437-0ubuntu1~16.04.1 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

This bug was fixed in the package curtin - 0.1.0~bzr437-0ubuntu1~16.04.1

---------------
curtin (0.1.0~bzr437-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/new-upstream-snapshot: change to not use bzr merge-upstream.
  * New upstream snapshot.
    - pep8: fix pep8 errors found with 'make pep8' on zesty.
    - Workaround failures caused by gpg2 daemons left running in chroot.
      (LP: #1645680)
    - Install u-boot-tools when running on a system with u-boot. (LP: #1640519)
    - block: fix partition kname for raid devices (LP: #1641661)
    - Fix up tox errors that slipped in and new pycodestyle 2.1.0 complaints.
    - vmtests: adjust vmtest image sync metadata filenames
    - vmtests: Add centos support
    - Disable WilyTestRaid5Bcache vmtest
    - tools/xkvm: fix --netdev=<bridge>
    - bytes2human: fix for values larger than 32 bit int on 32 bit python2.

 -- Scott Moser <email address hidden> Wed, 18 Jan 2017 12:39:01 -0500

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Verified on Yakkety against xgene hardware. Working with MAAS 2.2

python3-curtin:
  Installed: 0.1.0~bzr437-0ubuntu1~16.10.1
  Candidate: 0.1.0~bzr437-0ubuntu1~16.10.1
  Version table:
 *** 0.1.0~bzr437-0ubuntu1~16.10.1 500
        500 http://archive.ubuntu.com/ubuntu yakkety-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety

This bug was fixed in the package curtin - 0.1.0~bzr437-0ubuntu1~16.10.1

---------------
curtin (0.1.0~bzr437-0ubuntu1~16.10.1) yakkety-proposed; urgency=medium

  * debian/new-upstream-snapshot: change to not use bzr merge-upstream.
  * New upstream snapshot.
    - revert: Test Workaround: skip XenialTestNvme for a short time.
    - Test Workaround: skip XenialTestNvme for a short time.
    - pep8: fix pep8 errors found with 'make pep8' on zesty.
    - Workaround failures caused by gpg2 daemons left running in chroot.
      (LP: #1645680)
    - Install u-boot-tools when running on a system with u-boot. (LP: #1640519)
    - block: fix partition kname for raid devices (LP: #1641661)
    - Fix up tox errors that slipped in and new pycodestyle 2.1.0 complaints.
    - vmtests: adjust vmtest image sync metadata filenames
    - vmtests: Add centos support
    - Disable WilyTestRaid5Bcache vmtest
    - tools/xkvm: fix --netdev=<bridge>
    - bytes2human: fix for values larger than 32 bit int on 32 bit python2.

 -- Scott Moser <email address hidden> Mon, 23 Jan 2017 14:30:25 -0500