Groovy amd64 / arm64 / PowerPC deployment seems not working

Via MAAS, I was not able to deploy neither a ppc64le nor aarch64 system with Groovy.
Attaching console log for a POWER9 Boston (PowerNV 9006-12P)

Attaching console log for aarch64 Gigabyte Cavium THUNDERX gbt-mt30 system

What MAAS version were you using? Can you run `sosreport` on the MAAS server and share the output?

For the maas server we're using is:
MAAS version: 2.7.2 (8283-g.72f2ee59d-0ubuntu1~18.04.1)

The report file generated with `sosreport` is 1.74G, maybe it's not suitable for launchpad. Do you know if there is any place that I can upload it?
Thanks.

For the hypermaas which hosts ARM64 nodes, it's:
MAAS version: 2.8.1 (8567-g.c4825ca06-0ubuntu1~18.04.1)

For the powermaas which hosts PowerPC nodes, I can't connect to the MAAS web UI for now.
And I don't have admin SSH access to these two servers as well.

powermaas is : `Shared POWER MAAS version: 2.8.1 (8567-g.c4825ca06-0ubuntu1~18.04.1)`

Both logs show that even after mounting overlayfs with tmpfs backing the rootfs is read-only:

[ 25.370319] cloud-init[2118]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'

Can you try older kernel/initrd pairs?

I'm marking cloud-init task invalid; by the time cloud-init runs, rootfs should be rw using overlayfs backed by tmpfs.

@adam, I happened to have some cycles to look around. There are 2 MAAS servers which manage various ARCHS.

ARM64 & PPC64le

Both MAAS servers are running version 2.8.1 one appears to be from the snap, the other from PPA.

    MAAS name: maas MAAS
    MAAS version: 2.8.1 (8567-g.c4825ca06)

    MAAS name: Shared POWER MAAS
    MAAS version: 2.8.1 (8567-g.c4825ca06-0ubuntu1~18.04.1)

Regardless the bits in each of these builds I'm assuming is the same.

The ARM64, HWE MAAS is using the snap. I see the following failure when systems deploy:

[ OK ] Started LSB: automatic crash report generation.
[ OK ] Finished Set console scheme.
[ OK ] Created slice system-getty.slice.
[ OK ] Started Getty on tty1.
[ OK ] Reached target Login Prompts.
[ OK ] Started System Logging Service.
[ OK ] Started Login Service.
[ OK ] Started Unattended Upgrades Shutdown.
         Starting Authorization Manager...
[ OK ] Started Authorization Manager.
[ OK ] Started Accounts Service.
[ OK ] Started Dispatcher daemon for systemd-networkd.
[ OK ] Started Snap Daemon.
         Starting Wait until snapd is fully seeded...
[ OK ] Finished Pollinate to seed…seudo random number generator.
         Starting OpenBSD Secure Shell server...
[FAILED] Failed to start OpenBSD Secure Shell server.
See 'systemctl status ssh.service' for details.
[ OK ] Stopped OpenBSD Secure Shell server.
         Starting OpenBSD Secure Shell server...
[FAILED] Failed to start OpenBSD Secure Shell server.
See 'systemctl status ssh.service' for details.
[ OK ] Stopped OpenBSD Secure Shell server.
         Starting OpenBSD Secure Shell server...
[FAILED] Failed to start OpenBSD Secure Shell server.
See 'systemctl status ssh.service' for details.
[ OK ] Stopped OpenBSD Secure Shell server.
         Starting OpenBSD Secure Shell server...

Ubuntu 20.04.1 LTS ubuntu ttyAMA0

ubuntu login: Mounting Mount unit for snapd, revision 8543...
[ OK ] Mounted Mount unit for snapd, revision 8543.
[ OK ] Stopped Snap Daemon.
         Starting Snap Daemon...
[ OK ] Started Snap Daemon.
         Mounting Mount unit for core18, revision 1883...
[ OK ] Mounted Mount unit for core18, revision 1883.
         Mounting Mount unit for lxd, revision 16563...
[ OK ] Mounted Mount unit for lxd, revision 16563.
[ OK ] Listening on Socket unix for snap application lxd.daemon.
         Starting Service for snap application lxd.activate...
[ OK ] Finished Service for snap application lxd.activate.
[ OK ] Finished Wait until snapd is fully seeded.
         Starting Apply the settings specified in cloud-config...
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[ OK ] Finished Update UTMP about System Runlevel Changes.
[ 164.032142] cloud-init[3533]: Can not apply stage config, no datasource found! Likely bad things to come!
[ 164.032748] cloud-init[3533]: ------------------------------------------------------------
[ 164.033240] cloud-init[3533]: Traceback (most recent call last):
[ 164.033683] cloud-init[3533]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/m...

Attaching SOS report for arm64 maas 2.8.1

adding a console log for an amd64 (HPE Gen10 server)

just another heads-up: for the 3 systems that I've added the logs: ppc64le, arm64 and amd64 - I was able to deploy them with Focal.

Due to the major inconsistencies between all of the maas servers affected. I did this Started from scratch...

1.) Deployed Maas Snap from maas/stable 2.8
2.) Configured AMD64 groovy to download using the custom mirror (https://images.maas.io/ephemeral-v3/daily/)

3.) Because this was a test MAAS, i ensured DHCP was disabled and created a DHCP snippet on our production MAAS to direct all pxe requests to the test MAAS

4.) Test Deployed the machine( Success, the new test MAAS saw the enlisted machine)
5.) Successfully commissioned with 18.04
6.) Attempted to Deploy 20.10(Using the latest -daily)

 6a) This daily should include the latest fix for the following: https://bugs.launchpad.net/cloud-init/+bug/1886531 (per http://cloud-images.ubuntu.com/groovy/current/groovy-server-cloudimg-amd64.manifest this image has cloud-init 20.2-94-g3d06abc2-0ubuntu1 So 20200811 or later should work)

The deploy still fails.

I tried the oldest maas.io cloud image here: https://images.maas.io/ephemeral-v3/daily/groovy/amd64/ and still have the same outcome as the attached machine.txt

From the log:

Begin: Running /scripts/init-bottom ... Warning: overlayroot: configuring overlayroot with driver=overlay mode=tmpfs opts='' per /root.tmp/etc/overlayroot.local.conf
Failure: overlayroot: failed to modify /etc/fstab (step 2)
Success: overlayroot: configured root with 'tmpfs' using overlay per /root.tmp/etc/overlayroot.local.conf
done.

It appears that it's the same issue all around (the tmpfs overlay is not getting setup correctly) so the rootfs is not writable after the overlay.

But if it's reproducible on x86 then we can test this out in curtin vmtest. If we see the same failure then we can examine the initramfs logic and see why we're not getting proper overlayfs setup.

Probably related
 https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1886531

fwiw, fstab is essentially how overlayroot "communicates" to the system after pivot root about the different mounts.

You'll need to debug further to see whats going wrong. try using init=/bin/bash, and /run/initramfs/overlayroot.log should have more information.

You can also turn on debug with overlayroot=tmpfs,debug=1 then the 'debug' messages will go to the console.

But really I think this is just overlayroot not supporting this setup due to newer systemd or lack of fstab.

The tested maas.io image was : https://images.maas.io/ephemeral-v3/daily/groovy/amd64/20200811/ which i thought would have the latest fix for this bug: https://bugs.launchpad.net/cloud-init/+bug/1886531

I have no way to verify via a public manifest file. such as http://cloud-images.ubuntu.com/groovy/current/groovy-server-cloudimg-amd64.manifest

Possibly interesting is the fact that open-iscsi tests use overlayroot, and they are reportedly working:

http://autopkgtest.ubuntu.com/packages/open-iscsi/groovy/amd64

there is some more info on how those tests run at
https://git.launchpad.net/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md

I managed to mount the squashfs maas.io image from today (8.11.2020) and have found the following :

cloud-init is in fact the updated version, which is supposed to include the /etc/fstab fix. So it would appear there is still issues here.

ii cloud-guest-utils 0.31-22-g37d4e32a-0ubuntu1 all cloud gue>
ii cloud-init 20.2-94-g3d06abc2-0ubuntu1 all initializ>
ii cloud-initramfs-copymods 0.45ubuntu1 all copy init>
ii cloud-initramfs-dyn-netconf 0.45ubuntu1 all write a n>

@Sean you're right... the cloud-init fix was to handle the case there is *no* /etc/fstab ...
so any other place that expects /etc/fstab to exist must also handle missing file.

Looking at overlayroot, it expects to be able to update/modify the baked-in fstab, but it's not there.

https://git.launchpad.net/cloud-initramfs-tools/tree/overlayroot/scripts/init-bottom/overlayroot#n313

While I see that the cloud-initramfs-tools doesn't handle the case of a missing /etc/fstab I'm trying to find the situation that it's a valid use case to not have an /etc/fstab at all?

It looks like we've addressed other bugs by making sure an empty file exists:
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1877078

and that https://systemd.io/DISCOVERABLE_PARTITIONS/ states that it's not expected to not have the file at all.

We can look into addressing not having the file but it feels like it's just moving the issue around and something else can break with this assumption and we should have /etc/fstab, even if it's just empty.

My understanding is that overlayroot module tries to rewrite fstab to ensure it doesn't have invalid '/' entry. Well, if source image does not have fstab it means it doesn't need rewritting, and this should not be treated as an error.

In case one wants to test a local initrd change, you can do this:

wget http://images.maas.io/ephemeral-v3/daily/groovy/amd64/20200812/ga-20.10/generic/boot-initrd
UNPACKED=`pwd`/unpacked
unmkinitramfs boot-initrd unpacked

# Make changes in unpacked/main/; repack with
NEW_INITRD=`pwd`/boot-initrd.new
(cd $UNPACKED/early && find . -print0 | cpio --null --create --format=newc) > ${NEW_INITRD}
(cd $UNPACKED/early2 && find . -print0 | cpio --null --create --format=newc) >> ${NEW_INITRD}
(cd $UNPACKED/main && find . -print0 | cpio --null --create --format=newc | lz4 -9 -l) >> ${NEW_INITRD}

With the linked MR patch to overlayroot in main/scripts/init-bottom/overlayroot curtin's vmtest GroovyTestBasic now passes (where it hung in the past).

Note, I'm fairly sure that Paride had groovy tests passing back in June or early July, so this squashfs removal of etc/fstab for *MAAS* images must have changed after we landed groovy release to curtin vmtest.

@xnox,
Your attached patch, which says simply to not error if no fstab file
exists will ultimately hide further errors.

I suspect there are (at least) 3 scenarios:
a.) /etc/fstab exists. happy. this works fine.

b.) /etc/fstab does not exist, there is only one system mount, which can
be read from root= kernel command line. your patch makes this case work
fine.

c.) systemd .mount entries (or some other mechanism) are used.
Your patch turns a warning message from overlayroot into a silent failure.

Additionally, I'm not sure how your patch would work. I think the problem
is that root is not getting re-mounted RW. That is typically done
by systemd, and I thought fstab played a role in conveying that intent to
systemd.

Somehow overlayroot has to convey to systemd that / needs to be mounted
read-write. And also... with 'recurse' functionality, any system mounts
other than / need to be updated. For the 'c' case above, overlayroot
would need new function to do that.

@xnox 's patch does not actually fix the issue; it just prevents overlayroot from raising an error; the system is still left with a read-only rootfs and nothing to tell the rootfs' init to remount / as rw.

> Somehow overlayroot has to convey to systemd that / needs to be mounted
read-write.

overlayroot current communicates this via writing fstab with updated mounts; at minimum, the rootfs; possibly more depending on what's in the rootfs's /etc/fstab.

Then systemd-fstab-generator(8) runs and is what creates the .mount units which then mount/remount filesystems per fstab. It will read from an /etc/fstab (or kernel parameters) and it does not understand overlayroot kernel parameters, so without an fstab entry to indicate which path/device is root and that it should be mounted rw, it stays in read-only.

overlayroot should generate correct and valid /etc/fstab for the rootfs it will mount.
i.e. the
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0

However, it needs not to take any /etc/fstab as input to generate that. Currently the code requires for /media/root-ro/etc/fstab to exist, and contain '/' entry which will get replaced by above.

Surely the right thing to do here is for overlayroot module to write out the /media/root-ro/ etc without needing a mock 'fstab' without the false entry `LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0`.

@smoser @raharper

"the system is still left with a read-only rootfs and nothing to tell the rootfs' init to remount / as rw." => why is initrd mounting overlayfs RO, instead of RW.

"so this squashfs removal of etc/fstab for *MAAS* images" => note there are no MAAS-specific squashfs, the squashfs used by MAAS is the same one as used by LXD containers. MAAS images continue to be built untested into the default stream for groovy still. (being fixed).

If the kernel cmdline is specified as 'ro', we force remount the overlyfs as ro, and require valid fstab in target, to generate rootfs mount unit, to remount it rw later.

This is very archaic.

@rharper

How do you run curtin's vmtest with a patched initrd?

# curtin vmtest instructions

https://code.launchpad.net/~xnox/cloud-initramfs-tools/+git/cloud-initramfs-tools/+merge/389202/comments/1023604

<smoser> [ubuntu/groovy-proposed] cloud-initramfs-tools 0.46ubuntu1 (Accepted)

Fixed in groovy at cloud-initramfs-tools 0.46ubuntu1

FWIW, I tried a deployment on an AMD64-architecture server using an ASRock MT-C224 motherboard (one of the experimental Orange Box v3 nodes in 18T), and it failed. I can provide additional logs if required. I have not attempted to install any of the branches associated with this bug report. (I'm not entirely sure how I'd do so.)

@Rod

I wouldn't bother until you see an image with serial greater than 20200813.
If it doesn't have cloud-initramfs-tools > 0.45ubuntu1, its not going
to work with groovy.

then test that and complain with logs if its not working.

On Fri, Aug 14, 2020 at 1:15 PM Rod Smith <email address hidden> wrote:
>
> FWIW, I tried a deployment on an AMD64-architecture server using an
> ASRock MT-C224 motherboard (one of the experimental Orange Box v3 nodes
> in 18T), and it failed. I can provide additional logs if required. I
> have not attempted to install any of the branches associated with this
> bug report. (I'm not entirely sure how I'd do so.)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1890803
>
> Title:
> Groovy amd64 / arm64 / PowerPC deployment seems not working
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1890803/+subscriptions

I happened to check this updated -daily image again this morning which includes, cloud-initramfs-tools 0.46ubuntu1 and still see that the image remains broken.

I have attached the console logs from this latest deploy. Looks like we made quite a bit of headway, still missing some bits that need to be updated possibly for lack of /etc/fstab.

I have worked around the problem by creating a custom focal cloud image and deploying it via maas using a bionic ephemeral. However, that is simply a hack at this stage and not a valid fix.

I would appear systemd-remount-fs is failing to complete from looking at that inital logs, not sure if I can get more information out of the system yet i will try.

[ OK ] Finished Load Kernel Modules.
[ 18.650374] systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
[ 18.653462] systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
[ 18.661691] systemd[1]: Failed to start Remount Root and Kernel File Systems.
[FAILED] Failed to start Remount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
[ 18.669951] systemd[1]: Started Journal Service.

It turns out the 20200814 squashfs from maas *now* has an /etc/fstab in it ...

So after overlayroot runs, we have an fstab like this:

root@(none):/# cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
#
# This fstab is for overlayroot. The real one can be found at
# /media/root-ro/etc/fstab
# The original entry for '/' and other mounts have been updated to be placed
# under /media/root-ro.
# To permanently modify this (or any other file), you should change-root into
# a writable view of the underlying filesystem using:
# sudo overlayroot-chroot
#
#LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0

Overlayroot needs to handle with and without fstab entry so we'll need to fix that; However it's not clear why there is now an fstab entry in squashfs.

https://code.launchpad.net/~raharper/cloud-initramfs-tools/+git/cloud-initramfs-tools/+merge/389422

The fstab was removed in LP: #1877078, because it was invalid. The empty fstab is equivalent to not having an fstab and missing /etc/fstab is valid, at least I found no source (like FHS) saying that it must exist even if it would be empty.

[ubuntu/groovy-proposed] cloud-initramfs-tools 0.47ubuntu1 (Accepted)

^ landed Ryan's fix (comment #38 mp 389422)

Kernel SRU is mid cycle now, someone else will have to verify this is fixed, unfortunately the kernel team is using most of the MAAS servers that would probably be used for verification. So many teams may have to wait as we can't afford to have any more delays.

Once a new maas image is published with the updated initramfs, I can verify; As of today, I still only see 20200814 for groovy

% date
Wed 19 Aug 2020 11:03:04 AM CDT
% image-status maas3 | grep groovy
groovy amd64/ga-20.10/generic 20200814 squashfs
groovy amd64/ga-20.10/lowlatency 20200814 squashfs

I can verify on the same servers I've tested before.
Please let me know when the new image is available

Updated image is available:

% image-status maas3 | grep groovy
groovy amd64/ga-20.10/generic 20200819.1 squashfs
groovy amd64/ga-20.10/lowlatency 20200819.1 squashfs

GroovyTestBasic on amd64 passes.

Thu, 20 Aug 2020 09:40:51 -0500: vmtest start: nosetests3 -vv --nologcapture tests/vmtests/test_basic.py:GroovyTestBasic
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
fatal: No names found, cannot describe anything.
fatal: No names found, cannot describe anything.
2020-08-20 09:40:51,547 - vmtests - INFO - Logfile: /home/rharper/work/git/curtin/output/debug.log . Working dir: /home/rharper/work/git/curtin/output
2020-08-20 09:40:51,550 - GroovyTestBasic - INFO - Logfile: /home/rharper/work/git/curtin/output/debug.log . Working dir: /home/rharper/work/git/curtin/output
2020-08-20 09:40:51,553 - GroovyTestBasic - INFO - Starting setup for testclass: GroovyTestBasic (ubuntu/groovy -> ubuntu/groovy)
2020-08-20 09:40:51,619 - GroovyTestBasic - INFO - Using tempdir: /home/rharper/work/git/curtin/output/GroovyTestBasic
2020-08-20 09:40:51,620 - GroovyTestBasic - INFO - Loading testcase config file: examples/tests/basic.yaml
=> groovy/amd64/20200819.1/ga-20.10/generic/boot-initrd [86118105]
.^[[5;2~..............................................................................
=> groovy/amd64/20200819.1/ga-20.10/generic/boot-kernel [11662080]
...............................................................................
=> groovy/amd64/20200819.1/squashfs [377270272]
...............................................................................
2020-08-20 09:42:12,196 - GroovyTestBasic - INFO - Publishing ephemeral image as --publish=/srv/images/groovy/amd64/20200819.1/squashfs:root/squashfs
2020-08-20 09:42:12,196 - GroovyTestBasic - INFO - Curtin install source URI: cp:///media/root-ro
2020-08-20 09:42:12,207 - GroovyTestBasic - INFO - Loading testcase config file: examples/tests/basic.yaml
2020-08-20 09:42:12,214 - GroovyTestBasic - INFO - disk_serials: ['serial=disk-a', 'serial=disk-b', 'serial=disk-c', 'serial=disk-d']
2020-08-20 09:42:12,214 - GroovyTestBasic - INFO - nvme_serials: []
2020-08-20 09:42:12,214 - GroovyTestBasic - INFO - nvme disks: []
2020-08-20 09:42:12,231 - GroovyTestBasic - INFO - Using root_disk: []
2020-08-20 09:42:12,231 - GroovyTestBasic - INFO - Running curtin installer: /home/rharper/work/git/curtin/output/GroovyTestBasic/logs/install-serial.log
2020-08-20 09:44:42,576 - GroovyTestBasic - INFO - GroovyTestBasic[install]: boot took 150.34 seconds. returned True
2020-08-20 09:44:42,896 - GroovyTestBasic - INFO - Install OK
2020-08-20 09:44:42,897 - GroovyTestBasic - INFO - Booting target image: /home/rharper/work/git/curtin/output/GroovyTestBasic/logs/boot-serial.log
2020-08-20 09:44:54,320 - GroovyTestBasic - INFO - GroovyTestBasic[first_boot]: boot took 11.42 seconds. returned True
2020-08-20 09:44:54,367 - GroovyTestBasic - INFO - GroovyTestBasic: setUpClass finished. took 242.81 seconds. Running testcases.
get_test_files (vmtests.test_basic.GroovyTestBasic) ... ok
test_clear_holders_ran (vmtests.test_basic.GroovyTestBasic) ... ok
test_curtin_install_version (vmt...

tested image from 20200819.1 - 19-Aug-2020 for arm64 and ppc64le and it works now:

`Welcome to Ubuntu Groovy Gorilla (development branch) (GNU/Linux 5.4.0-42-generic aarch64)`
`Welcome to Ubuntu Groovy Gorilla (development branch) (GNU/Linux 5.4.0-42-generic ppc64le)`

LP: #1877078 fix got reverted and the sqashfs and LXD images ship with invalid fstab again booting into degraded mode.

Groovy deployment on ppc64el hardware now working. Marking "ubuntu-power-systems" series as "Fix Released". Thanks!

Closing this issue in ubuntu-kernel-tests, thank you all for your help!

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3770