network configuration is disabled by default in cloud-init for ubuntu live server beta

Status changed to 'Confirmed' because the bug affects multiple users.

Hello,

subiquity (the Ubuntu Server installer) now writes the netplan configuration directly to the newly installed system, rather than using cloud-init. This is the intended behavior. If you are facing a problem because of this, could you please explain more in details what the problem is?

I'll mark this report as Incomplete for the moment. Please change its status back to New after commenting and we'll look at it again. Thanks!

Hi,

When does subiquity write netplan configuration? At installation? Will it run again on an installed system? If no, can we remove subiquity-disable-cloudinit-networking.cfg after installation done?

The problem is cloud-init can never write netplan configuration with subiquity-disable-cloudinit-networking.cfg exists. This prevents some user scenarios working which I'm sure they are working on Ubuntu 18.04:
For ex: On VMware vSphere platform, an Ubuntu system network can be customized to different netplan configuration by cloud-init. I believe public clouds such as AWS and Azure are using cloud-init to configure network for new instance.

Please share more information on how subiquity works with cloud-init, change the status back to New.

Hi,

Subiquity writes the netplan configuration at install time. Subiquity's only duty is to install the system, it can't be used to reconfigure the installed system after the fact. You can certainly remove subiquity-disable-cloudinit-networking.cfg if you need to.

I'd like to make clear that subiquity is the ISO image installer, and ISO images are normally used to install Ubuntu on bare metal. Cloud instances are deployed using the Ubuntu cloud images, and the deployment process does not involve subiquity: the configuration is done only via cloud-init.

More context on what you are trying to achieve exactly would help in understanding if something is not working as intended, or if there are use cases we can support better.

Thanks for the explanation.

On VMware vSphere platform, Customer can install Ubuntu Live Server system through ISO image on a Virtual Machine, then the Live Server Ubuntu VM will contain subiquity-disable-cloudinit-networking.cfg. This leads to cloud-init can NOT manage network on this VM by default.

If the Subiquity works one time, is it possible that subiquity-disable-cloudinit-networking.cfg is deleted after installation.

I mean the subiquity-disable-cloudinit-networking.cfg is deleted automatically, not by Customer.

> This leads to cloud-init can NOT manage network on this VM by default.

Can you describe in a little more detail what problems this is causing for you? ISO-installed systems generally don't want or need cloud-init to manage network configuration and I'm trying to understand the use case here.

Sure, list the use case step by step as below:
1. Customer create a Ubuntu VM on vSphere
2. Customer attach Live Server installer ISO to VM's CD-ROM
3. Customer power-on the VM and go through installation wizard to install Ubuntu Live Server (subiquity works here)
4. Customer want to create more Ubuntu Live Server VMs, so he clone the step3 created VM and Customize the new VMs network (cloud-init works here, using OVF datasource) to have different IP addresses.
5. Since the subiquity-disable-cloudinit-networking.cfg exists, cloned VMs fail to configure Netplan network configuration by cloud-init.

Hi Dan, Paride,

Any updates on this?

Hi Pengpeng, we're pulling together some notes on the best way forward. It seems this process you describe isn't ideal for your customers and we want to put together the right plan that should work much more smoothly for them.

Thanks Richard for the updates.
As far as I know, Ubuntu 20.04 has been GAed at Apr.24, 2020. So the change is not included in GA version(please correct me if I'm wrong), could you please let us know when the change will probably be shipped to Ubuntu 20.04 Live server.

Hi team,

Any updates on this, thanks.

Hi team,

Could you please share some info on this, when and how this issue is going to be fixed? Thanks.

Verified no such issue on 20.04 cloud image (OVA).

hit this issue in ubuntu 20.04.2 live server.

hit this issue in ubuntu 21.04 live server daily build.

hit this issue in ubuntu 21.04 live server GA build.

I've tested this and if affects the 18.04.5 server ISO as well.

@rharding are we able to list which install media this would apply to?

If subscribers here own https://kb.vmware.com/s/article/80934 in may be apt to update this.

Hi Brett Milford,
thanks for the update.

Also wanted to let you know, i tried with latest 20.04.03.

Beside the fact that 99-installer.cfg disables all datasource for further provisionig, even after adding datasources back in it doesnt configure network properly.

hit this issue in ubuntu 21.10 live server beta image.

I was going through testing of desktop live cd images here as well and noticed the same symptoms (which I think generally make sense for Desktop images). But, I wanted to reflect this to subiquity project as well as the use-case may not be captured as a user-story that is considered "supported". As this Live CD subiquity-based solution will also be present on Desktop installer images a common understanding of how clones of server Live iso and desktop ISO should be expected to perform if supplemental network config is needed.

I would suggest that the Ubuntu Server Live installer ISO is not the image a customer would want to be cloning on VMWare as the live installers isn't currently intended for that type of use-case.
That said I think we can come to a couple of short-term solutions that "work around" this case:

 1. recommending that customers using VMWare use standard ubuntu cloud images from https://cloud-images.ubuntu.com/ which leave cloud-init active.

2. customers interested in "cloning" ubuntu live server images need to do the following before cloning the image to ensure new OVF datasource is detected
   * rm /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
   * sudo cloud-init clean --logs

Cloud-init typically caches the datasource type it originally discovered. On Server Live installers and Ubuntu Jammy Desktop installer case, that datasource is "None" due to config in /etc/cloud/cloud.cfg.d/99-installer.cfg. If that datasource cache (/var/lib/cloud/instance/obj.pkl) still exists across reboot cloud-init happily reuses that datasource and will not try to detect others such as OVF. This why I think `sudo cloud-init clean --logs` is needed. Without it, those cloned images will always reboot and reuse the None datasource cache and cannot transition to detect OVF config.

cloud-init is involved in converstations with subiquity devs this cycle for 22.04 and will try to iron out how this should work on both Server Live installer ISOs vs Ubuntu 22.04 (Jammy) Desktop installer ISOs. But, at the moment it's going to be the manual process you have already captured, plus `sudo cloud-init clean --logs`.

Yes, I agree with Chad that things are working as designed here: the live server installs a system as configured during the install and not one that is expected to be configured in the future using cloud-init.

In general, the concept of an "installer" for VMs is a bit redundant, the expectation is that you will clone a cloud image to provision a new VM.

If vsphere needs the concept of an install step to operate nicely, we should think how we can serve that use case well. It probably won't involve subiquity well.

Hi again pengpengs and yhzou,

 Thanks for your patience on this issue.

   We've begun to make progress on this ticket to improve the use-case for reusing live-installed images as the golden-image for cloning.

Because some network configuration values set during installer prompts, wifi passwords, require root-readonly netplan files to ensure that the wifi passwords aren't world-readable.

The cloud-init network renderers doesn't currently have the support in it's network renderers to emit root-readonly subsets of network configuration as separate netplan configuration file with different global permissions based on sensitive data.

As such, subiquity can't use cloud-init network configuration directly at the moment, but I've filed a separate low-weight bug to track this work as it will allow subiquity to use cloud-init's native networking in the future.

In the meantime, we currently have some pull requests coming together that will simplify the golden image creation in VMware a bit planned for the next release of cloud-init 22.3:

1. cloud-init to support a /etc/cloud/clean.d directory to run third party cleanup scripts when
`cloud-init clean` is called
https://github.com/canonical/cloud-init/pull/1581
2. installer (subiquity) to deliver a /etc/cloud/clean.d/99-installer script which will automatically remove any network and DataSourceNone configuration artifacts to allow the installed image to be used for cloning
https://github.com/canonical/subiquity/pull/1347

Any comments or suggestions welcome as we close out on these initial improvements.

Related cloud-init bug for handling sensitive network config v2 which will unblock subiquity once released in cloud-init https://bugs.launchpad.net/cloud-init/+bug/1981646

Hi Chad,

Thanks for working on this, I've read the PRs and I think the idea is great. `sudo cloud-init clean` execution is suggested to customer who want to create a cloud-init ready golden-image on vSphere platform. I see the PRs are in progress and targeting the next cloud-init version 22.3, as usual, we will check SRU on VMware side.

Best regards,
Pengpeng