Package ifupdown breaks network configuration by cloud-init
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| cloud-init |
Fix Released
|
High
|
Chad Smith | ||
| ifupdown (Ubuntu) |
New
|
Undecided
|
Unassigned | ||
Bug Description
It appears the `ifupdown` package breaks networking configuration that cloud-init would normally handle (?) on both providers. The result, if you image an instance with the `ifupdown` package installed, is that the image is not bootable by Azure/AWS. I'm not familiar with how Azure/AWS/etc use cloud-init for network start up however. I'm filing here for now, in case `cloud-init` needs to be more defensive against `ifupdown`.
To reproduce with Packer (I confirmed this method):
* Add necessary Azure subscription env vars from `ifupdown-bug.json`
* `packer build ifupdown-bug.json`
* Try to boot an instance with resulting image
* Instance will never boot -- more specifically, networking never comes up. See `boot.log` for an example boot
To reproduce manually (only a guess, I did not confirm):
* Boot 18.04 ubuntu instance
* Install ifupdown
* Image the instance
* Try to boot instance using new image
* Instance won't boot
I hit this with Packer, creating new images for booting instances in scaling groups -- `ifupdown` is brought in by `salt`. If this is reproducible via manual installation, there could be some backup images/snapshots that are unable to boot though.
This appears to be because cloud-init network start up operations are broken by whatever `ifupdown` sysv/systemd scripts perform on boot. With `ifupdown` installed on Azure, `eth0` does not get an IP address, the instance never fully boots according to Azure, and it will enter an error state. On AWS, the instance boots, but network operations like SSH fail.
The `boot.log` is from Azure. Eventually `cloud-init` gives an exception (see `exception.log`), but this seems like a secondary issue related to networking being down.
I did try to upgrade cloud-init using nightly PPAs (version 20.1 I believe), but no luck. This is on Ubuntu 18.04 instances from Azure marketplace and AWS Canonical AMI, cloud-init version `19.4-33-
| AJ (agjohnson) wrote | #1 |
| Ryan Harper (raharper) wrote | #2 |
| Changed in cloud-init: | |
| importance: | Undecided → High |
| status: | New → Confirmed |
| tags: | added: uec-images |
| Changed in cloud-init: | |
| status: | Confirmed → In Progress |
| assignee: | nobody → Chad Smith (chad.smith) |
| Chad Smith (chad.smith) wrote | #3 |
| Chad Smith (chad.smith) wrote | #4 |
| Changed in cloud-init: | |
| status: | In Progress → Fix Committed |
| Chad Smith (chad.smith) wrote Fixed in cloud-init version 20.2. | #5 |
| Changed in cloud-init: | |
| status: | Fix Committed → Fix Released |
| James Falcon (falcojr) wrote | #6 |
Thanks for filing the issue.
I can reproduce this issue:
% lxc launch ubuntu-daily:bionic b7
% lxc exec b7 -- bash -c 'apt -qy install ifupdown; rm -f /etc/netplan/
% lxc restart b7
The issues at play are:
1) bionic Ubuntu images include a different /etc/network/
% lxc launch ubuntu-daily:xenial x7
% lxc exec x7 -- cat /etc/network/
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# Source interfaces
# Please check /etc/network/
# as interfaces may have been defined in /etc/network/
# See LP: #1262951
source /etc/network/
vs. Bionic
% lxc exec b77 -- cat /etc/network/
# ifupdown has been replaced by netplan(5) on this system. See
# /etc/netplan for current configuration.
# To re-enable ifupdown on this system, you can run:
# sudo apt install ifupdown
In particular, the Bionic images are missing this line:
source /etc/network/
2) cloud-init renders network-config to /etc/network/
3) networking.service runs ifup -a; however, it does not find a network config for the interfaces due to the missing source line in bionic ifupdown eni:
root@b77:~# cat /etc/network/
# ifupdown has been replaced by netplan(5) on this system. See
# /etc/netplan for current configuration.
# To re-enable ifupdown on this system, you can run:
# sudo apt install ifupdown
root@b77:~# ifquery --list --exclude=log
lo
root@b77:~# vi /etc/network/
root@b77:~# cat /etc/network/
# ifupdown has been replaced by netplan(5) on this system. See
# /etc/netplan for current configuration.
# To re-enable ifupdown on this system, you can run:
# sudo apt install ifupdown
source /etc/network/
root@b77:~# ifquery --list --exclude=log
lo
eth0
As a workaround, you can add the missing line to /etc/network/
source /etc/network/
We'll need to sort out if ifupdown packaging or cloud-init will need to ensure that line is present in images.