resolv.conf not updated correctly for interfaces configured in initramfs

Bug #1432829 reported by Scott Moser
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
open-iscsi (Ubuntu)
Fix Released
High
Martin Pitt
systemd (Ubuntu)
Invalid
High
Unassigned

Bug Description

maas images utilize cloud-initramfs-dyn-netconf . The way this works is basically:
 * /etc/network/interfaces in image is a link to ../../run/network/dynamic-interfaces
 * kernel command line 'ip=' convince the initramfs to bring up networking using 'ipconfig'
   example: ip=::::maas-enlist:BOOTIF
 * ipconfig writes files in /run/net-*.conf for each interface it configures.
 * cloud-initramfs-dyn-netconf module writes /run/network/dyanmic-interfaces based on /run/net-*.conf files that 'ipconfig' creates.

end result is that after the move to real root, /etc/network/interfaces should be a symlink to /run/ that ends up having something like this:

 | ## This file is generated by cloud-initramfs-dyn-netconf
 | auto lo
 | iface lo inet loopback
 | manual eth0
 | iface eth0 inet dhcp
 | dns-nameservers 192.168.64.3
 | dns-search maas

resolvconf seems not to be working as well as it should be. In vivid (now using systemd), I have only the resolvconf header in the file.

 in all other supported ubuntu releases, doing the above results in functional resolv.conf via resolv.conf. Seen here from trusty:
 | # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
 | # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
 | nameserver 192.168.64.3

Related Bugs:
 * bug 1432821: something deleting /run/network after during boot
 * bug 1463461: resolvconf not updated by /lib/systemd/system/ifup@.service.d/open-iscsi.conf
 * bug 1501033: transient error results in /etc/resolv.conf not populated in iscsi root

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: resolvconf 1.69ubuntu1.1
ProcVersionSignature: Ubuntu 3.19.0-9.9-generic 3.19.1
Uname: Linux 3.19.0-9-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
Date: Mon Mar 16 20:42:24 2015
PackageArchitecture: all
ProcEnviron:
 TERM=vt102
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: resolvconf
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Scott Moser (smoser) wrote :
description: updated
Revision history for this message
Thomas Hood (jdthood) wrote :

First a parenthetical remark. According to interfaces(5) the "manual" keyword is used exclusively in the "method" field, as in `iface eth0 inet manual`. But in your example you use it at the beginning of a line. Perhaps you think that in that context "manual" means the opposite of "auto" (non-auto?) but so far as I can tell by reading the manual page and the source code it does not have any meaning in that context. If a logical interface is not declared as "auto" then it is non-auto and there is no way affirmatively to declare a logical interface as non-auto. If I am right, please see to it that the software you are using gets fixed so that it doesn't write out meaningless "manual ..." lines. If I am not right then please let me know. :)

The submitter wrote:

 | ## This file is generated by cloud-initramfs-dyn-netconf
 | auto lo
 | iface lo inet loopback
 | manual eth0
 | iface eth0 inet dhcp
 | dns-nameservers 192.168.64.3
 | dns-search maas

Revision history for this message
Thomas Hood (jdthood) wrote :

I just tried to reproduce the bug in Ubuntu 14.10 by editing the file /etc/network/interfaces to look like the following (complete with bogus "manual" line).

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback
manual eth0
iface eth0 inet dhcp
dns-nameservers 1.2.3.4
dns-search maas

I do `ifup eth0` and get the following in /etc/resolv.conf (content actually at /run/resolvconf/resolv.conf).

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 1.2.3.4
nameserver 192.168.1.254
search maas mydomain

(The nameserver address 192.168.1.254 and search domain "mydomain" come in via DHCP.)

This is exactly what I would expect. The bug must arise from something in your system that differs from mine.

AIUI you can still boot Vivid using Upstart. If you boot using Upstart, does the bug still occur?

Revision history for this message
Scott Moser (smoser) wrote :

Thomas,
 Yeah, you're right about 'manual <devname>'. Seems unnecessary.

wrt you not reproducing it, I'm seeing the issue
a.) when using systemd as init
b.) when configuring networking from the initramfs

the interface being used is already up, and in this case cant be bounced when init runs from the root. It can't be bounced because its on iscsi.

Revision history for this message
Scott Moser (smoser) wrote :

possibly related...
likely related, I find that on this system

$ ls /run/network/ifstate
ls: cannot access /run/network/ifstate: No such file or directory
$ ifquery --state --all
Segmentation fault (core dumped)
$ sudo ifquery --state --all
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:98
          inet addr:192.168.64.112 Bcast:192.168.95.255 Mask:255.255.224.0
          inet6 addr: fe80::5054:ff:fe12:3498/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:14050 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13591 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:152616463 (152.6 MB) TX bytes:1138294 (1.1 MB)

$ ls /run/network
ls: cannot access /run/network: No such file or directory

On an upstart system, /etc/init/iscsi-network-interface.conf (http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/open-iscsi/vivid/view/head:/debian/open-iscsi.iscsi-network-interface.upstart) is what handles this.

I suspect we dont have such in a systemd system.

Revision history for this message
Scott Moser (smoser) wrote :

Tomorrow I'll look into adding code in systemd package that would basically do what the open-iscsi job did previously.

Revision history for this message
Scott Moser (smoser) wrote :

I've made this affect systemd as that is where I have the fix for it.
I could fix in open-iscsi as it was in upstart, but I think adding general infrastructure that considers open-iscsi is more sane than open-iscsi playing around with tricking ifupdown.

affects: resolvconf (Ubuntu) → systemd (Ubuntu)
Changed in systemd (Ubuntu):
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Scott Moser (smoser) wrote :
tags: added: patch
Revision history for this message
Scott Moser (smoser) wrote :
Download full text (6.8 KiB)

-- adt --
  Note, my adt runs so far have failed each time (and after 30+ minutes)
  The last last failure due to guest running out of space, so changed to 20G
  (default 4)

  # per http://packaging.ubuntu.com/html/auto-pkg-test.html#executing-the-test
  sudo addgroup kvm
  sudo adduser $(id --user --name) kvm
  sudo apt-get install -qy autopkgtest qemu-system qemu-utils python-distro-info genisoimage

  adt-buildvm-ubuntu-cloud -v --disk-size 20G --release=vivid
  adt-run systemd_219-4ubuntu7~ppa2.dsc --- qemu ~/adt-vivid-amd64-cloud.img
  time adt-run --output-dir=test-output.d \
     --summary=test-output.d/sm-summary.log \
      --debug systemd_219-4ubuntu7~ppa2.dsc --- \
      qemu --cpus=4 --ram-size=8192 \
      ~/adt-vivid-amd64-cloud.img 2>&1 | tee run.log

-- physical server system booted (cloud-init installed) --
  system has 2 nics, both could dhcp. only one configured to auto.
  system booted fine.
  both 'systemctl status ifup@eth0' and 'ifup@eth1' show 'success'

  EN had:
     auto eth0
     iface eth0 inet dhcp
     iface eth1 inet dhcp

-- cloud image on openstack --
  - boot system, rm -Rf /var/lib/cloud /var/log/cloud*,
    apt-add-repository ppa:smoser -y && apt-get update && apt-get dist-upgrade
    sudo reboot
    system booted fine
    systemctl status ifup@eth0

-- laptop booting test --
- boot system with eth0 connected, but not listed in /etc/network/interfaces
  - [P] network-manager should bring up network interface on login
  - [P] ifup@eth0 service should succeed but do nothing

  $ systemctl status ifup@eth0
  ● <email address hidden> - ifup for eth0
     Loaded: loaded (/lib/systemd/system/ifup@.service; static; vendor preset: enabled)
     Active: active (exited) since Thu 2015-03-19 09:41:21 EDT; 3min 26s ago
    Process: 557 ExecStart=/lib/systemd/systemd-ifupdown-helper start %I (code=exited, status=0/SUCCESS)
   Main PID: 557 (code=exited, status=0/SUCCESS)
     CGroup: /<email address hidden>

  Mar 19 09:41:21 brickies systemd[1]: Starting ifup for eth0...
  Mar 19 09:41:21 brickies systemd[1]: Started ifup for eth0.

- boot system with 'iface eth0 inet dhcp' in ENI but not auto
  - [P] service should show success but no network brought up
  - [P] network-manager should not bring up network:

  $ systemctl status ifup@eth0
  ● <email address hidden> - ifup for eth0
     Loaded: loaded (/lib/systemd/system/ifup@.service; static; vendor preset: enabled)
     Active: active (exited) since Thu 2015-03-19 10:01:16 EDT; 1min 6s ago
    Process: 531 ExecStart=/lib/systemd/systemd-ifupdown-helper start %I (code=exited, status=0/SUCCESS)
   Main PID: 531 (code=exited, status=0/SUCCESS)
     CGroup: /<email address hidden>

  Mar 19 10:01:16 brickies systemd[1]: Starting ifup for eth0...
  Mar 19 10:01:16 brickies systemd[1]: Started ifup for eth0.

- boot system with 'iface eth0 inet dhcp' in ENI + 'auto eth0'
  - [P] service should show success and network brought up
  - [P] network-manager should not bring up network:

  $ systemctl status ifup@eth0
  ● <email address hidden> - ifup for eth0
     Loaded: loaded (/lib/systemd/system/ifup@.service; static; vendor pres...

Read more...

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 219-4ubuntu7

---------------
systemd (219-4ubuntu7) vivid; urgency=medium

  * use systemd-ifupdown-helper script to handle ifup and ifdown
    of auto or hotplug devices. This allows iscsi root volumes
    to work again. (LP: #1432829)
  * debian/control/tests: disable running of upstart tests until
    upstart packaging bug 1422681 is resolved.
 -- Scott Moser <email address hidden> Thu, 19 Mar 2015 11:43:14 -0400

Changed in systemd (Ubuntu):
status: Confirmed → Fix Released
Thomas Hood (jdthood)
summary: - resolvconf not updated correctly for interfaces configured in initramfs
+ resolv.conf not updated correctly for interfaces configured in initramfs
Revision history for this message
Martin Pitt (pitti) wrote :

Thanks Scott for fixing this! However, this is awkward to have in systemd: it meddles with ifupdown's and open-iscsi's internals and has open-iscsi specific knowledge which really belongs into open-iscsi itself. I. e. open-iscsi should ship the counterpart of debian/open-iscsi.iscsi-network-interface.upstart. Adding a task for this.

Changed in open-iscsi (Ubuntu):
status: New → Triaged
Revision history for this message
Martin Pitt (pitti) wrote :

I uploaded a new open-iscsi which uses the same logic for short-circuiting ifupdown for the upstart job and a new systemd ifup@.service drop-in. With that we keep the logic in one place.

Changed in open-iscsi (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
importance: Undecided → High
status: Triaged → Fix Committed
Revision history for this message
Martin Pitt (pitti) wrote :

I'll revert the systemd 219-4ubuntu7 changes on the next upload.

Changed in systemd (Ubuntu):
status: Fix Released → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-iscsi - 2.0.873-3ubuntu11

---------------
open-iscsi (2.0.873-3ubuntu11) vivid; urgency=medium

  * Factor out logic from debian/open-iscsi.iscsi-network-interface.upstart
    into debian/net-interface-handler, and install the latter in debian/rules.
  * Add debian/ifup@.service.conf: Drop-in for ifup@.service to run
    net-interface-handler under systemd. (LP: #1432829)
 -- Martin Pitt <email address hidden> Fri, 27 Mar 2015 09:51:05 +0100

Changed in open-iscsi (Ubuntu):
status: Fix Committed → Fix Released
Scott Moser (smoser)
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.