networking.service races with kernel interface detection, ifup@.service is not being triggered

I updated the package description to what you (most probably) mean. I need to check with the Debian systemd maintainers whether hotplug should additionally be there, too.

I have some trouble reproducing this. I installed systemd-sysv into a minimal VM without NetworkManager, and /etc/network/interfaces.d/eth0.cfg has

  # The primary network interface
  auth eth0
  iface eth0 inet dhcp

Indeed "ifquery -l --allow=hotplug" doesn't mention any interface, but eth0 comes up just fine anyway. When I comment out above file, it does not come up, so that file and ifup is definitively being used here.

But running "sudo ifup --verbose --allow=hotplug eth0" manually confirms that the stanza is being ignored. So the most probable explanation is that /etc/init.d/networking is being run during boot which takes care of the "auto" stanzas? Why isn't that being run for you? Can you please show "systemctl status networking"?

$ systemctl status networking

networking.service - LSB: Raise network interfaces.
   Loaded: loaded (/etc/init.d/networking)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: active (exited) since Mon 2014-09-29 09:22:44 UTC; 1min 51s ago
  Process: 332 ExecStart=/etc/init.d/networking start (code=exited, status=0/SUCCESS)

Right, I get the same (on both current utopic and your Ubuntu Core environment). So /etc/init.d/networking exists and is being run -- what's missing? What does "ip a" show? It gets a proper DHCP assignment and routing here.

ifup@.service works as intended. The idea is that /etc/init.d/networking brings up all "auto" interfaces, and that unit takes care of the hotplugged interfaces only.

The problem is that /etc/init.d/networking (or something it calls) doesn't work if / is read-only. I'll investigate what it's trying to do.

Further debugging: /etc/init.d/networking just gets redirected through /lib/lsb/init-functions.d/40-systemd. This gets closer to the problem:

networking.service - LSB: Raise network interfaces.
   Loaded: loaded (/etc/init.d/networking)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: active (exited) since Mon 2014-09-29 13:15:31 UTC; 41s ago
  Process: 405 ExecStart=/etc/init.d/networking start (code=exited, status=0/SUCCESS)

Sep 29 13:15:31 localhost.localdomain dhclient[486]: For info, please visit https://www.isc.org/software/dhcp/
Sep 29 13:15:31 localhost.localdomain networking[405]: For info, please visit https://www.isc.org/software/dhcp/
Sep 29 13:15:31 localhost.localdomain dhclient[486]:
Sep 29 13:15:31 localhost.localdomain networking[405]: Cannot find device "eth0"
Sep 29 13:15:31 localhost.localdomain networking[405]: Error getting hardware address for "eth0": No such device
Sep 29 13:15:31 localhost.localdomain networking[405]: Failed to bring up eth0.
Sep 29 13:15:31 localhost.localdomain networking[405]: ...done.
Sep 29 13:15:31 localhost.localdomain systemd[1]: Started LSB: Raise network interfaces..
Sep 29 13:15:31 localhost.localdomain ntpdate[566]: Can't find host ntp.ubuntu.com: Name or service not known (-2)
Sep 29 13:15:31 localhost.localdomain ntpdate[566]: no servers can be used, exiting

Running "sudo systemctl start networking.service" manually works, so somehow this is running at a time when there's no eth0 yet. This also explains why adding --allow=auto to the job works, as then it gets re-run when eth0 is detected.

So for some reason your system configuration causes /etc/init.d/networking to be run before the kernel detects eth0 (journalctl shows this quite nicely).

I talked to Michael Biebl, some notes:

 * I suppose due to keeping the root fs readonly the entire boot is a tad faster and thus exposes the race.
 * Due to that race it is recommended that physical interfaces use 'allow-hotplug', and virtual interfaces (bridges etc) use 'auto'.
 * debian-installer uses allow-hotplug for writing /etc/network/interfaces, to cope with slow detection on the kernel side

However, it also doesn't work with allow-hotplug, I'll investigate further.

In my test system, ifup@.service actually never gets called, which is why the hotplugging fails. (I. e. changing to --allow=auto doesn't work either). "sudo systemctl start <email address hidden>" works (you then see that it moves to active(exited) and ran the commands), but the BindsTo=sys-subsystem-net-devices-%i.device does not seem to work.

OK, I know what's wrong with ifup@.service. I'll fix it to actually get triggered.

James: To avoid the race between the kernel and /etc/init.d/networking, is it possible to fix your image scripts to mark physical interfaces (eth0) as "allow-hotplug" instead of "auto"? That would mirror what d-i does.

Uploaded fix.

This bug was fixed in the package systemd - 208-8ubuntu5

---------------
systemd (208-8ubuntu5) utopic; urgency=medium

  * Extend systemd's "net" device udev rule to trigger ifup@.service on
    network devices. In Debian this is done through net.agent, but we do not
    install this in Ubuntu. (LP: #1374521)
  * debian/ifup@.service: Additionally run for "auto" class. We don't really
    support "allow-hotplug" in Ubuntu at the moment, so we need to deal with
    "auto" devices appearing after "/etc/init.d/networking start" already ran.
    (LP: #1374521)
  * Pull latest keymaps from upstream trunk.
 -- Martin Pitt <email address hidden> Tue, 30 Sep 2014 07:55:12 +0200