Waiting for network message on bootup when all network devices are up

Status changed to 'Confirmed' because the bug affects multiple users.

Please show the output of the following commands on this system:
 ls -ld /run
 ls -ld /var/run
 ls -l /run/network/

Please also attach the current /etc/network/interfaces file from the system.

Output of commands:

$ ls -ld /run
drwxr-xr-x 21 root root 780 Mar 10 06:40 /run
$ ls -ld /var/run
lrwxrwxrwx 1 root root 4 Mar 9 22:31 /var/run -> /run
$ ls -l /run/network
total 4
-rw-r--r-- 1 root root 6 Mar 10 06:39 ifstate
-rw-r--r-- 1 root root 0 Mar 10 06:39 ifup.lo
-rw-r--r-- 1 root root 0 Mar 10 06:39 ifup.wlan0
drwxr-xr-x 2 root root 40 Mar 10 06:39 static-network-up-emitted

/etc/network/interfaces is attached

So according to the contents of /run/network, all of your static networks came up quickly and as intended, and the /run/network/static-network-up-emitted directory confirms that the 'static-network-up' event should have been generated.

This should have been sufficient to stop the 'failsafe' job and avoid any messages about waiting for the network.

One thing I know could have caused boot delays recently in precise is a bug in apparmor, bug #949891, which caused apparmor_parser to take a very long time at boot - and apparmor_parser is called before each network interface is brought up, so this could very well account for seeing the delay in cases when you *do* have interfaces other than lo configured statically in /etc/network/interfaces. But if you *don't* have any interfaces configured statically (as in your current /etc/network/interfaces), this shouldn't have mattered.

I have hit upon a possible race condition that would explain this. If the order of boot events is:

 net-device-up IFACE=lo
 static-network-up
 filesystem
 runlevel

the 'static-network-up' event will not stop the failsafe job because it's not started yet, but the 'filesystem' event will finish satisfying the start condition and the job will start, and stay running until 'runlevel' is emitted.

And the runlevel event is emitted at the bottom of the rc-sysinit job, only after processing scripts in /etc/rcS.d.

Can you show the output of 'ls -l /etc/rcS.d'?

Regardless of what this command shows, I think there's definitely one bug here in /etc/init/failsafe.conf, and possibly another in upstart itself for not resetting the status of the failsafe job when static-network-up is emitted.

The definite bug is that we're waiting for the end of /etc/init/rc-sysinit.conf to clear the message, even though nothing during rc-sysinit is related to networking. Normally rc-sysinit is fast, but when it isn't (i.e, when it takes longer than 20 seconds), this message is confusing.

Clint, I think the right thing for failsafe to do here is 'stop on static-network-up or starting rc-sysinit'. Does that sound reasonable to you? Since it's a task, we can't use 'started rc-sysinit', but 'starting' should be fine and eliminates the remaining risk of spurious messages here.

/etc/init/failsafe.conf is in the upstart package, so reassigning back :)

I agree with your position Steve. "starting rc-sysinit" is actually a more accurate representation of the intent of failsafe anyway. Once rc-sysinit has been started, we're no longer in danger of waiting forever. There is the danger that some other job that is 'start on starting rc-sysinit' then stops rc-sysinit, but that would be a bug in that job if it didn't also do something to move booting forward, so I think this is a safe path to take.

Output of ls -l /etc/rcS.d:

 $ ls -l /etc/rcS.d
total 4
-rw-r--r-- 1 root root 447 Feb 17 18:17 README
lrwxrwxrwx 1 root root 16 Mar 8 20:00 S25brltty -> ../init.d/brltty
lrwxrwxrwx 1 root root 18 Mar 8 19:53 S37apparmor -> ../init.d/apparmor
lrwxrwxrwx 1 root root 17 Mar 8 19:29 S55urandom -> ../init.d/urandom
lrwxrwxrwx 1 root root 20 Mar 8 19:37 S70x11-common -> ../init.d/x11-common

Ah, apparmor also runs in /etc/rcS.d, so that could explain the delay entirely. For now I'll assume that our change to failsafe.conf fixes this.

This bug was fixed in the package upstart - 1.4-0ubuntu9

---------------
upstart (1.4-0ubuntu9) precise; urgency=low

  [ Steve Langasek ]
  * debian/conf/failsafe.conf: instead of waiting for the 'runlevel' event
    before considering failsafe done, stop this job as soon as we're
    starting rc-sysinit; that way, any delays in /etc/rcS.d will not cause
    confusing messages about networking delays when the network is not the
    problem. (LP: #950662)

  [ James Hunt ]
  * init/log.c:log_read_watch(): Set remote_closed for scenarios where error
     handler never called. (LP: #935585)

  [ Serge Hally ]
  * debian/conf/power-status-changed.conf: shut down on getting SIGPWR.
    Unprivileged tasks can't send this signal. In particular this will
    allow clean shutdown of containers from the host.
    (See http://www.makelinux.net/man/7/P/power-status-changed)

  [ Stéphane Graber ]
  * Rename Serge's job to shutdown.conf to avoid a name conflict with the
    event power-status-changed.
 -- Stephane Graber <email address hidden> Fri, 16 Mar 2012 13:48:04 -0400

Error message has disappeared and it now takes only 10 seconds from when the Ubuntu logo first appears to login prompt.

Many thanks.

Since Ubuntu 11.04 this message appear at boot, if there are static networks in /etc/network/interfaces.
The bug (I prefer call it "bad coding") continues in Ubuntu 12.04.

I had a static firewire network in /etc/network/interfaces:

auto firewire0
iface firewire0 inet static
   address 192.168.0.6
   netmask 255.255.255.0
   network 192.168.0.0
   gateway 192.168.0.1
   broadcast 192.168.0.255

To avoid 2 minutes delay, I edit /etc/init/failsafe.conf and comment the sleeps:

    # Plymouth errors should not stop the script because we *must* reach
    # the end of this script to avoid letting the system spin forever
    # waiting on it to start.
 $PLYMOUTH message --text="Waiting for network configuration..." || :
 #sleep 40

 $PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
 #sleep 59
 $PLYMOUTH message --text="Booting system without full network configuration..." || :

Please consider reopening this bug. I have the most recent upstart for precise:

ii upstart 1.5-0ubuntu7 event-based init daemon

...and still see this issue if there's an invalid network interface specified in /etc/network/interfaces using "auto".

On Mon, May 07, 2012 at 05:45:41PM -0000, Alexander List wrote:
> Please consider reopening this bug. I have the most recent upstart for
> precise:

> ii upstart 1.5-0ubuntu7 event-based init daemon

> ...and still see this issue if there's an invalid network interface
> specified in /etc/network/interfaces using "auto".

If you have an invalid network interface marked 'auto', then you have a
broken config. Upstart can't save you from a broken config without
sacrificing reliability. You need to fix your config.

I have the latest upstart as well (1.5-0ubuntu7). I do not have any invalid network interfaces. I do have the "Waiting for network configuration...", the "Waiting up to 60 more seconds for network configuration...", and the "Booting system without full network configuration..." messages every time I boot.

I looked at syslog from my last boot and I have noticed that my ethernet interface, eth0, has its state changed a number of times before finally being activated.

I have attached the portion of the syslog that pertains to eth0.

Perhaps this will tell you more than it does me as I am not sure what the log is telling me.

This bug appeared on my brother's Samsung NC10 after an ubgrade from Kubuntu 10.04 to 12.04. I have checked the interfaces file for invalid entries, but everything was ok.

Excerpts from Uwe Mock's message of Sat May 12 11:11:02 UTC 2012:
> This bug appeared on my brother's Samsung NC10 after an ubgrade from
> Kubuntu 10.04 to 12.04. I have checked the interfaces file for invalid
> entries, but everything was ok.
>

Hi Uwe. Please open a new bug report from the affected system with
'ubuntu-bug upstart' so we can investigate.

This bug seems to be a duplicate of bug 950662.

Sorry, this is already the bug 950662.

I also experience this issue and here is my findings:
The delays are caused by failsafe boot delay (/etc/init/failsafe.conf)
which is started on:
start on filesystem and net-device-up IFACE=lo
and stopped on:
stop on static-network-up or starting rc-sysinit
rc-sysinit is started itself on static-network-up (or failsafe-boot, i.e. after all these delays).

However, inspection of /etc/init/ reveals that there is no place, where static-network-up would be emitted
$ grep static-network-up /etc/init/*
/etc/init/failsafe.conf:stop on static-network-up or starting rc-sysinit
/etc/init/failsafe.conf: # these messages. currently only static-network-up counts for that.
/etc/init/networking.conf:emits static-network-up
/etc/init/network-interface.conf:emits static-network-up
/etc/init/procps.conf:start on virtual-filesystems or static-network-up
/etc/init/rc-sysinit.conf:start on (filesystem and static-network-up) or failsafe-boot

static-network-up is emitted by /etc/network/if-up.d/upstart when all static interfaces from /etc/network/interfaces are configured.

hi.. I have the same problem on a pppoe connection.

pppoeconf adds

"auto dsl-provider"

to /etc/network/interfaces to start the pppoe connection at boot

but when there is a problem (dsl modem or provider) I have to wait every time when I reboot.

Its there no may to add NET_NO_WAIT to /etc/defaults/rcS or something to skipp this?

Same problem here, not fixed at all in 12.04

History:
I did a fresh CLI install using wired ethernet connection (eth1)
Then update& upgrade & xubuntu-desktop
I enabled WIFI (eth0)
reboot
"waiting for network configuration" message
changed in /etc/network/interfaces to eth0 as primary network adapter, eth1 as second
no possibility to go online using wifi, needed to plug in the wire, which gave me network access, but nm-applet is gone.

Again: not fixed

Best workaround sofar:
http://askubuntu.com/questions/213614/waiting-for-network-configuration-problem
and
http://askubuntu.com/questions/144743/ubuntu-server-12-04-waiting-for-network-configuration-and-network-down

edit /etc/network/interfaces

Remove (or comment out) everything, only leave:

#The loopback network interface
auto lo
iface lo inet loopback

This will bring up wifi and a signal indicator is visible.
However, sofar nm-applet does not indicate signal strength nor does it show a list of available wireless networks. I will update when I find a solution for that.

taj, I'm sorry that you're having issues. What you describe is not related to this bug. If you have configured anything in /etc/network/interfaces, then it has nothing to do with nm-applet or NetworkManager. You will need to choose, either /etc/network/interfaces, or NetworkManager. I recommend removing your wireless interfaces from /etc/network/interfaces and letting NetworkManager handle their configuration.

Anyway, if you feel there is something that the software is doing wrong, please open a new bug and explain what you expect, and what actually happens.

Dear Clint,

You are right. This is not an nm-applet issue, nm-applet not showing up was an indicator of the problem.

It is however an upstart issue that happens with a fresh install as I described in #22. I had not touched /etc/network/interfaces, its settings were generated during system install, I guess.

The "waiting for network" message on bootup problem started after I removed the ethernet cable to boot with only wifi. After that even booting with ethernet cable gave the message and waiting time, though I was able to use my network connection after being logged in. This issue seems pretty similar to the bug description. The issue lasted until I decided to remove all network interfaces from /etc/network/interfaces, except the loopback interface (#23).

Anyway, for me the #23 workaround works (and after a while it even shows available wifi networks). Maybe for others, too.

This issue is manifesting itself again in 14.10

I would like to second that this problem is active in 14.10.

System: Custom server with dual Intel Xeon E5-2630 CPU and dual ethernet ports
OS: Bare metal install of Ubuntu server 14.10 plus xfce4 (xubuntu) desktop

After the OS install, the system has consistently been hanging as reported above during the boot sequence.

Following the boot message: "Starting configure network device"

/etc/init/failsafe.conf displays these messages with the associated two minute sleeps:

Waiting for network configuration...
Waiting up to 60 more seconds for network configuration...
Booting system without full network configuration...

However, shortly after the original "Starting configure network device" message, I am able to ping both network interfaces from a remote machine. It appears that although the interfaces are configured and active, plymouth remains confused and fails to issue a successful return code. This system was automatically configuring the two interfaces as p1p1 and p2p1, so I have written an /etc/udev/rules.d/70-persistent-net.rules file to rename these back to a more respectable eth0 and eth1. However, the hanging problem preceded any of this renaming activity.

This delay is a real pain and appears to affect quite a few people. It would be good to pin down the nature of the problem and fix it once and for all. This appears to extend back to at least 2011 and can be found elsewhere under bug-ids 885596 and 881079.

Here are the particulars for my configuration:

-----------------------------------------------------------------
# cat /etc/network/interfaces
#
auto lo
    iface lo inet loopback
#
auto eth0
   iface eth0 inet static
   address 204.107.91.3
   netmask 255.255.255.0
   gateway 204.107.91.254
   dns-nameservers 199.181.164.1 199.181.164.2 68.87.69.146 68.87.85.98
#
auto eth1
    iface eth1 inet static
    address 204.107.91.4
    netmask 255.255.255.0
    gateway 204.107.91.254
    dns-nameservers 199.181.164.1 199.181.164.2 68.87.69.146 68.87.85.98

-----------------------------------------------------------------
# ifconfig -a
eth0 Link encap:Ethernet HWaddr e0:3f:49:e8:1a:31
          inet addr:204.107.91.3 Bcast:204.107.91.255 Mask:255.255.255.0
          inet6 addr: fe80::e23f:49ff:fee8:1a31/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5704 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4706 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2651217 (2.6 MB) TX bytes:707849 (707.8 KB)
          Interrupt:16 Memory:b3f00000-b3f20000

eth1 Link encap:Ethernet HWaddr e0:3f:49:e8:1a:32
          inet addr:204.107.91.4 Bcast:204.107.91.255 Mask:255.255.255.0
          inet6 addr: fe80::e23f:49ff:fee8:1a32/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1117 errors:0 dropped:0 overruns:0 frame:0
          TX packets:125 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:140077 (140.0 KB) TX bytes:14374 (14.3 KB)
          Interrupt:17 Memory:b3e00000-b3e20000

lo Link encap:Local Loo...