Wifi "device not ready" after booting into OS for the 1st time

Status changed to 'Confirmed' because the bug affects multiple users.

This affects the entire System76 product line, including:

Lemur 6, Lemur 7, Gazelle 10, Gazelle 11, Kudu 2, Kudu 3, Serval 10, Bonobo 10, Bonobo 11, Meerkat 2, Sable 6, Ratel 5, and other desktop hardware with additional wifi cards added.

To reproduce:

- Install Ubuntu in OEM mode
- Prepare for shipping to end user
- Power off system
- Power on system.
- Go through user setup
- Either join or don't join a Wifi during user setup, does not change outcome
- Log into user account
- Try to join a WiFi network

No wifi networks will be shown.

This affects Ubuntu 16.04.1 and 16.10, including the following kernels:

4.8.0-26-generic
4.8.0-22-generic
4.4.0-35-generic
4.4.0-47-generic

Looking at log messages, there may be a race condition present that a fresh boot does not have. On first boot:

[8.824974] iwlwifi 0000:03:00.0 wlp3s0: renamed from wlan0
[9.133313] IPv6: ADDRCONF(NETDEV_UP): wlp3s0: link is not ready
[9.206671] IPv6: ADDRCONF(NETDEV_UP): wlp3s0: link is not ready
[9.324861] IPv6: ADDRCONF(NETDEV_UP): wlp3s0: link is not ready

On reboot:

[3.515363] iwlwifi 0000:03:00.0 wlp3s0: renamed from wlan0
[4.386126] IPv6: ADDRCONF(NETDEV_UP): wlp3s0: link is not ready
[4.610799] IPv6: ADDRCONF(NETDEV_UP): wlp3s0: link is not ready
[4.712271] IPv6: ADDRCONF(NETDEV_UP): wlp3s0: link is not ready
[7.614299] IPv6: ADDRCONF(NETDEV_UP): wlp3s0: link becomes ready

It appears that the system gives up/times out trying to make the device ready when being booted the first time, but the process happens much faster on later reboots.

This is solved by either rebooting, or restarting network-manager, as indicated above.

Our systems contain an Intel 8260, 7260, or 3165 Wifi card, and this bug affects all 3 cards.

Experience the same on a new Macbook Air running Kubuntu 16.04

This is caused by wpa_supplicant being killed by systemctl isolate in oem-config-firstboot

A potential fix is to add IgnoreOnIsolate=true to [Unit] in wpa_supplicant.service

By the way, it is possible to trigger the bug running on your own system. All you have to do is run `sudo systemctl isolate default.target` while connected to WiFi

Bluetooth may also go down.

<pitti> rbasak: hmm, jackpot51 isn't here, but "isolate" is pretty much de
fined as "stop everything which is not part of the given target"

<pitti> so this sounds like a case of YAFIYGI to me -- isolate isn't somet
hing which users should ever want to actually use...

I suggest talking to the person who wrote the "systemctl isolate ..." line in the first place. Did that line come from Debian?

<pitti> rbasak: I guess what you said -- find out who added that isolate c
all there

Let's set this to Triaged, as Jeremy appears to have pinned down the cause so it's solely in the realm of "developers" now. The affected package may need to change as we figure out where it should go.

Jeremy: are you continuing to drive this bug?

The call was added by Mathieu in bin/oem-config-firstboot when he added systemd support, subscribing him to the bug

Yes, I am. I just need to know what is permissible - we can add IgnoreOnIsolate in a ubuntu patch but it seems like the source of the issue is in network manager - it should detect that wpa supplicant is not running and restart it dynamically.

Let's wait for Mathieu's opinion.

This seems like a good candidate for 16.04.2 if we can get it fixed in time.

It's exactly as Jeremy points out -- NM should be detecting that wpasupplicant is not running and start it -- this should already have been working by way of wpasupplicant being dbus-activated.

The isolate command is there for a good reason. We want a dramatically different environment while oem-config is running -- users should not be logged in, daemons should not be running -- as anything running may block the steps taking in oem-config, such as the removal of the oem user at the end. Once all that is done, we isolate back to default mode -- this makes it so we don't have to do a full reboot just to get a working system after oem-config has been run.

It seems to me like IgnoreOnIsolate for wpasupplicant would be the right thing to do, or to figure out why it isn't being properly started when NM tries to use it.

'systemctl isolate' is documented as being equivalent to changing runlevels in a system without systemd: that's pretty much exactly what we want after changing default.target back to what it really should be.

If this doesn't work, we'll need to come up with a vastly different way of having oem-config work.

Adding an oem-config/ubiquity task in case it turns out it needs to be fixed there.

Would it be acceptable to have a patch to `wpa` to add `IgnoreOnIsolate` for the time being? If we hold it in the Ubuntu debian/patches then we can wait for a better solution to be done upstream.

Here is how I have been working around this bug for System76's machines: https://launchpadlibrarian.net/303137969/wpa_2.4-0ubuntu6_2.4-0ubuntu7~system76~1.diff.gz

This bug was fixed in the package wpa - 2.4-0ubuntu9

---------------
wpa (2.4-0ubuntu9) zesty; urgency=medium

  * debian/patches/wpa_service_ignore-on-isolate.patch: add IgnoreOnIsolate=yes
    so that when switching "runlevels" in oem-config will not kill off wpa
    and cause wireless to be unavailable on first boot. (LP: #1576024)

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 13 Mar 2017 13:46:12 -0400

Hi, is this issue released with Ubuntu 16.04.2 LTS. I am still experiencing this one. If not what should be done exactly to fix it.

Yes, I believe this still affects 16.04. I applied the patch on machines that System76 ships, but it is not upstream. It is the same patch as was applied to 17.04.

Status changed to 'Confirmed' because the bug affects multiple users.

This affects the entire Ekimia product line also ( intel 8285 M.2 )

We will apply the patch for WPA

Shall the WPA package be updated in 16.04 ? I guess the impact is so limited that it's possible.

Hello Kristin, or anyone else affected,

Accepted wpa into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/wpa/2.4-0ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

wpasupplicant 2.4-0ubuntu6.1 amd64 fixed it for me.

My case was going from graphical.target to multi-user.target and back to graphical.target.

I see there are some nplan regressions for this upload. I looked at them briefly and I see that the nplan testsuite is in terrible shape and tends to just fail a lot, so they're probably are unrelated, but could the uploader please take a look and double-confirm? I can then release this fix to -updates.

Note that this should be rebased on 6.2 now that a security release was published without the 6.1 changes.

Any progress in seeing this fixed in Xenial?

@cyphermox, has there been any proper solution of this since then? Is the patch still needed?