[mako] after transitioning to Attached = 0; ofono never transitions back to Attached = 1

Bug #1234434 reported by Mathieu Trudel-Lapierre
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
network-manager (Ubuntu)
Fix Released
High
Mathieu Trudel-Lapierre
ofono (Ubuntu)
Fix Released
High
Tony Espy

Bug Description

This is the relevant excerpt from my syslog:

Oct 2 22:10:31 ubuntu-phablet kernel: [ 862.705020] adjust_soc: ibat_ua = -220300, vbat_uv = 4375587, soc = 96, batt_temp=306
Oct 2 22:10:43 ubuntu-phablet NetworkManager[1087]: <info> (/ril_0): device state change: activated -> disconnected (reason 'user-requested') [100 30 39]
Oct 2 22:10:43 ubuntu-phablet NetworkManager[1087]: <info> (/ril_0): deactivating device (reason 'user-requested') [39]
Oct 2 22:10:43 ubuntu-phablet NetworkManager[1087]: <warn> DNS: plugin dnsmasq update failed
Oct 2 22:10:43 ubuntu-phablet dnsmasq[2077]: setting upstream servers from DBus
Oct 2 22:10:43 ubuntu-phablet NetworkManager[1087]: <info> Removing DNS information from /sbin/resolvconf
Oct 2 22:10:43 ubuntu-phablet NetworkManager[1087]: <info> WWAN now disabled by management service
Oct 2 22:10:43 ubuntu-phablet dbus[653]: [system] Activating service name='org.freedesktop.nm_dispatcher' (using servicehelper)
Oct 2 22:10:43 ubuntu-phablet whoopsie[1495]: offline
Oct 2 22:10:43 ubuntu-phablet NetworkManager[1087]: <info> (/ril_0) marked enabled: 0
Oct 2 22:10:43 ubuntu-phablet dbus[653]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Oct 2 22:10:51 ubuntu-phablet kernel: [ 882.695192] adjust_soc: ibat_ua = -216800, vbat_uv = 4375587, soc = 96, batt_temp=307
Oct 2 22:11:03 ubuntu-phablet NetworkManager[1087]: <warn> (/ril_0) failed to disconnect modem: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Oct 2 22:11:11 ubuntu-phablet kernel: [ 902.685151] adjust_soc: ibat_ua = -214500, vbat_uv = 4375293, soc = 96, batt_temp=308
Oct 2 22:11:31 ubuntu-phablet kernel: [ 922.675080] adjust_soc: ibat_ua = -213400, vbat_uv = 4376176, soc = 96, batt_temp=308
Oct 2 22:11:45 ubuntu-phablet ofonod[915]: ril_deactivate_data_call_cb: replay failure: GENERIC_FAILURE
Oct 2 22:11:51 ubuntu-phablet kernel: [ 942.665160] adjust_soc: ibat_ua = -208900, vbat_uv = 4376176, soc = 96, batt_temp=308
Oct 2 22:12:11 ubuntu-phablet kernel: [ 962.655150] adjust_soc: ibat_ua = -205400, vbat_uv = 4376766, soc = 96, batt_temp=308
Oct 2 22:12:31 ubuntu-phablet kernel: [ 982.645170] adjust_soc: ibat_ua = -195800, vbat_uv = 4374701, soc = 96, batt_temp=308

This test was achieved by wrapping a Nexus 4 in tinfoil to block radio signals. When tinfoil is added; the connection properly drops and is disconnected by NetworkManager (note there is one message being sent that ofono never seems to respond to). Ofono successfully transitions to Attached = 0; Status = searching, etc. to indicate the GPRS connection was severed.

When tinfoil is removed, oFono eventually moves back to Status = registered, but Attached never gets set back to 1. Because of this, NetworkManager cannot activate a data session. The context still appears to be active throughout, though.

[ /ril_0 ]
    Features = sms net gprs sim
    Emergency = 0
    Powered = 1
    Lockdown = 0
    Interfaces = org.ofono.CallVolume org.ofono.MessageManager org.ofono.NetworkRegistration org.ofono.VoiceCallManager org.ofono.ConnectionManager org.ofono.NetworkTime org.ofono.SimManager
    Online = 1
    Model = Fake Modem Model
    Revision = M9615A-CEFWMAZM-2.0.1700.84
    Type = hardware
    Serial = <omitted>
    Manufacturer = Fake Manufacturer
    [ org.ofono.CallVolume ]
        Muted = 1
        SpeakerVolume = 0
        MicrophoneVolume = 0
    [ org.ofono.MessageManager ]
    [ org.ofono.NetworkRegistration ]
        Status = registered
        Strength = 22
        Name = Bell
        LocationAreaCode = <omitted>
        Mode = auto
        MobileCountryCode = 302
        Technology = umts
        CellId = <omitted>
        MobileNetworkCode = 610
    [ org.ofono.VoiceCallManager ]
        EmergencyNumbers = 08 000 999 110 112 911 118 119
    [ org.ofono.ConnectionManager ]
        Attached = 0
        RoamingAllowed = 0
        Powered = 1
    [ org.ofono.NetworkTime ]
    [ org.ofono.SimManager ]
        Retries =
        MobileCountryCode = 302
        FixedDialing = 0
        SubscriberNumbers = <omitted>
        PreferredLanguages = en fr
        BarredDialing = 0
        ServiceNumbers = [Client Care - Int'l] = '+15144207748' [Client Care - Canada] = '*611' [#TAXI] = '#8294' [Directory Assistance] = '411' [Client Care - US] = '18006677626'
        CardIdentifier = <omitted>
        LockedPins =
        MobileNetworkCode = 610
        SubscriberIdentity = <omitted>
        Present = 1
        PinRequired = none

[ /ril_0 ]
    [ /ril_0/context1 ]
        Username =
        Protocol = ip
        Name = Internet
        Settings = { DomainNameServers=206.47.201.246,207.231.231.254, Method=static Netmask=255.255.255.252 Address=10.110.xx.xx Interface=rmnet_usb0 Gateway=10.110.xx.xx }
        IPv6.Settings = { }
        Active = 1
        AccessPointName = inet.bell.ca
        Password =
        Type = internet

Related branches

Tony Espy (awe)
Changed in ofono (Ubuntu):
assignee: nobody → Tony Espy (awe)
Revision history for this message
Dave Morley (davmor2) wrote :
Revision history for this message
Dave Morley (davmor2) wrote :

root@ubuntu-phablet:/# /usr/share/ofono/scripts/list-modems
[ /ril_0 ]
    Features = sms net gprs sim
    Emergency = 0
    Powered = 1
    Lockdown = 0
    Interfaces = org.ofono.CallVolume org.ofono.MessageManager org.ofono.NetworkRegistration org.ofono.ConnectionManager org.ofono.NetworkTime org.ofono.SimManager org.ofono.VoiceCallManager
    Online = 1
    Model = Fake Modem Model
    Revision = I9250XXLJ1
    Type = hardware
    Serial = 351554052322998
    Manufacturer = Fake Manufacturer
    [ org.ofono.CallVolume ]
        Muted = 1
        SpeakerVolume = 0
        MicrophoneVolume = 0
    [ org.ofono.MessageManager ]
    [ org.ofono.NetworkRegistration ]
        Status = registered
        Strength = 16
        Name = T-Mobile Orange
        LocationAreaCode = 3218
        Mode = auto
        MobileCountryCode = 234
        Technology = umts
        CellId = 210926480
        MobileNetworkCode = 33
    [ org.ofono.ConnectionManager ]
        Attached = 0
        RoamingAllowed = 0
        Powered = 1
    [ org.ofono.NetworkTime ]
    [ org.ofono.SimManager ]
        Retries =
        MobileCountryCode = 234
        FixedDialing = 0
        SubscriberNumbers = 07572378845
        PreferredLanguages = en
        BarredDialing = 0
        CardIdentifier = 8944302031557237537
        LockedPins =
        MobileNetworkCode = 30
        SubscriberIdentity = 234308342877614
        Present = 1
        PinRequired = none
    [ org.ofono.VoiceCallManager ]
        EmergencyNumbers = 112 999 911

root@ubuntu-phablet:/# /usr/share/ofono/scripts/list-contexts
[ /ril_0 ]
    [ /ril_0/context1 ]
        Username = User
        Protocol = ip
        Name = Internet
        Settings = { DomainNameServers=149.254.230.7,149.254.192.126, Method=static Netmask=255.255.255.0 Address=10.151.149.138 Interface=rmnet0 Gateway=10.151.149.1 }
        IPv6.Settings = { }
        Active = 1
        AccessPointName = general.t-mobile.uk
        Password = mms
        Type = internet

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ofono (Ubuntu):
status: New → Confirmed
Revision history for this message
Tony Espy (awe) wrote :

Damn janitor beat me too.

"Confirmed" on build #78 running on mako and using Mathieu's tin-foil technique:

    [ org.ofono.ConnectionManager ]
        Attached = 0
        RoamingAllowed = 0
        Powered = 1
    [ org.ofono.NetworkTime ]
    [ org.ofono.SimManager ]
        Retries =
        MobileCountryCode = 310
        FixedDialing = 0
        SubscriberNumbers =
        PreferredLanguages = en
        BarredDialing = 0
        CardIdentifier = 89014104243154141218
        LockedPins =
        MobileNetworkCode = 410
        SubscriberIdentity = 310410315414121
        Present = 1
        PinRequired = none

# ./list-contexts
[ /ril_0 ]
    [ /ril_0/context1 ]
        Username =
        Protocol = ip
        Name = MEdia Net (phones)
        Settings = { DomainNameServers=172.16.145.103,172.16.145.103, Method=static Netmask=255.255.255.248 Address=10.177.179.187 Interface=rmnet_usb0 Gateway=10.177.179.185 }
        IPv6.Settings = { }
        Active = 1
        AccessPointName = wap.cingular
        Password =
        Type = internet

# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
# nmcli c
NAME UUID TYPE TIMESTAMP-REAL
/310410315414121/context1 395588c4-584f-4d8c-e6fa-41c75a326c7d gsm Thu Oct 3 21:17:49 2013
#

Revision history for this message
Tony Espy (awe) wrote :

Just side note, my first two attempts I wrapped the phone and put it in the oven, and each time I removed it, the phone was off. I'm guessing this was due to critical temp being reached ( note, the oven was off... )? I'll check the logs.

Revision history for this message
Tony Espy (awe) wrote :

I tested the same scenario with my maguro this morning, and cannot reproduce the same behavior. GPRS seems to re-attach just fine.

I will re-test with tracing enabled to see if I can spot the differences between the two devices.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Seems to me like toggling Mobile data Enable/Disable triggers this bug -- when it gets disabled for more than a few seconds, here my mako does not get back to Attached.

Revision history for this message
Tony Espy (awe) wrote :

Just confirmed with build #87 that this bug is confined to mako. I'm able to enable/disable mobile data and pass the tinfoil test using maguro.

summary: - after transitioning to Attached = 0; ofono never transitions back to
- Attached = 1
+ [mako] after transitioning to Attached = 0; ofono never transitions back
+ to Attached = 1
Revision history for this message
Dave Morley (davmor2) wrote :

Tony disabling via setting and re-enabling fails to reconnect for me on maguro :) Maybe you got the one with working 3g ;)

Revision history for this message
Dave Morley (davmor2) wrote :

root@ubuntu-phablet:/# /usr/share/ofono/scripts/list-contexts
[ /ril_0 ]
    [ /ril_0/context1 ]
        Username = User
        Protocol = ip
        Name = Internet
        Settings = { }
        IPv6.Settings = { }
        Active = 0
        AccessPointName = general.t-mobile.uk
        Password = mms
        Type = internet

root@ubuntu-phablet:/# /usr/share/ofono/scripts/list-modems
[ /ril_0 ]
    Features = sms net gprs sim
    Emergency = 0
    Powered = 1
    Lockdown = 0
    Interfaces = org.ofono.CallVolume org.ofono.MessageManager org.ofono.NetworkRegistration org.ofono.ConnectionManager org.ofono.NetworkTime org.ofono.VoiceCallManager org.ofono.SimManager
    Online = 1
    Model = Fake Modem Model
    Revision = I9250XXLJ1
    Type = hardware
    Serial = 351554052322998
    Manufacturer = Fake Manufacturer
    [ org.ofono.CallVolume ]
        Muted = 1
        SpeakerVolume = 0
        MicrophoneVolume = 0
    [ org.ofono.MessageManager ]
    [ org.ofono.NetworkRegistration ]
        Status = registered
        Strength = 22
        Name = T-Mobile Orange
        LocationAreaCode = 3218
        Mode = auto
        MobileCountryCode = 234
        Technology = umts
        CellId = 210959600
        MobileNetworkCode = 33
    [ org.ofono.ConnectionManager ]
        Attached = 1
        RoamingAllowed = 0
        Suspended = 0
        Powered = 1
    [ org.ofono.NetworkTime ]
    [ org.ofono.VoiceCallManager ]
        EmergencyNumbers = 112 999 911
    [ org.ofono.SimManager ]
        Retries =
        MobileCountryCode = 234
        FixedDialing = 0
        SubscriberNumbers = 07572378845
        PreferredLanguages = en
        BarredDialing = 0
        CardIdentifier = 8944302031557237537
        LockedPins =
        MobileNetworkCode = 30
        SubscriberIdentity = 234308342877614
        Present = 1
        PinRequired = none

Revision history for this message
Tony Espy (awe) wrote :

Confirmed the bug again using both scenarios on mako with build #87.

Revision history for this message
Tony Espy (awe) wrote :

I've been working on a possible fix which so far looks promising on mako ( both scenarios now work ). Tested in conjunction with the new minor version of NetworkManager which is in the queue for upload.

I still need to test on maguro, and also in conjunction with the telepathy-ofono netreg fix ( ie. remove the code in telepathy-ofono which calls the Register method; see bug #1226145 for more details ).

Revision history for this message
Tony Espy (awe) wrote :

So, while my potential fix makes things better, it still occasionally fails to restore the attach state. Still more work to do...

Tony Espy (awe)
Changed in ofono (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Tony Espy (awe) wrote :

OK, some good progress to report. I can disable/enable mobile data multiple times in a row on mako and everything works as expected. Likewise with the tin-foil/oven scenario; in fact, the data connection is usually up & running before I can unwrap the phone fully and turn on the screen.

Turns out the problem was that when 'Attach' was toggled either via DBus or via connection loss, the primary data context wasn't being properly cleaned up, so when the re-attach happened, an active context was found, so no action was taken. This was due to
ril_deactivate_data_call_cb() calling set_context_disaconnected(gcd), before it called ofono_gprs_context_deactivated(), which takes gcd->active_ctx_cid as a parameter. As set_context_disconnected() sets active_ctx_cid to -1, the correct context is never cleared in the core gprs code.

I'll work on cleaning up the patch and create a pull request for the fix.

Revision history for this message
Tony Espy (awe) wrote :

Here's test package with the fix.

Revision history for this message
Tony Espy (awe) wrote :

...and the -dbg package.

Revision history for this message
Tony Espy (awe) wrote :

Here's the corresponding branch on github:

https://github.com/tonyespy/ofono/tree/gprs-lp1234434

I'll create a pull request tomorrow, as I have more testing to do on maguro.

Any feedback on the test package would be most appreciated as well!

Changed in ofono (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Dave Morley (davmor2) wrote :

Tony.

Maguro still isn't recovering from switching off 3g in setting and switching it back on again.

Other than that it's run all morning with no obvious issues for me

Revision history for this message
Tony Espy (awe) wrote :

@Dave

What build are you running? Also can you double-check that you installed the right package? I'll also double-check that I uploaded the right package as well.

I just installed on a fresh instance of build #87 ( non-MIR ) and I'm able to disable / re-enable Cellular Data with no issues after more then 10 attempts.

Also, as mentioned above, maguro seemed to work OK prior to my fix. Did it work for you before you installed the package?

Once you've verified everything, would you mind attaching your syslog if it's still not working for you?

Revision history for this message
Tony Espy (awe) wrote :

Just re-installed the package attached to the bug, and also installed Mathieu's NM test packages ( new minor version of NM ).

No issues at all disabling and re-enabling cellular data on maguro... works like a charm for me. I will try the tin-foil test next.

Revision history for this message
Dave Morley (davmor2) wrote :

I'm running a fresh build of 91 (mir enabled by default)

root@ubuntu-phablet:/# dpkg -l | grep ofono
ii libofono-qt1 1.5+git20120419+bcf0c04-0ubuntu1 armhf Qt oFono library
ii ofono 1.12+bzr-0ubuntu1~awe2 armhf Mobile telephony stack (daemon)
ii ofono-dbg 1.12+bzr-0ubuntu1~awe2 armhf Mobile telephony stack (debugging files)
ii ofono-scripts 1.12+bzr6836-0ubuntu1 all Mobile telephony stack (test and maintenance script files)
ii telepathy-ofono 0.2+13.10.20131008.2-0ubuntu1 armhf telepathy oFono connection manager

That's the right packages yes?

In the meantime I'll reflash is cyphermox's nm update in the usual place? I'll install everything fresh, just do the cellular up down test and then add my syslog, if it fails to reconnect, so it is fairly compact for you currently it is huge lots of different tests this morning.

Revision history for this message
Dave Morley (davmor2) wrote :

davmor2@boromir:~$ adb shell /usr/share/ofono/scripts/list-modems
[ /ril_0 ]
    Features = sms net gprs sim
    Emergency = 0
    Powered = 1
    Lockdown = 0
    Interfaces = org.ofono.CallVolume org.ofono.MessageManager org.ofono.NetworkRegistration org.ofono.ConnectionManager org.ofono.NetworkTime org.ofono.VoiceCallManager org.ofono.SimManager
    Online = 1
    Model = Fake Modem Model
    Revision = I9250XXLJ1
    Type = hardware
    Serial = 351554052322998
    Manufacturer = Fake Manufacturer
    [ org.ofono.CallVolume ]
        Muted = 1
        SpeakerVolume = 0
        MicrophoneVolume = 0
    [ org.ofono.MessageManager ]
    [ org.ofono.NetworkRegistration ]
        Status = registered
        Strength = 16
        Name = T-Mobile Orange
        LocationAreaCode = 3218
        Mode = auto
        MobileCountryCode = 234
        Technology = umts
        CellId = 210924067
        MobileNetworkCode = 33
    [ org.ofono.ConnectionManager ]
        Attached = 1
        RoamingAllowed = 0
        Suspended = 0
        Powered = 1
    [ org.ofono.NetworkTime ]
    [ org.ofono.VoiceCallManager ]
        EmergencyNumbers = 112 999 911
    [ org.ofono.SimManager ]
        Retries =
        MobileCountryCode = 234
        FixedDialing = 0
        SubscriberNumbers = 07572378845
        PreferredLanguages = en
        BarredDialing = 0
        CardIdentifier = 8944302031557237537
        LockedPins =
        MobileNetworkCode = 30
        SubscriberIdentity = 234308342877614
        Present = 1
        PinRequired = none

davmor2@boromir:~$ adb shell /usr/share/ofono/scripts/list-contexts
[ /ril_0 ]
    [ /ril_0/context1 ]
        Username = User
        Protocol = ip
        Name = Internet
        Settings = { }
        IPv6.Settings = { }
        Active = 0
        AccessPointName = general.t-mobile.uk
        Password = mms
        Type = internet

Changed in network-manager (Ubuntu):
status: New → In Progress
importance: Undecided → High
Changed in ofono (Ubuntu):
importance: Undecided → High
Changed in network-manager (Ubuntu):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ofono - 1.12+bzr6839-0ubuntu1

---------------
ofono (1.12+bzr6839-0ubuntu1) saucy; urgency=low

  [ Tony Espy ]
  * rilmodem: Fix GPRS attach/detach logic (LP: #1234434)
 -- Ricardo Salveti de Araujo <email address hidden> Wed, 16 Oct 2013 11:30:48 -0300

Changed in ofono (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 0.9.8.4-0ubuntu3

---------------
network-manager (0.9.8.4-0ubuntu3) trusty; urgency=low

  * Set "allow-stderr" for debian/tests/apparmor_workaround to avoid failure
    on AppArmor warnings.
 -- Martin Pitt <email address hidden> Tue, 05 Nov 2013 09:48:28 +0100

Changed in network-manager (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.