lava_test_run (wifi-enablement) failed on TI Panda ubuntu daily test.

Looks like we found a bug. Here's the kernel dump:

[ 739.621673] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[ 739.630249] pgd = ed0a8000
[ 739.633117] [00000004] *pgd=a393c831, *pte=00000000, *ppte=00000000
[ 739.639770] Internal error: Oops: 817 [#1] PREEMPT SMP THUMB2
[ 739.645843] Modules linked in: omapdrm_pvr(O) arc4 omapdce(C) bnep rpmsg_omx wl12xx ppdev wlcore rfcomm omaprpc(C) mac80211 omap_remoteproc rpmsg_resmgr_common lp cfg80211 remoteproc parport omap_rpmsg_resmgr rpmsg_resmgr virtio_rpmsg_bus gator wlcore_sdio btwilink bluetooth
[ 739.671783] CPU: 1 Tainted: G WC O (3.4.0-2-linaro-lt-omap #2~ci+121124091337-Ubuntu)
[ 739.681091] PC is at __rmqueue+0x14c/0x2e4
[ 739.685424] LR is at rmqueue_bulk.constprop.43+0x2f/0x8a
[ 739.691040] pc : [<c009b72c>] lr : [<c009b8f3>] psr: 600701b3
[ 739.691040] sp : e9de5ce0 ip : c11c6014 fp : 00000000
[ 739.703186] r10: e9de4000 r9 : c11c6014 r8 : c0817798
[ 739.708709] r7 : 0000000a r6 : 00000004 r5 : c11c6000 r4 : 00000002
[ 739.715606] r3 : c08177b8 r2 : 00000000 r1 : c0483e30 r0 : 00100100
[ 739.722503] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA Thumb Segment user
[ 739.730316] Control: 50c5387d Table: ad0a804a DAC: 00000015
[ 739.736389]
[ 739.736389] PC: 0xc009b6ac:
[ 739.740905] b6ac 80c7f000 270a4e80 eb062234 e0b81104 23009e04 6807fb02 0850f108 58ce4640
[ 739.749633] b6cc f0002e03 f85080ad eb00c036 330405c6 45ac9005 6b03d0f3 90034680 3b0146e1
[ 739.758331] b6ec 97062e04 0514f1ac d0166303 dc042f04 d0022c01 6a5b4b6d 9804b17b 46224629
[ 739.767059] b70c ffd1f7fe 7f00f5b0 4b68da02 b11b6a5b 46214628 fdf4f7ff e8994626 4860000c
[ 739.775756] b72c 601a6053 e8894b5f f8590009 f04f3c08 f84933ff 23003c08 f8c92f0a d1073008
[ 739.784454] b74c d00b2e04 46214628 fddaf7ff e00646a1 bf142e04 f04f46a1 e0000904 210146b1
[ 739.793182] b76c 0834f1a8 09c9ea4f 801cf8cd f107fa01 9a0546b0 3a34e01c eb020849 3f010c09
[ 739.801879] b78c 6009f852 1341eb05 0014f103 615e6070 f8c39e03 f842c018 1b900009 19809e07
[ 739.810607]
[ 739.810607] LR: 0xc009b873:
[ 739.815124] b870 465a4639 47a83608 0306ebc8 f853444b 2b003c08 463dd1f1 f422466a f02353ff
[ 739.823822] b890 685a031f 605a3a01 0798681b f3d2d501 4628fe21 e8bdb009 bf008ff0 00100100
[ 739.832550] b8b0 00200200 c0483e10 c079fee8 c0819510 c0819538 4ff8e92d f85db500 f100eb04
[ 739.841247] b8d0 46050824 468a4640 46994614 b028f8dd f3d32700 e025f9c5 21004628 f7ff464a
[ 739.849975] b8f0 4606fe77 f100b300 f1bb0314 d1050f00 60536822 61846142 e0046023 60636862
[ 739.858673] b910 61826144 21006013 46302202 fc44f7ff d0032804 bf142805 20054648 0414f106
[ 739.867370] b930 61f03701 d1d74557 21004628 f00c427a 4640fa4b fc95f3d3 e8bd4638 e92d8ff8
[ 739.876098] b950 b50043f7 eb04f85d 68c34604 d0003380 6807de02 0901f04f 26004b29 463169c5
[ 739.884796] b970 fa090fbf eb03f805 46332787 687a4638 44429600 f8e3f7ff d03c2800 463869a3
[ 739.893524]
[ 739.893524] SP: 0xe9de5c60:
[ 739.898040] 5c60 e9de4000 eb20c000 eb20c000 600f0113 eb7d4800 600f0113 cd908fc8 c00b96b3
[ 739.906738] 5c80 eb20c000 c009b72c 600701b3 ffffffff e9de5ccc c046f5f9 00100100 c0483e30
[ 739.9154...

I see from the log that it times out waiting on telnet, but the kernel bug is pointing elsewhere.

Can you tell me what it's doing at that point with update-apt-xapi, in terms of where the filesystems are? Are the filesystems in heavy use on USB or SD card or both at that point?

The first backtrace seems to be secondary fallout from trying to handle the real exception after it's reached a state of brain damage.

cpu #0 is here saving wget content

[ 756.669464] [<c02570fd>] (do_raw_spin_lock+0xb5/0xd4) from [<c009b8e7>] (rmqueue_bulk.constprop.43+0x23/0x8a)
[ 756.679962] [<c009b8e7>] (rmqueue_bulk.constprop.43+0x23/0x8a) from [<c009bef3>] (get_page_from_freelist+0xd3/0x1fc)
[ 756.691101] [<c009bef3>] (get_page_from_freelist+0xd3/0x1fc) from [<c009c0cd>] (__alloc_pages_nodemask+0xb1/0x3f8)
[ 756.702056] [<c009c0cd>] (__alloc_pages_nodemask+0xb1/0x3f8) from [<c009821b>] (grab_cache_page_write_begin+0x4b/0x88)
[ 756.713378] [<c009821b>] (grab_cache_page_write_begin+0x4b/0x88) from [<c011ef25>] (ext4_da_write_begin+0xed/0x170)
[ 756.724426] [<c011ef25>] (ext4_da_write_begin+0xed/0x170) from [<c0097e37>] (generic_perform_write+0x83/0x154)

but I am not sure if that's where it originally blew chunks or if the spinlock held by what actually died just deadlocked this at that point. The other traces seem to be telling us it had a hard time going on after the data abort.

In any event if we can translate the region of activities where it dies to a reproducer, like "wget big file to USB memory stick and watch it blow up" that will be helpful.

Jassi does it make any more sense to you?

Andy : Sorry no, not with what we have so far.

Fixing this bug does not fit in our development plans. Moving forward, all LAVA bugs will be disposed/scrubbed/triaged weekly.