Panic session closed still happening in unit tests

Bug #1416006 reported by Curtis Hovey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
Critical
Menno Finlay-Smits

Bug Description

Recent ci unit test jobs have been consistently failing with "panic: Session already closed".

Revision c3bb7a6e0d43bcbba68cf2a480d3b3ae830730f4 was trying to fix one failure point.
<http://github.com/juju/juju/pull/1499>
<http://reviews.vapour.ws/r/821/>

However it seems we are still getting the panic:

<http://reports.vapour.ws/releases/2282>

<http://data.vapour.ws/juju-ci/products/version-2282/run-unit-tests-precise-i386/build-1323/consoleText>
In cmd/jujud/agent/machine_test.go:
ok github.com/juju/juju/cmd/jujud 32.778s
panic: Session already closed
...
goroutine 4990 [chan receive]:
launchpad.net/tomb.(*Tomb).Wait(0x1a104e48, 0x0, 0x0)
 /home/ubuntu/juju-core_1.23-alpha1/src/launchpad.net/tomb/tomb.go:110 +0x42
github.com/juju/juju/state/presence.(*Pinger).Stop(0x1a104e40, 0x0, 0x0)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/state/presence/presence.go:528 +0x171
github.com/juju/juju/cmd/jujud/agent.func·074(0x19e36640)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/cmd/jujud/agent/machine_test.go:170 +0x2c
github.com/juju/testing.(*CleanupSuite).callStack(0x19bac390, 0x19e36640, 0x19d1dbc0, 0x7, 0x8)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/testing/cleanup.go:41 +0x4b
github.com/juju/testing.(*CleanupSuite).TearDownTest(0x19bac390, 0x19e36640)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/testing/cleanup.go:36 +0x40
github.com/juju/testing.(*FakeHomeSuite).TearDownTest(0x19bac390, 0x19e36640)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/testing/home.go:114 +0x3e
github.com/juju/juju/testing.(*FakeJujuHomeSuite).TearDownTest(0x19bac384, 0x19e36640)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/testing/environ.go:151 +0x54
github.com/juju/juju/cmd/jujud/agent.(*commonMachineSuite).TearDownTest(0x19bac380, 0x19e36640)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/cmd/jujud/agent/machine_test.go:153 +0x3d
...

? github.com/juju/juju/feature [no test files]
panic: Session already closed

goroutine 851 [semacquire]:
sync.runtime_Syncsemacquire(0x19dc19a0)
 /usr/lib/go/src/pkg/runtime/sema.goc:257 +0xd0
sync.(*Cond).Wait(0x19dc1998)
 /usr/lib/go/src/pkg/sync/cond.go:62 +0x7a
gopkg.in/mgo%2ev2.(*mongoCluster).AcquireSocket(0x19dc1980, 0x19dc1901, 0xf8475800, 0xd, 0xf8475800, ...)
 /home/ubuntu/juju-core_1.23-alpha1/src/gopkg.in/mgo.v2/cluster.go:546 +0x65c
gopkg.in/mgo%2ev2.(*Session).acquireSocket(0x19d70750, 0x19c57501, 0x0, 0x0, 0x0)
 /home/ubuntu/juju-core_1.23-alpha1/src/gopkg.in/mgo.v2/session.go:3408 +0x243
gopkg.in/mgo%2ev2.(*Query).One(0x19dc1900, 0x0, 0x0, 0x0, 0x0)
 /home/ubuntu/juju-core_1.23-alpha1/src/gopkg.in/mgo.v2/session.go:2365 +0xa1
gopkg.in/mgo%2ev2.(*Database).Run(0x19cd1820, 0x88487a0, 0x19cd1840, 0x0, 0x0, ...)
 /home/ubuntu/juju-core_1.23-alpha1/src/gopkg.in/mgo.v2/session.go:580 +0x1a1
gopkg.in/mgo%2ev2.(*Session).Run(0x19d70750, 0x8723820, 0x19f011b0, 0x0, 0x0, ...)
 /home/ubuntu/juju-core_1.23-alpha1/src/gopkg.in/mgo.v2/session.go:1639 +0x9f
gopkg.in/mgo%2ev2.(*Session).Ping(0x19d70750, 0x19dc1980, 0xf8475800)
 /home/ubuntu/juju-core_1.23-alpha1/src/gopkg.in/mgo.v2/session.go:1668 +0x59
gopkg.in/mgo%2ev2.DialWithInfo(0x19c57480, 0x19c57480, 0x8b049b4, 0x40996c04)
 /home/ubuntu/juju-core_1.23-alpha1/src/gopkg.in/mgo.v2/session.go:391 +0x3df
github.com/juju/testing.(*MgoInstance).MustDial(0x9762460, 0x19f3c060)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/testing/mgo.go:414 +0x37
github.com/juju/testing.(*MgoInstance).Reset(0x9762460)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/testing/mgo.go:500 +0xad
github.com/juju/juju/provider/dummy.(*environState).destroy(0x19cf6960)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/provider/dummy/environs.go:332 +0x143
github.com/juju/juju/provider/dummy.Reset()
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/provider/dummy/environs.go:307 +0x100
github.com/juju/juju/juju/testing.(*JujuConnSuite).tearDownConn(0x19a3ea88, 0x19d66e60)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/juju/testing/conn.go:507 +0x426
github.com/juju/juju/juju/testing.(*JujuConnSuite).TearDownTest(0x19a3ea88, 0x19d66e60)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/juju/testing/conn.go:109 +0x33
github.com/juju/juju/featuretests.(*leadershipSuite).TearDownTest(0x19a3ea80, 0x19d66e60)
 /home/ubuntu/juju-core_1.23-alpha1/src/github.com/juju/juju/featuretests/leadership_test.go:111 +0x116
...

Martin Packman (gz)
description: updated
Changed in juju-core:
assignee: nobody → Menno Smits (menno.smits)
status: Triaged → In Progress
Revision history for this message
John Weldon (johnweldon4) wrote :

FWIW, I get something similar on a 64bit AWS image that's not too fast (m3.large)

http://paste.ubuntu.com/9942915/

Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

Merged a possible fix in bd581d0f. It's hard to tell whether it's helped because many of the jobs that were failing due to prematurely closed sessions are now failing because some part of the tests took over 20 mins.

Changed in juju-core:
status: In Progress → Fix Committed
Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

This issue appears to be fixed now but I'd like to see a few more CI runs before calling it done.

Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.23 → 1.23-beta1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.