juju-core

worker/peergrouper: multiple test failures

Bug #1298133 reported by Dave Cheney
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Andrew Wilkins
juju-core 1.17.7

Bug Description

I suspect most can be traced back to slice or map ordering. Fails on both gc and gccgo

lucky(~) % go test launchpad.net/juju-core/worker/peergrouper

----------------------------------------------------------------------
FAIL: worker_test.go:183: workerSuite.TestAddressChange

[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "10"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "11"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "12"
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.SetHasVote 10 true" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.000 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
mustNext 0xc2085fbdd0
mustNext done 0xc2085fbdd0, ok true
[LOG] 0:00.002 INFO juju.worker.peergrouper starting worker
[LOG] 0:00.002 INFO juju.worker.peergrouper worker starting
mustNext 0xc2085fbdd0
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.StateServerInfo" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "10"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 10" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "11"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 11" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "12"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 12" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 10" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 11" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 12" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.004 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.004 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.004 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.004 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc208242140)}
[LOG] 0:00.004 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.004 DEBUG juju.worker.peergrouper maxId: 0
[LOG] 0:00.004 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.004 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.004 DEBUG juju.worker.peergrouper machine "11" is not ready (has status: false)
[LOG] 0:00.004 DEBUG juju.worker.peergrouper machine "12" is not ready (has status: false)
[LOG] 0:00.004 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.004 DEBUG juju.worker.peergrouper desired peer group members: []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e14a8), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e14a0)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e14e0), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e14b8)}}
[LOG] 0:00.004 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.004 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e14a8), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e14a0)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e14e0), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e14b8)}}
[LOG] 0:00.005 INFO juju.worker.peergrouper successfully changed replica set to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e14a8), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e14a0)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e14e0), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e14b8)}}
mustNext done 0xc2085fbdd0, ok true
mustNext 0xc2085fbdd0
[LOG] 0:00.005 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 11" -> <nil>
[LOG] 0:00.005 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.006 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.006 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.006 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.006 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.006 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.006 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc2082433b0), &peergrouper.machine{id: "11", wantsVote: true, hostPort: "0.1.99.99:9876"}:(*replicaset.Member)(0xc208243400), &peergrouper.machine{id: "12", wantsVote: true, hostPort: "0.1.2.12:1234"}:(*replicaset.Member)(0xc208243450)}
[LOG] 0:00.006 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.006 DEBUG juju.worker.peergrouper maxId: 2
[LOG] 0:00.006 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.006 DEBUG juju.worker.peergrouper machine "11" is not ready (has status: false)
[LOG] 0:00.006 DEBUG juju.worker.peergrouper machine "12" is not ready (has status: false)
[LOG] 0:00.006 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.006 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.006 DEBUG juju.worker.peergrouper desired peer group members: []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208186870), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081868b0)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.99.99:9876", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208186810), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208186850)}}
[LOG] 0:00.006 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.007 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208186870), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081868b0)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.99.99:9876", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208186810), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208186850)}}
[LOG] 0:00.007 INFO juju.worker.peergrouper successfully changed replica set to []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208186870), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081868b0)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.99.99:9876", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208186810), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208186850)}}
mustNext done 0xc2085fbdd0, ok true
worker_test.go:208:
    c.Assert(memberWatcher.Value(), jc.DeepEquals, expectMembers)
... obtained []replicaset.Member = []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208187340), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208187380)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.99.99:9876", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081873e0), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208187420)}}
... expected []replicaset.Member = []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.99.99:9876", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208187760), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208187768)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208187780), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208187788)}}
... mismatch at [0].Id: unequal; obtained 2; expected 0

[LOG] 0:00.014 ERROR juju.worker.peergrouper peergrouper loop terminated: tomb: dying

----------------------------------------------------------------------
FAIL: worker_test.go:119: workerSuite.TestSetsAndUpdatesMembers

[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "10"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "11"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "12"
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.SetHasVote 10 true" -> <nil>
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.001 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
mustNext 0xc208504990
mustNext done 0xc208504990, ok true
[LOG] 0:00.002 INFO juju.worker.peergrouper starting worker
[LOG] 0:00.002 INFO juju.worker.peergrouper worker starting
mustNext 0xc208504990
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.StateServerInfo" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "10"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 10" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "11"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 11" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "12"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 12" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 10" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 11" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 12" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.003 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.003 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc20857f770)}
[LOG] 0:00.003 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.003 DEBUG juju.worker.peergrouper maxId: 0
[LOG] 0:00.003 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.003 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.003 DEBUG juju.worker.peergrouper machine "11" is not ready (has status: false)
[LOG] 0:00.003 DEBUG juju.worker.peergrouper machine "12" is not ready (has status: false)
[LOG] 0:00.003 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.004 DEBUG juju.worker.peergrouper desired peer group members: []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2084d3258), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2084d3250)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2084d3280), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2084d3268)}}
[LOG] 0:00.004 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.004 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2084d3258), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2084d3250)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2084d3280), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2084d3268)}}
[LOG] 0:00.004 INFO juju.worker.peergrouper successfully changed replica set to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2084d3258), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2084d3250)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2084d3280), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2084d3268)}}
mustNext done 0xc208504990, ok true
updating new member status
mustNext 0xc208504990
[LOG] 0:00.009 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.009 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.009 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.010 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.010 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.010 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.010 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc2081a8a00), &peergrouper.machine{id: "11", wantsVote: true, hostPort: "0.1.2.11:1234"}:(*replicaset.Member)(0xc2081a8a50), &peergrouper.machine{id: "12", wantsVote: true, hostPort: "0.1.2.12:1234"}:(*replicaset.Member)(0xc2081a8aa0)}
[LOG] 0:00.010 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.010 DEBUG juju.worker.peergrouper maxId: 2
[LOG] 0:00.010 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.010 DEBUG juju.worker.peergrouper machine "11" is a potential voter
[LOG] 0:00.010 DEBUG juju.worker.peergrouper machine "12" is a potential voter
[LOG] 0:00.010 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.010 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.011 DEBUG juju.worker.peergrouper desired peer group members: []replicaset.Member{replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
[LOG] 0:00.011 ERROR juju.worker.peergrouper errorFor "Machine.SetHasVote 11 true" -> <nil>
[LOG] 0:00.011 ERROR juju.worker.peergrouper errorFor "Machine.SetHasVote 12 true" -> <nil>
[LOG] 0:00.011 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.011 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
[LOG] 0:00.011 INFO juju.worker.peergrouper successfully changed replica set to []replicaset.Member{replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
mustNext done 0xc208504990, ok true
worker_test.go:144:
    c.Assert(memberWatcher.Value(), jc.DeepEquals, mkMembers("0v 1v 2v"))
... obtained []replicaset.Member = []replicaset.Member{replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
... expected []replicaset.Member = []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
... mismatch at [0].Id: unequal; obtained 1; expected 0

[LOG] 0:00.017 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 11" -> <nil>
[LOG] 0:00.017 ERROR juju.worker.peergrouper peergrouper loop terminated: tomb: dying

----------------------------------------------------------------------
FAIL: worker_test.go:290: workerSuite.TestStateServersArePublished

[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "10"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "11"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "12"
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.SetHasVote 10 true" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.001 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
[LOG] 0:00.001 INFO juju.worker.peergrouper worker starting
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.StateServerInfo" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "10"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 10" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "11"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 11" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper found new machine "12"
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "State.Machine 12" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 10" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 11" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 12" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.003 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc20853a190)}
[LOG] 0:00.003 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.003 DEBUG juju.worker.peergrouper maxId: 0
[LOG] 0:00.003 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.003 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.003 DEBUG juju.worker.peergrouper machine "11" is not ready (has status: false)
[LOG] 0:00.003 DEBUG juju.worker.peergrouper machine "12" is not ready (has status: false)
[LOG] 0:00.003 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.003 DEBUG juju.worker.peergrouper desired peer group members: []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e0818), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e0810)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e0850), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e0838)}}
[LOG] 0:00.003 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.003 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e0818), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e0810)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e0850), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e0838)}}
[LOG] 0:00.003 INFO juju.worker.peergrouper successfully changed replica set to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e0818), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e0810)}, replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081e0850), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081e0838)}}
worker_test.go:305:
    c.Assert(servers, gc.DeepEquals, expectedAPIHostPorts(3))
... obtained [][]instance.HostPort = [][]instance.HostPort{[]instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.12", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}, []instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.10", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}, []instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.11", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}}
... expected [][]instance.HostPort = [][]instance.HostPort{[]instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.10", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}, []instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.11", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}, []instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.12", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}}

[LOG] 0:00.007 ERROR juju.worker.peergrouper peergrouper loop terminated: tomb: dying

----------------------------------------------------------------------
FAIL: worker_test.go:362: workerSuite.TestWorkerPublishesInstanceIds

[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "10"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "11"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "12"
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.SetHasVote 10 true" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.000 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
[LOG] 0:00.000 INFO juju.worker.peergrouper worker starting
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "State.StateServerInfo" -> <nil>
[LOG] 0:00.000 DEBUG juju.worker.peergrouper found new machine "10"
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "State.Machine 10" -> <nil>
[LOG] 0:00.000 DEBUG juju.worker.peergrouper found new machine "11"
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "State.Machine 11" -> <nil>
[LOG] 0:00.000 DEBUG juju.worker.peergrouper found new machine "12"
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "State.Machine 12" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 10" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 11" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 12" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.001 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.001 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc208131450)}
[LOG] 0:00.001 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.001 DEBUG juju.worker.peergrouper maxId: 0
[LOG] 0:00.001 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.001 DEBUG juju.worker.peergrouper machine "12" is not ready (has status: false)
[LOG] 0:00.001 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.001 DEBUG juju.worker.peergrouper machine "11" is not ready (has status: false)
[LOG] 0:00.001 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.001 DEBUG juju.worker.peergrouper desired peer group members: []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081345d0), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081345b8)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081345a8), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081345a0)}}
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.001 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081345d0), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081345b8)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081345a8), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081345a0)}}
[LOG] 0:00.002 INFO juju.worker.peergrouper successfully changed replica set to []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081345d0), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081345b8)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc2081345a8), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc2081345a0)}}
worker_test.go:382:
    c.Assert(instanceIds, jc.DeepEquals, []instance.Id{"id-10", "id-11", "id-12"})
... obtained []instance.Id = []instance.Id{"id-11", "id-12", "id-10"}
... expected []instance.Id = []instance.Id{"id-10", "id-11", "id-12"}
... mismatch at [0]: unequal; obtained "id-11"; expected "id-10"

[LOG] 0:00.008 ERROR juju.worker.peergrouper peergrouper loop terminated: tomb: dying

----------------------------------------------------------------------
FAIL: worker_test.go:324: workerSuite.TestWorkerRetriesOnPublishError

[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "10"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "11"
[LOG] 0:00.000 INFO juju.worker.peergrouper fakeState.addMachine "12"
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Machine.SetHasVote 10 true" -> <nil>
[LOG] 0:00.000 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.000 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}}
[LOG] 0:00.001 INFO juju.worker.peergrouper worker starting
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "State.StateServerInfo" -> <nil>
[LOG] 0:00.001 DEBUG juju.worker.peergrouper found new machine "10"
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "State.Machine 10" -> <nil>
[LOG] 0:00.001 DEBUG juju.worker.peergrouper found new machine "11"
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "State.Machine 11" -> <nil>
[LOG] 0:00.001 DEBUG juju.worker.peergrouper found new machine "12"
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "State.Machine 12" -> <nil>
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 10" -> <nil>
[LOG] 0:00.001 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 11" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.Refresh 12" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper cannot publish API server addresses: publish error
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.002 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.002 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc208081040)}
[LOG] 0:00.002 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.002 DEBUG juju.worker.peergrouper maxId: 0
[LOG] 0:00.002 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.002 DEBUG juju.worker.peergrouper machine "12" is not ready (has status: false)
[LOG] 0:00.002 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.002 DEBUG juju.worker.peergrouper machine "11" is not ready (has status: false)
[LOG] 0:00.002 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.002 DEBUG juju.worker.peergrouper desired peer group members: []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208255570), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208255558)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208255548), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208255540)}}
[LOG] 0:00.002 ERROR juju.worker.peergrouper errorFor "Session.Set" -> <nil>
[LOG] 0:00.002 INFO juju.worker.peergrouper setting replicaset members to []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208255570), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208255558)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208255548), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208255540)}}
[LOG] 0:00.002 INFO juju.worker.peergrouper successfully changed replica set to []replicaset.Member{replicaset.Member{Id:2, Address:"0.1.2.12:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208255570), Tags:map[string]string{"juju-machine-id":"12"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208255558)}, replicaset.Member{Id:0, Address:"0.1.2.10:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(nil), Tags:map[string]string{"juju-machine-id":"10"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(nil)}, replicaset.Member{Id:1, Address:"0.1.2.11:1234", Arbiter:(*bool)(nil), BuildIndexes:(*bool)(nil), Hidden:(*bool)(nil), Priority:(*float64)(0xc208255548), Tags:map[string]string{"juju-machine-id":"11"}, SlaveDelay:(*time.Duration)(nil), Votes:(*int)(0xc208255540)}}
[LOG] 0:00.008 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.008 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.008 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.008 ERROR juju.worker.peergrouper cannot publish API server addresses: publish error
[LOG] 0:00.008 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.008 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.008 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.008 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "12", wantsVote: true, hostPort: "0.1.2.12:1234"}:(*replicaset.Member)(0xc20805bea0), &peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc20805bf40), &peergrouper.machine{id: "11", wantsVote: true, hostPort: "0.1.2.11:1234"}:(*replicaset.Member)(0xc2080160a0)}
[LOG] 0:00.009 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.009 DEBUG juju.worker.peergrouper maxId: 2
[LOG] 0:00.009 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.009 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.009 DEBUG juju.worker.peergrouper machine "11" is not ready (has status: false)
[LOG] 0:00.009 DEBUG juju.worker.peergrouper machine "12" is not ready (has status: false)
[LOG] 0:00.009 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.009 DEBUG juju.worker.peergrouper no change in desired peer group
[LOG] 0:00.014 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 11" -> <nil>
[LOG] 0:00.014 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 12" -> <nil>
[LOG] 0:00.014 ERROR juju.worker.peergrouper errorFor "Machine.InstanceId 10" -> <nil>
[LOG] 0:00.014 ERROR juju.worker.peergrouper cannot publish API server addresses: publish error
[LOG] 0:00.014 ERROR juju.worker.peergrouper errorFor "Session.CurrentStatus" -> <nil>
[LOG] 0:00.014 ERROR juju.worker.peergrouper errorFor "Session.CurrentMembers" -> <nil>
[LOG] 0:00.015 DEBUG juju.worker.peergrouper calculating desired peer group
[LOG] 0:00.015 DEBUG juju.worker.peergrouper members: map[*peergrouper.machine]*replicaset.Member{&peergrouper.machine{id: "12", wantsVote: true, hostPort: "0.1.2.12:1234"}:(*replicaset.Member)(0xc208017e00), &peergrouper.machine{id: "10", wantsVote: true, hostPort: "0.1.2.10:1234"}:(*replicaset.Member)(0xc208017e50), &peergrouper.machine{id: "11", wantsVote: true, hostPort: "0.1.2.11:1234"}:(*replicaset.Member)(0xc208017ea0)}
[LOG] 0:00.015 DEBUG juju.worker.peergrouper extra: []replicaset.Member(nil)
[LOG] 0:00.015 DEBUG juju.worker.peergrouper maxId: 2
[LOG] 0:00.015 DEBUG juju.worker.peergrouper assessing possible peer group changes:
[LOG] 0:00.015 DEBUG juju.worker.peergrouper machine "10" is already voting
[LOG] 0:00.015 DEBUG juju.worker.peergrouper machine "11" is not ready (has status: false)
[LOG] 0:00.015 DEBUG juju.worker.peergrouper machine "12" is not ready (has status: false)
[LOG] 0:00.015 DEBUG juju.worker.peergrouper assessed
[LOG] 0:00.015 DEBUG juju.worker.peergrouper no change in desired peer group
worker_test.go:350:
    c.Assert(servers, jc.DeepEquals, expectedAPIHostPorts(3))
... obtained [][]instance.HostPort = [][]instance.HostPort{[]instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.11", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}, []instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.12", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}, []instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.10", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}}
... expected [][]instance.HostPort = [][]instance.HostPort{[]instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.10", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}, []instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.11", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}, []instance.HostPort{instance.HostPort{Address:instance.Address{Value:"0.1.2.12", Type:"ipv4", NetworkName:"", NetworkScope:""}, Port:5678}}}
... mismatch at [0][0].Address.Value: unequal; obtained "0.1.2.11"; expected "0.1.2.10"

[LOG] 0:00.023 ERROR juju.worker.peergrouper peergrouper loop terminated: tomb: dying
OOPS: 6 passed, 5 FAILED
--- FAIL: TestPackage (0.97 seconds)
FAIL
FAIL launchpad.net/juju-core/worker/peergrouper 1.079s

Tags: gccgo ppc64el

Related branches

lp://staging/~axwalk/juju-core/lp1298133-peergrouper-tests-maporder
Juju Engineering: Pending requested
Diff: 101 lines (+16/-9)
2 files modified
worker/peergrouper/desired_test.go (+7/-0)
worker/peergrouper/worker_test.go (+9/-9)
Andrew Wilkins (axwalk)
Changed in juju-core:
assignee: nobody → Andrew Wilkins (axwalk)
status: Triaged → In Progress
Andrew Wilkins (axwalk)
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.