Canonical Juju

firewaller panic in 2.2.6

Bug #1732614 reported by Junien Fridrick
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Critical
Heather Lanigan
Canonical Juju 2.3-rc1

Bug Description

Now with a stack trace ! This is happening frequently (multiple times per minute) on our controller.

Thanks !

2017-11-16 06:59:22 DEBUG juju.worker.dependency engine.go:504 "firewaller" manifold worker stopped: panic resulted in: runtime error: invalid memory address or nil pointer dereference
2017-11-16 06:59:22 ERROR juju.worker.dependency engine.go:551 "firewaller" manifold worker returned unexpected error: panic resulted in: runtime error: invalid memory address or nil pointer dereference
2017-11-16 06:59:22 DEBUG juju.worker.dependency engine.go:553 stack trace:
panic resulted in: runtime error: invalid memory address or nil pointer dereference
stacktrace:
goroutine 8103268 [running]:
runtime/debug.Stack(0x26fb101, 0x15, 0xc47024d360)
        /var/lib/jenkins/crossbuild/golang-1.8/src/runtime/debug/stack.go:24 +0x79
github.com/juju/juju/worker/catacomb.runSafely.func1(0xc47024df90)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/worker/catacomb/catacomb.go:286 +0xd1
panic(0x22285c0, 0x3b3beb0)
        /var/lib/jenkins/crossbuild/golang-1.8/src/runtime/panic.go:489 +0x2cf
github.com/juju/juju/provider/openstack.secGroupMatchesIngressRule(0x0, 0x0, 0xc461dfd530, 0xc443cf2540, 0x24, 0xc461dfd510, 0xf, 0x0, 0x0, 0xc461dfd540, ...)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/provider/openstack/firewaller.go:853 +0x34
github.com/juju/juju/provider/openstack.(*neutronFirewaller).closePortsInGroup(0xc424ae18a0, 0xc4735c06c0, 0x2f, 0xc438cea900, 0x1, 0x1, 0x211d800, 0xc440c77100)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/provider/openstack/firewaller.go:887 +0x24e
github.com/juju/juju/provider/openstack.(*neutronFirewaller).(github.com/juju/juju/provider/openstack.closePortsInGroup)-fm(0xc4735c06c0, 0x2f, 0xc438cea900, 0x1, 0x1, 0xc47024d8a8, 0x42c6bf)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/provider/openstack/firewaller.go:750 +0x5c
github.com/juju/juju/provider/openstack.(*firewallerBase).closeInstancePorts(0xc424ae18a0, 0xc47024d930, 0xc4737b3b65, 0x1, 0xc438cea900, 0x1, 0x1, 0xc4203a5e30, 0x390d344)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/provider/openstack/firewaller.go:379 +0xa4
github.com/juju/juju/provider/openstack.(*neutronFirewaller).CloseInstancePorts(0xc424ae18a0, 0x3b78880, 0xc46fbb1590, 0xc4737b3b65, 0x1, 0xc438cea900, 0x1, 0x1, 0x27430a3, 0x26)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/provider/openstack/firewaller.go:787 +0x219
github.com/juju/juju/provider/openstack.(*switchingFirewaller).CloseInstancePorts(0xc4425c6ee0, 0x3b78880, 0xc46fbb1590, 0xc4737b3b65, 0x1, 0xc438cea900, 0x1, 0x1, 0x2, 0x2)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/provider/openstack/firewaller.go:198 +0xd9
github.com/juju/juju/provider/openstack.(*openstackInstance).ClosePorts(0xc46fbb1590, 0xc4737b3b65, 0x1, 0xc438cea900, 0x1, 0x1, 0x0, 0x0)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/provider/openstack/provider.go:459 +0x8a
github.com/juju/juju/worker/firewaller.(*Firewaller).reconcileInstances(0xc43a245b00, 0xc4737b3b69, 0x1)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/worker/firewaller/firewaller.go:544 +0x583
github.com/juju/juju/worker/firewaller.(*Firewaller).loop(0xc43a245b00, 0xc450d1f760, 0xc48701bac0)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/worker/firewaller/firewaller.go:248 +0x3c2
github.com/juju/juju/worker/firewaller.(*Firewaller).(github.com/juju/juju/worker/firewaller.loop)-fm(0xc400000008, 0x279f1e0)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/worker/firewaller/firewaller.go:167 +0x2a
github.com/juju/juju/worker/catacomb.runSafely(0xc473722d00, 0x3b55a60, 0xc475787540)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/worker/catacomb/catacomb.go:289 +0x6c
github.com/juju/juju/worker/catacomb.Invoke.func3(0xc43a245b00, 0xc43a245b00, 0xc473722d00, 0x0, 0x0, 0x0)
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/worker/catacomb/catacomb.go:116 +0x81
created by github.com/juju/juju/worker/catacomb.Invoke
        /tmp/tmpwONtpt/juju-core_2.2.6/src/github.com/juju/juju/worker/catacomb/catacomb.go:117 +0x23a

Heather Lanigan (hmlanigan)
Changed in juju:
assignee: nobody → Heather Lanigan (hmlanigan)
status: New → In Progress
Revision history for this message
Heather Lanigan (hmlanigan) wrote :

https://github.com/juju/juju/pull/8095

Ian Booth (wallyworld)
Changed in juju:
milestone: none → 2.3-rc1
importance: Undecided → Critical
Ian Booth (wallyworld)
Changed in juju:
status: In Progress → Fix Committed
Revision history for this message
Heather Lanigan (hmlanigan) wrote :

@Junien is it possible for me to get info on the security group rules in this config? Make sure we're doing the right thing?

Revision history for this message
Junien Fridrick (axino) wrote :

Sure, what security groups exactly ? This is a controller with 184 models and 480 openedPorts. Note that the log preceding the panic is always an attempt to close a port, apparently : https://pastebin.canonical.com/203461/

Revision history for this message
Heather Lanigan (hmlanigan) wrote :

@Junien, Based on the pastebin, the security groups for machine-0 thru machine-4, though one might do it. The name of the security group will be named: juju-<controller-uuid>-<model uuid>-<machine number>. Let's also get the model security group which is named: juju-<controller-uuid>-<model uuid>.

Revision history for this message
Junien Fridrick (axino) wrote :

So I found the root cause here : we had icmp rules in a machine secgroup. icmp rules typically don't have ports associated (they show as "-1" in nova secgroup-list-rule). Once I removed said icmp rules, the panics stopped.

Anastasia (anastasia-macmood)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.