Bit of a convoluted one. and it may just be a specific kernel issue, but I'll post it here because it broke the charmhelpers test suite.
This situation is this. I have a Wily host running kernel 4.2.0-7-lowlatency. For some reason libkmod cannot read /lib/modules/4.2.0-7-lowlatency/modules.dep.bin (naively I'd say this is an ABI issue in the tooling), which causes the attempt to load the ip6_tables module in is_ipv6_ok to fail, and therefore the tests that rely on this also fail.
You can work around this issue by simply manually loading ip6_tables with modprobe.
The failing test output looks like this (I pulled out some of the less useful summary information).
-*- mode: compilation; default-directory: "~/src/charm-helpers/trunk/" -*-
Compilation started at Mon Sep 7 11:44:00
make -k test
Checking for Python syntax...
Py2 OK
Py3 OK
Starting Py2 tests...
.venv/bin/nosetests -s --nologcapture tests/
...........................................................................................................................................................................................................................juju-log: INFO: Loading kernel module ip6_tables
modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
Ejuju-log: INFO: Loading kernel module ip6_tables
modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
E.........E........juju-log: INFO: Loading kernel module ip6_tables
modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
E.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................INFO:root:Checking out lp:charm-helpers to /tmp/foo/charm-helpers.
..INFO:root:Adding missing __init__.py: /tmp/hooks/__init__.py
INFO:root:Adding missing __init__.py: /tmp/hooks/helpers/__init__.py
INFO:root:Adding missing __init__.py: /tmp/hooks/helpers/foo/__init__.py
...........INFO:root:Syncing directory: /tmp/charm-helpers/charmhelpers/core -> hooks/charmhelpers/core.
..INFO:root:Syncing pyfile: /tmp/charm-helpers/core/host.py -> hooks/charmhelpers/core.
...
======================================================================
ERROR: test_enable_fail (tests.contrib.network.test_ufw.TestUFW)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/.venv/local/lib/python2.7/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tealeg/src/charm-helpers/trunk/tests/contrib/network/test_ufw.py", line 62, in test_enable_fail
self.assertFalse(ufw.enable())
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 136, in enable
if not is_ipv6_ok(soft_fail):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 101, in is_ipv6_ok
raise UFWIPv6Error("IPv6 firewall support broken")
UFWIPv6Error: IPv6 firewall support broken
======================================================================
ERROR: test_enable_ok (tests.contrib.network.test_ufw.TestUFW)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/.venv/local/lib/python2.7/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tealeg/src/charm-helpers/trunk/tests/contrib/network/test_ufw.py", line 48, in test_enable_ok
self.assertTrue(ufw.enable())
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 136, in enable
if not is_ipv6_ok(soft_fail):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 101, in is_ipv6_ok
raise UFWIPv6Error("IPv6 firewall support broken")
UFWIPv6Error: IPv6 firewall support broken
======================================================================
ERROR: test_no_ip6_tables (tests.contrib.network.test_ufw.TestUFW)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/.venv/local/lib/python2.7/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tealeg/src/charm-helpers/trunk/tests/contrib/network/test_ufw.py", line 324, in test_no_ip6_tables
self.assertTrue(ufw.enable())
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 136, in enable
if not is_ipv6_ok(soft_fail):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 90, in is_ipv6_ok
modprobe('ip6_tables')
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/core/kernel.py", line 39, in modprobe
with open('/etc/modules', 'r+') as modules:
IOError: [Errno 13] Permission denied: '/etc/modules'
======================================================================
ERROR: test_with_ipv6 (tests.contrib.network.test_ufw.TestUFW)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/.venv/local/lib/python2.7/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tealeg/src/charm-helpers/trunk/tests/contrib/network/test_ufw.py", line 408, in test_with_ipv6
ufw.enable()
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 136, in enable
if not is_ipv6_ok(soft_fail):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 101, in is_ipv6_ok
raise UFWIPv6Error("IPv6 firewall support broken")
UFWIPv6Error: IPv6 firewall support broken
FAILED (errors=4)
Makefile:60: recipe for target 'test2' failed
make: *** [test2] Error 1
Starting Py3 tests...
.venv3/bin/nosetests -s --nologcapture tests/
...........................................................................................................................................................................................................................juju-log: INFO: Loading kernel module ip6_tables
modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
Ejuju-log: INFO: Loading kernel module ip6_tables
modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
E.........E........juju-log: INFO: Loading kernel module ip6_tables
modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
E...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SSS..........................................................SSSS.................INFO:root:Checking out lp:charm-helpers to /tmp/foo/charm-helpers.
..INFO:root:Adding missing __init__.py: /tmp/hooks/__init__.py
INFO:root:Adding missing __init__.py: /tmp/hooks/helpers/__init__.py
INFO:root:Adding missing __init__.py: /tmp/hooks/helpers/foo/__init__.py
...........INFO:root:Syncing directory: /tmp/charm-helpers/charmhelpers/core -> hooks/charmhelpers/core.
..INFO:root:Syncing pyfile: /tmp/charm-helpers/core/host.py -> hooks/charmhelpers/core.
...
======================================================================
ERROR: test_enable_fail (tests.contrib.network.test_ufw.TestUFW)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 90, in is_ipv6_ok
modprobe('ip6_tables')
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/core/kernel.py", line 37, in modprobe
check_call(cmd)
File "/usr/lib/python3.4/subprocess.py", line 561, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['modprobe', 'ip6_tables']' returned non-zero exit status 1
During handling of the above exception, another exception occurred:````````
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/.venv3/lib/python3.4/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tealeg/src/charm-helpers/trunk/tests/contrib/network/test_ufw.py", line 62, in test_enable_fail
self.assertFalse(ufw.enable())
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 136, in enable
if not is_ipv6_ok(soft_fail):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 101, in is_ipv6_ok
raise UFWIPv6Error("IPv6 firewall support broken")
charmhelpers.contrib.network.ufw.UFWIPv6Error: IPv6 firewall support broken
======================================================================
ERROR: test_enable_ok (tests.contrib.network.test_ufw.TestUFW)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 90, in is_ipv6_ok
modprobe('ip6_tables')
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/core/kernel.py", line 37, in modprobe
check_call(cmd)
File "/usr/lib/python3.4/subprocess.py", line 561, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['modprobe', 'ip6_tables']' returned non-zero exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/.venv3/lib/python3.4/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tealeg/src/charm-helpers/trunk/tests/contrib/network/test_ufw.py", line 48, in test_enable_ok
self.assertTrue(ufw.enable())
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 136, in enable
if not is_ipv6_ok(soft_fail):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 101, in is_ipv6_ok
raise UFWIPv6Error("IPv6 firewall support broken")
charmhelpers.contrib.network.ufw.UFWIPv6Error: IPv6 firewall support broken
======================================================================
ERROR: test_no_ip6_tables (tests.contrib.network.test_ufw.TestUFW)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/.venv3/lib/python3.4/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tealeg/src/charm-helpers/trunk/tests/contrib/network/test_ufw.py", line 324, in test_no_ip6_tables
self.assertTrue(ufw.enable())
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 136, in enable
if not is_ipv6_ok(soft_fail):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 90, in is_ipv6_ok
modprobe('ip6_tables')
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/core/kernel.py", line 39, in modprobe
with open('/etc/modules', 'r+') as modules:
PermissionError: [Errno 13] Permission denied: '/etc/modules'
======================================================================
ERROR: test_with_ipv6 (tests.contrib.network.test_ufw.TestUFW)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 90, in is_ipv6_ok
modprobe('ip6_tables')
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/core/kernel.py", line 37, in modprobe
check_call(cmd)
File "/usr/lib/python3.4/subprocess.py", line 561, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['modprobe', 'ip6_tables']' returned non-zero exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tealeg/src/charm-helpers/trunk/.venv3/lib/python3.4/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tealeg/src/charm-helpers/trunk/tests/contrib/network/test_ufw.py", line 408, in test_with_ipv6
ufw.enable()
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 136, in enable
if not is_ipv6_ok(soft_fail):
File "/home/tealeg/src/charm-helpers/trunk/charmhelpers/contrib/network/ufw.py", line 101, in is_ipv6_ok
raise UFWIPv6Error("IPv6 firewall support broken")
charmhelpers.contrib.network.ufw.UFWIPv6Error: IPv6 firewall support broken
Tests should at minimum detect this condition and skip if the requirements are not met.
Unit tests should not need to modprobe anything, by definition. Mocking/ stubbing/ injecting the calls to the ipv6 stack should be used instead.