OpenStack Compute (nova)

iptables being run on a nova-api server

Bug #1002111 reported by Sam Morrison on 2012-05-21

This bug affects 7 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Thierry Carrez	OpenStack Compute (nova) 2012.2 "folsom"
	nova (Ubuntu)	Fix Released	Undecided	Adam Gandelman

Bug Description

When starting nova-api I get the below error in nova-api.log

I'm running the latest packages in Ubuntu 12.04.

The server is running the following nova services:
nova-api
nova-cert
nova-scheduler
nova-consoleauth

Not sure why it is trying to do iptables commands?

Cheers,
Sam

2012-05-21 12:22:58 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=27739) inner /usr/lib/python2.7/dist-packages/nova/utils.py:927
2012-05-21 12:22:58 DEBUG nova.utils [-] Got semaphore "iptables" for method "apply"... from (pid=27739) inner /usr/lib/python2.7/dist-packages/nova/utils.py:931
2012-05-21 12:22:58 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=27739) inner /usr/lib/python2.7/dist-packages/nova/utils.py:935
2012-05-21 12:22:58 DEBUG nova.utils [-] Got file lock "iptables" for method "apply"... from (pid=27739) inner /usr/lib/python2.7/dist-packages/nova/utils.py:942
2012-05-21 12:22:58 DEBUG nova.utils [-] Running cmd (subprocess): sudo /usr/bin/nova-rootwrap iptables-save -t filter from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-05-21 12:22:58 DEBUG nova.utils [-] Result was 99 from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:235
2012-05-21 12:22:58 DEBUG nova.utils [-] ['sudo', '/usr/bin/nova-rootwrap', 'iptables-save', '-t', 'filter'] failed. Retrying. from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:248
2012-05-21 12:22:59 DEBUG nova.utils [-] Running cmd (subprocess): sudo /usr/bin/nova-rootwrap iptables-save -t filter from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-05-21 12:22:59 DEBUG nova.utils [-] Result was 99 from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:235
2012-05-21 12:22:59 DEBUG nova.utils [-] ['sudo', '/usr/bin/nova-rootwrap', 'iptables-save', '-t', 'filter'] failed. Retrying. from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:248
2012-05-21 12:23:01 DEBUG nova.utils [-] Running cmd (subprocess): sudo /usr/bin/nova-rootwrap iptables-save -t filter from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-05-21 12:23:01 DEBUG nova.utils [-] Result was 99 from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:235
2012-05-21 12:23:01 DEBUG nova.utils [-] ['sudo', '/usr/bin/nova-rootwrap', 'iptables-save', '-t', 'filter'] failed. Retrying. from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:248
2012-05-21 12:23:03 DEBUG nova.utils [-] Running cmd (subprocess): sudo /usr/bin/nova-rootwrap iptables-save -t filter from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-05-21 12:23:03 DEBUG nova.utils [-] Result was 99 from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:235
2012-05-21 12:23:03 DEBUG nova.utils [-] ['sudo', '/usr/bin/nova-rootwrap', 'iptables-save', '-t', 'filter'] failed. Retrying. from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:248
2012-05-21 12:23:04 DEBUG nova.utils [-] Running cmd (subprocess): sudo /usr/bin/nova-rootwrap iptables-save -t filter from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-05-21 12:23:04 DEBUG nova.utils [-] Result was 99 from (pid=27739) execute /usr/lib/python2.7/dist-packages/nova/utils.py:235
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 336, in fire_timers
    timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/service.py", line 101, in run_server
    server.start()
  File "/usr/lib/python2.7/dist-packages/nova/service.py", line 366, in start
    self.manager.init_host()
  File "/usr/lib/python2.7/dist-packages/nova/api/manager.py", line 42, in init_host
    self.network_driver.metadata_accept()
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 424, in metadata_accept
    iptables_manager.apply()
  File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 943, in inner
    retval = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 328, in apply
    attempts=5)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 812, in _execute
    return utils.execute(*cmd, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 242, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo /usr/bin/nova-rootwrap iptables-save -t filter
Exit code: 99
Stdout: 'Unauthorized command: iptables-save -t filter\n'
Stderr: 'sudo: unable to resolve host server-3093\n'

Tags:

Related branches

lp://staging/~openstack-ubuntu-testing/nova/quantal-folsom-proposed

lp://staging/~openstack-ubuntu-testing/nova/precise-folsom-proposed

lp://staging/~openstack-ubuntu-testing/nova/quantal-folsom

lp://staging/~openstack-ubuntu-testing/nova/precise-folsom

lp://staging/ubuntu/quantal/nova

lp://staging/~openstack-ubuntu-testing/nova/raring-grizzly

Revision history for this message

Piotr Siwczak (psiwczak) wrote on 2012-06-06:

It's setting up iptables rules for the metadata service, which is considered to be a part of nova-api. I put some debugging code into the function metadata_accept and here's the rule which is set up by nova-api upon restart:

DEBUG nova.network.linux_net [-] INPUT -s 0.0.0.0/0 -d 10.0.2.15 -p tcp -m tcp --dport 8775 -j ACCEPT from (pid=18013) metadata_accept /opt/stack/nova/nova/network/linux_net.py:426

But the bug is definitely there. It refers to nova-rootwrap script.

From the log you posted, it can be seen that this script fails in case of applying "iptables-save" commands. This is because nova-rootwrap uses internal filters for commands ran as root. The filters for iptables are defined here:
/usr/lib/python2.7/dist-packages/nova/rootwrap/network.py

If you copy this file over to your nova-api server, the iptables should succeed.

But in general - it should not work this way - one should have a separate policy file called:

/usr/lib/python2.7/dist-packages/nova/rootwrap/api.py

There's no such file - so I guess it's reasonable to vote for a bug here!

Revision history for this message

Piotr Siwczak (psiwczak) wrote on 2012-06-06:

Just to add some clarification:

rootwrap/network.py normally resides on nova-network node

Tom Fifield (fifieldt) on 2012-06-07

Changed in nova:
status:	New → Confirmed

Thierry Carrez (ttx) on 2012-06-07

Changed in nova:
assignee:	nobody → Thierry Carrez (ttx)
importance:	Undecided → High
tags:	added: rootwrap

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-06-07: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/8293

Changed in nova:
status:	Confirmed → In Progress

Revision history for this message

Sam Morrison (sorrison) wrote on 2012-06-07:

So does this mean that nova controls iptables on an api host? If so is it possible to disable?

We run our own firewall on our infrastructure nodes and would rather not have nova touching them.

Revision history for this message

Piotr Siwczak (psiwczak) wrote on 2012-06-07:

Sam,

It seems to be controlling only entry for local metadata service:
INPUT -s 0.0.0.0/0 -d 10.0.2.15 -p tcp -m tcp --dport 8775 -j ACCEPT

Revision history for this message

Vish Ishaya (vishvananda) wrote on 2012-06-07:

It is actually only used to create an input allow rule for the metadata server.

If you run nova-api without metadata in enabled apis (the default) it does not do anything

it is needed for nova-api if you enable the metdata server in enabled_apis or when running nova-api-metadata

Revision history for this message

Thierry Carrez (ttx) wrote on 2012-06-08:

Vish: It was kinda good that nova-api was not able to run *any* command as root. For the future could it be a better solution to completely separate nova-api from nova-metadata and allow only the latter to run commands as root ?

Revision history for this message

Vish Ishaya (vishvananda) wrote on 2012-06-10:

Thierry, they already run separately by default, perhaps it makes more sense to rename rootwrap/api.py to rootwrap/metadata.py or rootwrap/api-metadata.py to make it clear that it needs to be installed with nova-api-metadata ?

Revision history for this message

Thierry Carrez (ttx) wrote on 2012-06-11:

It's a bit of a packaging issue, actually.

If nova-api could only specify a single API and was always installed as a nova-api-ec2, nova-api-osapi or nova-api-metadata package, then we could ship api-metadata.py only with nova-api-metadata...

But as long as you allow a single nova-api to run with "metadata" as one of its enabled api you kinda need to ship api-metadata.py within the "nova-api" package, just in case...

This is all packaging though, so we should just have a "api-metadata.py" and leave it to distros to deploy it only with nova-api--metadata packages.

The only question left on our side is... should we actually really support multiple enabled APIs on a single nova-api... or should we force them (or strongle encourage them) to run as separate nova-api-ec2/nova-api-osapi/nova-api-metadata, which means support only one enabled API at a time. From a security standpoint, that would definitely be a win if we encouraged those to be separate, if -metadata needs root access whereas -osapi and -ec2 do not.

Thierry Carrez (ttx) on 2012-06-11

Changed in nova:
assignee:	Thierry Carrez (ttx) → Piotr Siwczak (psiwczak)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-07-24:

#10

Fix proposed to branch: master
Review: https://review.openstack.org/10222

Changed in nova:
assignee:	Piotr Siwczak (psiwczak) → Thierry Carrez (ttx)

Revision history for this message

Thierry Carrez (ttx) wrote on 2012-07-24:

#11

When the change hits Nova, it needs to be packaged: api-metadata.filters needs to be deployed with nova-api binary.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-07-24: Fix merged to nova (master)

#12

Reviewed: https://review.openstack.org/10222
Committed: http://github.com/openstack/nova/commit/6c888001f15fba44aaaaaf367757dd81b8512f65
Submitter: Jenkins
Branch: master

commit 6c888001f15fba44aaaaaf367757dd81b8512f65
Author: Thierry Carrez <email address hidden>
Date: Tue Jul 24 17:03:13 2012 +0200

Provide rootwrap filters for nova-api-metadata

    The metadata service in nova-api needs access to
    ip[6]tables-{save-restore} to accept connections to the
    metadata service. This change adds an api-metadata.filters file
    that needs to be deployed on setups running nova-api-metadata
    or nova-api with "metadata" in enabled_apis.

Fixes bug 1002111.

Change-Id: I5aecb223876e12550394f31dbc7df893868baa8b

Changed in nova:
status:	In Progress → Fix Committed

Adam Gandelman (gandelman-a) on 2012-07-25

Changed in nova (Ubuntu):
assignee:	nobody → Adam Gandelman (gandelman-a)
status:	New → In Progress

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2012-07-25:

#13

FYI-

According to flags.py, the metadata endpoint is enabled by default:

    cfg.ListOpt('enabled_apis',
                default=['ec2', 'osapi_compute', 'osapi_volume', 'metadata'],
                help='a list of APIs to enable by default'),

So, on the packaging side we need to ship the filters with both nova-api and nova-api-metadata (the packages Conflict already due to port conflicts, anyway).

Thierry Carrez (ttx) on 2012-08-16

Changed in nova:
milestone:	none → folsom-3
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-08-17:

#14

This bug was fixed in the package nova - 2012.2~f3-0ubuntu1

---------------
nova (2012.2~f3-0ubuntu1) quantal; urgency=low

  [ Chuck Short ]
  * New upstream version.
  * debian/rules: Re-enable testsuite.
  * debian/control:
    - Add python-quantumclient as a build depends.
    - Bump standards to 3.9.3
    - Fix lintian warnings.
    - Recommend python-glanceclient and python-keystoneclient.
    - Add dependency of iptables for nova-network.
  * debian/watch: Update
  * debian/rules: Do not run pep8 tests since upstream is still using an
    older pep8.
  * debian/patches/0001-Update-tools-hacking-for-pep8-1.2-and-
    beyond.patch: Get the testsuite running again.
  * debian/nova-volume.install, debian/nova_tgt: Add support for
    persistent volumes.

  [ Adam Gandelman ]
  * debian/{nova-api.install, nova-api-metadata.install}: Install
    api-metadata.filters. (LP: #1002111)
  * debian/control: Added python-glanceclient.
-- Chuck Short <email address hidden> Thu, 16 Aug 2012 14:04:11 -0500