nova-docker

Instance creation fails using Docker /w libcontainer execution backend

Bug #1292479 reported by Pavel Sviderski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nova-docker
Fix Committed
Medium
Aaron Rosen

Bug Description

When docker version 0.9.0 is used the nova docker driver cannot create an instance.

2014-03-14 14:50:23.293 ERROR nova.compute.manager [req-f553b192-ff5b-4d7e-ac9d-f7cca3804f74 admin demo] [instance: af7ffeee-da4d-4120-a412-61bad3991ced] Error: Cannot setup network: Cannot find any PID under container "612ada12e14f52771d32e3ae4c410ec31838c28aec8b53357531422d7c02b7d8"
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] Traceback (most recent call last):
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/compute/manager.py", line 1070, in _build_instance
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] set_access_ip=set_access_ip)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] return function(self, context, *args, **kwargs)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/compute/manager.py", line 1478, in _spawn
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] six.reraise(self.type_, self.value, self.tb)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/compute/manager.py", line 1475, in _spawn
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] block_device_info)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/virt/docker/driver.py", line 313, in spawn
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] instance_id=instance['name'])
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] InstanceDeployFailure: Cannot setup network: Cannot find any PID under container "612ada12e14f52771d32e3ae4c410ec31838c28aec8b53357531422d7c02b7d8"
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced]

The driver fails in the '_find_container_pid' method. The reason is that in 0.9 there is a new built-in execution driver used by default (libcontainer). Instead of cgroup path /sys/fs/cgroup/devices/lxc/ it uses /sys/fs/cgroup/devices/docker/.

Could it be possible to rewrite _find_container_pid using the docker remote api and do not access cgroup filesystem directly?

See original description
Tags: docker
Pavel Sviderski (spyyy)
description: updated
Revision history for this message
Pavel Sviderski (spyyy) wrote :

possible solution is in the attachment

Revision history for this message
Erica Windisch (ewindisch) wrote :

It should be noted that while libcontainer is now default, it remains possible to run docker with '-e lxc'.

summary: - Instance creation fails using Docker 0.9.0
+ Instance creation fails using Docker /w libcontainer execution backend
Revision history for this message
Erica Windisch (ewindisch) wrote :

Pavel, thank you for the patch. We're moving the docker driver into Stackforge and will be able to apply it once we complete the migration.

Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Pavel Sviderski (spyyy)
Changed in nova:
assignee: nobody → Pavel Sviderski (pavel-a)
Revision history for this message
Erica Windisch (ewindisch) wrote :

@Pavel - would you mind submitting this using gerrit / git-review?

The repository is https://github.com/stackforge/nova-docker, we use gerrit and git-review like other projects.

Thanks!

Erica Windisch (ewindisch)
affects: nova → nova-docker
Aaron Rosen (arosen)
Changed in nova-docker:
assignee: Pavel Sviderski (pavel-a) → Aaron Rosen (arosen)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova-docker (master)

Fix proposed to branch: master
Review: https://review.openstack.org/84830

Changed in nova-docker:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-docker (master)

Reviewed: https://review.openstack.org/84830
Committed: https://git.openstack.org/cgit/stackforge/nova-docker/commit/?id=ab6d09cacce89ec87049d96ec89e8630dd2e146a
Submitter: Jenkins
Branch: master

commit ab6d09cacce89ec87049d96ec89e8630dd2e146a
Author: Aaron Rosen <email address hidden>
Date: Wed Apr 2 11:16:43 2014 -0700

    Use docker api in order to find container pid

    Previously, the docker driver would try to obtain a containers pid by
    looking it up in /sys/fs/cgroup/drivces/ though this path is not always
    correct depending on the version of docker one is using. This patch
    corrects this by getting the pid by querying docker instead.

    Change-Id: If3cc342dc781a360f2414a1291efda57d102a85b
    Closes-bug: 1292479
    Co-Authored-By: Pavel Sviderski <email address hidden>

Changed in nova-docker:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.