Keystone (and other services) fail to execute parallel

Bug #1597872 reported by Ken Wronkiewicz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-kubernetes
Fix Released
High
Ken Wronkiewicz

Bug Description

If you launch multiple containers within the same pod that share the same volume and try to write to the same file on a machine with multiple cores available, they'll tend to try to start up at the same time and open the same file and thus bootstrap fails.

Thus, each pod in a bootstrap job needs to have its own log drive.

(It's dramatically easier to debug, IMHO, if the bootstrap jobs just write to stdout, considering Kubernetes will do centralized log collection and tagging for stdout right out of the box)

Changed in kolla-kubernetes:
assignee: nobody → Ken Wronkiewicz (wh-ubuntuone)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-kubernetes (master)

Reviewed: https://review.openstack.org/330777
Committed: https://git.openstack.org/cgit/openstack/kolla-kubernetes/commit/?id=5a6c09ff5e90e463445d1310e3698c1f03052a7d
Submitter: Jenkins
Branch: master

commit 5a6c09ff5e90e463445d1310e3698c1f03052a7d
Author: Ken Wronkiewicz <email address hidden>
Date: Thu Jun 16 13:29:12 2016 -0700

    Fix keystone bootstrap failure

    Keystone was failing to bootstrap because keystone-bootstrap couldn't
    get access to /var/log/kolla/keystone/keystone.log so I mounted one
    pod's /var/log/kolla/keystone as an emptyDir named kolla-logs and
    another as an emptyDir named kolla-logs-bootstrap.

    This only occurs when there are multiple cores, so if you have a one
    CPU vagrant, it won't happen.

    Change-Id: I09e668f35e8d4450981db8ccd7e5d04b3087a405
    Partially-implements: blueprint openstack-services
    Partial-Bug: 1597872

Ryan Hallisey (rthall14)
Changed in kolla-kubernetes:
importance: Undecided → High
status: In Progress → Fix Committed
milestone: none → newton
Changed in kolla-kubernetes:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.