Leftover /var/run files prevent Keystone start

Bug #1489676 reported by Tim Potter
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Sam Yaple

Bug Description

I'm running containers using the liberty2 release and have seen a couple of times the apache server in the keystone container not start up due to leftover state files in /var/run. This is running the kollaglue/centos-rdo-keystone:liberty2 pulled from the public Docker registry.

I'd like to see the cleanup of /var/run (or equivalent for other OS's) everywhere to avoid this happening to other services in other containers.

Here's a log from the latest startup of the container:

Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
database is active @ 192.168.1.3
Checking for required variables
Setting up Keystone DB
Configuring keystone.conf
Setting up PKI
No handlers could be found for logger "oslo_config.cfg"
2015-08-24 05:32:40.336 32 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/private/cakey.pem already exists
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Initializing Keystone DB
No handlers could be found for logger "oslo_config.cfg"
Starting Keystone
Verifying Keystone is running
waiting for Keystone @ http://192.168.1.3:35357/v2.0
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
httpd (pid 52) already running
waiting for Keystone @ http://192.168.1.3:35357/v2.0
waiting for Keystone @ http://192.168.1.3:35357/v2.0

The waiting for keystone message continues indefinitely.

I've logged in to the container and here's what's in /var/run/httpd:

tpot@userver:~$ docker exec -it compose_keystone_1 bash
[root@userver /]# ls /var/run/httpd/
authdigest_shm.52 htcacheclean httpd.pid wsgi.52.0.1.sock wsgi.52.0.2.sock

Process listing is:

[root@userver /]# ps auxww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.0 11744 1596 ? Ss 01:01 0:00 /bin/bash /opt/kolla/config-internal.sh
root 743 0.7 0.0 13464 2024 ? Ss 01:07 0:00 bash
root 780 0.0 0.0 4320 368 ? S 01:08 0:00 sleep 1
root 781 0.0 0.0 21372 1308 ? R+ 01:08 0:00 ps auxww

Fix is to enter the container and rm -rf /var/run/httpd/* and then restart it. The log now looks more useful with just a single "waiting for Keystone" message.

Running the kolla-common script
Checking connectivity to the DB
database is active @ 192.168.1.3
Checking for required variables
Setting up Keystone DB
Configuring keystone.conf
Setting up PKI
No handlers could be found for logger "oslo_config.cfg"
2015-08-20 22:34:35.852 32 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/private/cakey.pem already exists
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Initializing Keystone DB
No handlers could be found for logger "oslo_config.cfg"
Starting Keystone
Verifying Keystone is running
waiting for Keystone @ http://192.168.1.3:35357/v2.0
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
keystone is active @ http://192.168.1.3:35357/v2.0
Creating Keystone tenant, user, role, service and endpoints
using existing tenant admin (e1a0d4522a1c4d6f81f79497394799f9)
using existing user admin (cab433e4f2304ec3996ae8983b9e123e)
updating enabled=True for user admin
updating password for user admin
using existing role admin (b0c0987b16b3424c847ca4e221fbaea3)

Revision history for this message
Tim Potter (tpot) wrote :

Just cutting and pasting comments from my gerrit submission I made incorrectly instead of filing a bug on launchpad:

Sam Yaple Aug 24 4:42 PM

Debian based systems do not use httpd, but instead apache2. Additionally, with different versions of packages and configuration options Debian-based systems *may* not be affected.

Either way, this will need to be in a block that only runs on Centos/RHEL/Fedora based systems.

Steven Dake Aug 24 10:15 PM

note there is an environment variable set in the docker files as follows;

https://github.com/stackforge/kolla/blob/master/docker_templates/base/Dockerfile.j2#L6

Steven Dake (sdake)
Changed in kolla:
milestone: none → liberty-rc1
status: New → Triaged
importance: Undecided → High
Steven Dake (sdake)
Changed in kolla:
milestone: liberty-rc1 → liberty-rc2
Revision history for this message
Sam Yaple (s8m) wrote :

I believe the best approach to fixing this is to not have apache use a pid file. The only service that needs a pid file Kolla can manage is haproxy to do hot reload

Changed in kolla:
assignee: nobody → Sam Yaple (s8m)
Steven Dake (sdake)
Changed in kolla:
milestone: liberty-rc2 → liberty-rc3
Steven Dake (sdake)
Changed in kolla:
importance: High → Critical
Changed in kolla:
milestone: liberty-rc3 → mitaka-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/241980
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=7df3a0bea56f654a3d65da27c0e783f775b4ed18
Submitter: Jenkins
Branch: master

commit 7df3a0bea56f654a3d65da27c0e783f775b4ed18
Author: Paul Bourke <email address hidden>
Date: Thu Nov 5 11:46:14 2015 +0000

    Clean up keystone httpd pid files for RPM distros

    In some cases we're seeing httpd not cleaning up properly after itself,
    which results in the keystone container failing to restart. This is
    confirmed to happen on rpm based distros, but have not had any reports
    on Ubuntu.

    Change-Id: I58b006189e700f1c851601b4f64dd0fae931103c
    Closes-Bug: #1489676
    Co-Authored-By: Tim Potter <email address hidden>

Changed in kolla:
status: Triaged → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/242106

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/liberty)

Reviewed: https://review.openstack.org/242106
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=d30fb21f8185a042d12a2a9d0b76ebef4a8733cc
Submitter: Jenkins
Branch: stable/liberty

commit d30fb21f8185a042d12a2a9d0b76ebef4a8733cc
Author: Paul Bourke <email address hidden>
Date: Thu Nov 5 11:46:14 2015 +0000

    Clean up keystone httpd pid files for RPM distros

    In some cases we're seeing httpd not cleaning up properly after itself,
    which results in the keystone container failing to restart. This is
    confirmed to happen on rpm based distros, but have not had any reports
    on Ubuntu.

    Change-Id: I58b006189e700f1c851601b4f64dd0fae931103c
    Closes-Bug: #1489676
    Co-Authored-By: Tim Potter <email address hidden>
    (cherry picked from commit 7df3a0bea56f654a3d65da27c0e783f775b4ed18)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.openstack.org/244048

Sam Yaple (s8m)
Changed in kolla:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.