Leftover /var/run files prevent Keystone start
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
kolla |
Fix Released
|
Critical
|
Sam Yaple |
Bug Description
I'm running containers using the liberty2 release and have seen a couple of times the apache server in the keystone container not start up due to leftover state files in /var/run. This is running the kollaglue/
I'd like to see the cleanup of /var/run (or equivalent for other OS's) everywhere to avoid this happening to other services in other containers.
Here's a log from the latest startup of the container:
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
database is active @ 192.168.1.3
Checking for required variables
Setting up Keystone DB
Configuring keystone.conf
Setting up PKI
No handlers could be found for logger "oslo_config.cfg"
2015-08-24 05:32:40.336 32 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/
/etc/keystone/
/etc/keystone/
/etc/keystone/
Initializing Keystone DB
No handlers could be found for logger "oslo_config.cfg"
Starting Keystone
Verifying Keystone is running
waiting for Keystone @ http://
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
httpd (pid 52) already running
waiting for Keystone @ http://
waiting for Keystone @ http://
The waiting for keystone message continues indefinitely.
I've logged in to the container and here's what's in /var/run/httpd:
tpot@userver:~$ docker exec -it compose_keystone_1 bash
[root@userver /]# ls /var/run/httpd/
authdigest_shm.52 htcacheclean httpd.pid wsgi.52.0.1.sock wsgi.52.0.2.sock
Process listing is:
[root@userver /]# ps auxww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.0 11744 1596 ? Ss 01:01 0:00 /bin/bash /opt/kolla/
root 743 0.7 0.0 13464 2024 ? Ss 01:07 0:00 bash
root 780 0.0 0.0 4320 368 ? S 01:08 0:00 sleep 1
root 781 0.0 0.0 21372 1308 ? R+ 01:08 0:00 ps auxww
Fix is to enter the container and rm -rf /var/run/httpd/* and then restart it. The log now looks more useful with just a single "waiting for Keystone" message.
Running the kolla-common script
Checking connectivity to the DB
database is active @ 192.168.1.3
Checking for required variables
Setting up Keystone DB
Configuring keystone.conf
Setting up PKI
No handlers could be found for logger "oslo_config.cfg"
2015-08-20 22:34:35.852 32 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/
/etc/keystone/
/etc/keystone/
/etc/keystone/
Initializing Keystone DB
No handlers could be found for logger "oslo_config.cfg"
Starting Keystone
Verifying Keystone is running
waiting for Keystone @ http://
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
keystone is active @ http://
Creating Keystone tenant, user, role, service and endpoints
using existing tenant admin (e1a0d4522a1c4d
using existing user admin (cab433e4f2304e
updating enabled=True for user admin
updating password for user admin
using existing role admin (b0c0987b16b342
Changed in kolla: | |
milestone: | none → liberty-rc1 |
status: | New → Triaged |
importance: | Undecided → High |
Changed in kolla: | |
milestone: | liberty-rc1 → liberty-rc2 |
Changed in kolla: | |
milestone: | liberty-rc2 → liberty-rc3 |
Changed in kolla: | |
importance: | High → Critical |
Changed in kolla: | |
milestone: | liberty-rc3 → mitaka-1 |
Changed in kolla: | |
status: | Fix Committed → Fix Released |
Just cutting and pasting comments from my gerrit submission I made incorrectly instead of filing a bug on launchpad:
Sam Yaple Aug 24 4:42 PM
Debian based systems do not use httpd, but instead apache2. Additionally, with different versions of packages and configuration options Debian-based systems *may* not be affected.
Either way, this will need to be in a block that only runs on Centos/RHEL/Fedora based systems.
Steven Dake Aug 24 10:15 PM
note there is an environment variable set in the docker files as follows;
https:/ /github. com/stackforge/ kolla/blob/ master/ docker_ templates/ base/Dockerfile .j2#L6