keystone charm has config-change hook failed

charm location and bundles has been placed as well.

jenkins@intelpod5jumphost:~/joid/ci$ pastebinit fetch-charms.sh
http://paste.ubuntu.com/17302391/
jenkins@intelpod5jumphost:~/joid/ci$ pastebinit bundles.yaml
http://paste.ubuntu.com/17302397/
jenkins@intelpod5jumphost:~/joid/ci$

Race condition on restart_pid_check. Apache should be down but is not:

2016-06-13 18:00:45 INFO config-changed Traceback (most recent call last):
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 707, in <module>
2016-06-13 18:00:45 INFO config-changed main()
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 700, in main
2016-06-13 18:00:45 INFO config-changed hooks.execute(sys.argv)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-06-13 18:00:45 INFO config-changed self._hooks[hook_name]()
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1574, in wrapped_f
2016-06-13 18:00:45 INFO config-changed restart_functions)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 486, in restart_on_change_helper
2016-06-13 18:00:45 INFO config-changed restart_functions[service_name](service_name)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 386, in restart_pid_check
2016-06-13 18:00:45 INFO config-changed check_pids_gone(ptable_string)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 42, in _retry_on_exception_inner_2
2016-06-13 18:00:45 INFO config-changed return f(*args, **kwargs)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 381, in check_pids_gone
2016-06-13 18:00:45 INFO config-changed assert(subprocess.call(["pgrep", svc_string]) == 1)
2016-06-13 18:00:45 INFO config-changed AssertionError

2016-06-13 18:21:09 INFO config-changed 16846
2016-06-13 18:21:09 INFO config-changed 16847
2016-06-13 18:21:09 INFO config-changed Traceback (most recent call last):
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 707, in <module>
2016-06-13 18:21:09 INFO config-changed main()
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 700, in main
2016-06-13 18:21:09 INFO config-changed hooks.execute(sys.argv)
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-06-13 18:21:09 INFO config-changed self._hooks[hook_name]()
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1574, in wrapped_f
2016-06-13 18:21:09 INFO config-changed restart_functions)
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 475, in restart_on_change_helper
2016-06-13 18:21:09 INFO config-changed ...

apache2 is failing to stop. I think the key to this bug is the line in the log:

2016-06-13 17:59:53 INFO config-changed Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.

I have yet to recreate this in the lab.

Narinder, if you see it again can you plese run the following commands on the keystone node and paste the results here:

systemctl status apache2.service
journalctl -xe

I encountered this same issue. Editing the /etc/hosts file and adding the hostname to 127.0.0.1 resolved this. Ultimately it was an issue on the server with the /etc/resolve.conf file. Once this was fixed I rebooted the containers and everything seems working correctly now