ocata->pike upgrade - keystone db sync cannot access keystone.log

Bug #1678011 reported by Jiří Stránský
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Invalid
High
Jiří Stránský

Bug Description

When upgrading Ocata to Pike, /var/log/keystone was owned by the keystone user, but /var/log/keystone/keystone.log in the logs docker volume was owned by root and so the keystone container failed when it tried to write to keystone.log.

Changed in tripleo:
importance: Critical → High
Revision history for this message
Jiří Stránský (jistr) wrote :

[root@overcloud-controller-0 _data]# pwd
/var/lib/docker/volumes/logs/_data
[root@overcloud-controller-0 _data]# ls -la keystone
total 4
drwxr-xr-x. 2 42425 42425 26 bře 31 11:25 .
drwxr-xr-x. 9 root root 103 bře 31 11:26 ..
-rw-r--r--. 1 root root 1400 bře 31 11:25 keystone.log

Revision history for this message
Jiří Stránský (jistr) wrote :

[root@overcloud-controller-0 ~]# docker ps -a | grep keystone
7b6063d3ab2e 192.168.24.1:8787/tripleoupstream/centos-binary-keystone:latest "kolla_start" 6 minutes ago Exited (1) 6 minutes ago keystone_db_sync
b041b7f603f8 192.168.24.1:8787/tripleoupstream/centos-binary-keystone:latest "/bin/bash -c 'mkdir " 6 minutes ago Exited (0) 6 minutes ago keystone-init-log
cd0b219b247a 192.168.24.1:8787/tripleoupstream/centos-binary-keystone:latest "kolla_start" 6 minutes ago Up 6 minutes keystone

[root@overcloud-controller-0 ~]# docker logs 7b6063d3ab2e
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
Traceback (most recent call last):
  File "/usr/bin/keystone-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/keystone/cmd/manage.py", line 44, in main
    cli.main(argv=sys.argv, config_files=config_files)
  File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 1371, in main
    keystone.conf.setup_logging()
  File "/usr/lib/python2.7/site-packages/keystone/conf/__init__.py", line 112, in setup_logging
    log.setup(CONF, 'keystone')
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 269, in setup
    _setup_logging_from_conf(conf, product_name, version)
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 366, in _setup_logging_from_conf
    filelog = file_handler(logpath)
  File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/keystone/keystone.log'

summary: - ocata->pike upgrade - keystone cannot access keystone.log
+ ocata->pike upgrade - keystone db sync cannot access keystone.log
Revision history for this message
Jiří Stránský (jistr) wrote :

Ah ha! Kolla entrypoint actually manages the ownership, but on a wrong file with respect to TripleO configs. It uses

/var/log/kolla/keystone/keystone.log

rather than

/var/log/keystone/keystone.log.

I don't think we should use the kolla path, as we want to reuse our non-containerized config values as much as possible.

Changed in tripleo:
assignee: nobody → Jiří Stránský (jistr)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/452200

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Jiri Stransky (<email address hidden>) on branch: master
Review: https://review.openstack.org/452200
Reason: With corrected repos, i haven't hit this since, it's probably not necessary.

Changed in tripleo:
status: In Progress → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.