"swift_store_auth_address = ://:/v2.0/" in glance-api.conf

Bug #1599298 reported by Andreas Hasenack
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Autopilot Log Analyser
Fix Committed
Low
Andreas Hasenack
OpenStack Glance Charm
Incomplete
Low
Unassigned
glance (Juju Charms Collection)
Invalid
Low
Unassigned

Bug Description

landscape reference: https://ci.lscape.net/job/landscape-system-tests/3027

ceph/swift autopilot deployment, using 16.04 charms.

A "juju run --all /bin/true" call that landscape makes to wait for relations to settle got stuck in a few units, glance/0 among them.

Inspection showed that /etc/glance/glance-api.conf had the [glance_store] section not fully rendered:
[glance_store]
filesystem_store_datadir = /var/lib/glance/images/
stores = glance.store.filesystem.Store,glance.store.http.Store,glance.store.swift.Store
default_store = swift
swift_store_auth_version = 2
swift_store_auth_address = ://:/v2.0/
swift_store_user = :
swift_store_key =
swift_store_create_container_on_put = True
swift_store_container = glance
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 200
swift_enable_snet = False

glance/1 and glance/2 had it correct:
[glance_store]
filesystem_store_datadir = /var/lib/glance/images/
stores = glance.store.filesystem.Store,glance.store.http.Store,glance.store.swift.Store
default_store = swift
swift_store_auth_version = 2
swift_store_auth_address = http://10.96.16.243:5000/v2.0/
swift_store_user = services:glance
swift_store_key = mTJTkHG9zkFJbgsrtWMpLfdNjWFVKks8WYMcFqFhNLgGrRg5w2h43jZJh3rzg2Zq
swift_store_create_container_on_put = True
swift_store_container = glance
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 200
swift_enable_snet = False

Logs and config files for glance/{0,1,2} attached. We also have the same for all the other units if needed, but it's a 89Mb zip file.

Tags: landscape
Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Changed in autopilot-log-analyser:
status: New → In Progress
assignee: nobody → Andreas Hasenack (ahasenack)
importance: Undecided → Low
tags: removed: kanban-cross-team
Changed in autopilot-log-analyser:
status: In Progress → Fix Committed
Revision history for this message
James Page (james-page) wrote :

"A "juju run --all /bin/true" call that landscape makes to wait for relations to settle got stuck in a few units, glance/0 among them."

would indicate that the glance/0 hook execution had not completed right? in which case I might expect that section to be incomplete

Revision history for this message
James Page (james-page) wrote :

The template should probably be better:

{% if swift_store -%}
swift_store_auth_version = 2
swift_store_auth_address = {{ service_protocol }}://{{ service_host }}:{{ service_port }}/v2.0/
swift_store_user = {{ admin_tenant_name }}:{{ admin_user }}
swift_store_key = {{ admin_password }}
swift_store_create_container_on_put = True
swift_store_container = glance
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 200
swift_enable_snet = False
{% endif -%}

as it relies on keystone data, we might make that conditional on service_host being present as well before writing things out (but that's a low priority change IMHO).

Changed in glance (Juju Charms Collection):
importance: Undecided → Low
status: New → Incomplete
Revision history for this message
James Page (james-page) wrote :

glance/0 was completing a db sync when the log dump was taken:

2016-07-04 00:23:27 INFO juju-log shared-db:37: Cluster leader, performing db sync
2016-07-04 00:23:28 INFO shared-db-relation-changed Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
2016-07-04 00:23:28 INFO shared-db-relation-changed /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py:1056: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
2016-07-04 00:23:28 INFO shared-db-relation-changed expire_on_commit=expire_on_commit, _conf=conf)

It looks like that was blocking - maybe that was the case for other services as well which might indicate some sort of underlying DB issue.

Revision history for this message
James Page (james-page) wrote :

I can't see into this problem any further given the provided data; Andreas did you figure out why hooks where not completing? and have you seen this issue since?

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Not exactly. I have another failed run from yesterday which has the incomplete URL in glance-api.conf, but what made the run fail was something different and it just so happens that at that point in time the rendering of glance-api.conf was incomplete in the same way.

James Page (james-page)
Changed in charm-glance:
importance: Undecided → Low
status: New → Incomplete
Changed in glance (Juju Charms Collection):
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.