[Glance] Long creating image via Horizon

Bug #1566760 reported by Evgeny Sikachev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Committed
High
Kairat Kushaev

Bug Description

ENVIRONMENT: MOS 9.0 ISO 135
shotgun2: http://paste.openstack.org/show/492858/

STEPS TO REPRODUCE:
1. Login to Horizon
2. Navigate to "images"
3. Click "Create image"
4. Name=test, Image Source=Image Location, Image Location=http://sahara-files.mirantis.com/images/upstream/mitaka/sahara-mitaka-cloudera-5.5.0-ubuntu.qcow2
5. Click "Create"

EXPECTED RESULT:
Image created

ACTUAL RESULT:
Image has "Saving" state ~2 days

Connection is correct, for download image manually need 2h

api.log: http://paste.openstack.org/show/493118/
registery.log: http://paste.openstack.org/show/493119/

Tags: area-glance
description: updated
tags: removed: area-horizon
Revision history for this message
Mike Fedosin (mfedosin) wrote :

Now it's active:
root@node-1:~# glance image-show 18a532b1-3b48-42d9-a3ee-abd8356b61bc
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 4409fb7cfc1ef18043ec6924630b42e7 |
| container_format | bare |
| created_at | 2016-04-06T09:17:16Z |
| description | |
| direct_url | rbd://96afe4ab-cb8d-4f7c-aaca-0690179f0b51/images/18a532b1-3b48-42d9-a3ee- |
| | abd8356b61bc/snap |
| disk_format | qcow2 |
| id | 18a532b1-3b48-42d9-a3ee-abd8356b61bc |
| min_disk | 0 |
| min_ram | 0 |
| name | nema |
| owner | 5bbe6a3b3ac042d7ac2bd55d8014001e |
| protected | False |
| size | 3838836736 |
| status | active |
| tags | [] |
| updated_at | 2016-04-06T09:39:21Z |
| virtual_size | None |
| visibility | private |

I think the cause of the issue is in network misconfiguration which makes image download very slow.

Btw, even after 2 days glance was able to download the image without error. It proves that this bug https://bugs.launchpad.net/mos/7.0.x/+bug/1456573 is not valid :)

Changed in mos:
status: New → Invalid
Revision history for this message
Evgeny Sikachev (esikachev) wrote :

if I try to get image via wget or download manually - downloading is correct. Network established but maybe slow. I think image must be downloaded in this case

Changed in mos:
status: Invalid → New
Revision history for this message
Evgeny Sikachev (esikachev) wrote :

Or image must be not in Saving state with described problem

Revision history for this message
Dina Belova (dbelova) wrote :

Evgeny, please collaborate on this issues investigation with Glance team on the bugged env. You need to grab tcpdump info and understand if that's really networking issue of something else.

Changed in mos:
status: New → Confirmed
importance: Undecided → High
assignee: MOS Glance (mos-glance) → Evgeny Sikachev (esikachev)
Revision history for this message
Mike Fedosin (mfedosin) wrote :

Sorry for too early decision with bug status. I know the cause of this bug and will propose the solution tomorrow.

Changed in mos:
assignee: Evgeny Sikachev (esikachev) → Mike Fedosin (mfedosin)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/glance (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Mike Fedosin <email address hidden>
Review: https://review.fuel-infra.org/19445

Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/glance (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/19445
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 6a60d5795225655bbf2e6bbc3d6a4b5ce0a5034a
Author: Mike Fedosin <email address hidden>
Date: Fri Apr 8 15:54:12 2016

Add trust support to Glance v1

This commit adds trusts support in Glance v1, that prevent
token expiration during longtime upload.

The algorighm is the following:
1. If galnce uses registry server to communicate with db,
create a trust
2. Upload an image
3. Try to update the image status to 'active' in registry
4. If trust has been created succcessfully and 401 occured
during updating the image status then renew the token and
try to update the image record in registry again.

If keystone server doesn't support trust functionality
then glance automatically returns to the basic workflow with
user token.

Change-Id: Ie0f7f9eeb7bd7281eeb126f837829877e5ecbdf4
Closes-Bug: #1566760

Changed in mos:
status: In Progress → Fix Committed
Ilya Menkov (imenkov)
tags: added: on-verification
Revision history for this message
Evgeny Sikachev (esikachev) wrote :

reproduced on 245 iso, mos 9.0. shotgun2 http://paste.openstack.org/show/495484/

Changed in mos:
status: Fix Committed → Confirmed
tags: removed: on-verification
Revision history for this message
Kairat Kushaev (kkushaev) wrote :

So we are using keystoneauth1 to upload plugin, namely:
https://github.com/openstack/keystoneauth/blob/80b58cc2f2aa73e2d3fc88ef2e27fa3db6ed0025/keystoneauth1/loading/conf.py#L122

The keystone_authtoken section contains the following:

identity_uri=http://192.168.0.2:35357/
admin_password=s3A8bXLf1DyOrVHOX3xPhRSv
admin_tenant_name=services
admin_user=glance

so there is no auth_type at all and because of that we cannot create trust.
Keystone folks, could you please clarify how this going to be fixed because we relied on that functionality when creating trusts for glance. Does keystoneauth1 need to be fixed or we need to fix pupper/Fuel to specify proper auth_type?

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

So main questions:
auth_type=None is expected in keystone.middelware configs?
If yes how to support previous functionality with uploading plugins from configurations?

Changed in mos:
assignee: Mike Fedosin (mfedosin) → MOS Keystone (mos-keystone)
Revision history for this message
Boris Bobrov (bbobrov) wrote :

> so there is no auth_type at all and because of that we cannot create trust.

What makes you think this is the reason for not being able to create a trust?

> Keystone folks, could you please clarify how this going to be fixed because we relied on that functionality when creating trusts for glance.

I don't see what needs to be fixed. If you think that something worked some time ago and then stopped working because of keystone (or keystoneauth), please provide more details (logs, snippets of code to run) and/or report a bug to keystoneauth.

Changed in mos:
assignee: MOS Keystone (mos-keystone) → Kairat Kushaev (kkushaev)
Revision history for this message
Kairat Kushaev (kkushaev) wrote :

So after additional investigation with bbobrov it turned out that upstream implementation of trusts requires auth_type to be specified. In 8.0 we used admin_user, admin_password to create a trustee but 9.0 (=upstream) requires Password auth plugin.
That was done because it is not recommended to use admin credentials to create trustee, it is better to use service credentials (in that case glance credentials) instead. I will create a separate bug to request change in glance-api.conf.

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

I have created a separate bug that describes the problem for puppet guys.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/glance (mcp/newton)

Fix proposed to branch: mcp/newton
Change author: Mike Fedosin <email address hidden>
Review: https://review.fuel-infra.org/33425

Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/glance (mcp/ocata)

Fix proposed to branch: mcp/ocata
Change author: Mike Fedosin <email address hidden>
Review: https://review.fuel-infra.org/34892

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/glance (11.0/ocata)

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33810

Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/glance (mcp/ocata)

Reviewed: https://review.fuel-infra.org/34892
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/ocata

Commit: 8de549bb3477946025103fdebd8e42366204ceb9
Author: Mike Fedosin <email address hidden>
Date: Mon Apr 24 12:19:21 2017

Add trust support to Glance v1

This commit adds trusts support in Glance v1, that prevent
token expiration during longtime upload.

The algorighm is the following:
1. If galnce uses registry server to communicate with db,
create a trust
2. Upload an image
3. Try to update the image status to 'active' in registry
4. If trust has been created succcessfully and 401 occured
during updating the image status then renew the token and
try to update the image record in registry again.

If keystone server doesn't support trust functionality
then glance automatically returns to the basic workflow with
user token.

Change-Id: Ie0f7f9eeb7bd7281eeb126f837829877e5ecbdf4
Closes-Bug: #1566760

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/glance (mcp/newton)

Reviewed: https://review.fuel-infra.org/33425
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/newton

Commit: 32c81e286353294596307e7414aee6d53a84a170
Author: Mike Fedosin <email address hidden>
Date: Tue Apr 25 08:53:07 2017

Add trust support to Glance v1

This commit adds trusts support in Glance v1, that prevent
token expiration during longtime upload.

The algorighm is the following:
1. If galnce uses registry server to communicate with db,
create a trust
2. Upload an image
3. Try to update the image status to 'active' in registry
4. If trust has been created succcessfully and 401 occured
during updating the image status then renew the token and
try to update the image record in registry again.

If keystone server doesn't support trust functionality
then glance automatically returns to the basic workflow with
user token.

Change-Id: Ie0f7f9eeb7bd7281eeb126f837829877e5ecbdf4
Closes-Bug: #1566760

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.