Mirantis OpenStack

Can not remove hanged environment

Bug #1369551 reported by Sergey Murashov on 2014-09-15

This bug affects 5 people

	Status	Importance	Assigned to	Milestone
Mirantis OpenStack	Invalid	High	MOS Murano	Mirantis OpenStack 6.0-updates
6.0.x	Won't Fix	High	Denis Puchkin	Mirantis OpenStack 6.0-updates
6.1.x	Won't Fix	High	Denis Puchkin	Mirantis OpenStack 6.1-updates
7.0.x	Invalid	High	MOS Maintenance	Mirantis OpenStack 7.0-updates
8.0.x	Invalid	High	MOS Murano	Mirantis OpenStack 6.0-updates

Bug Description

Steps to reproduce:
1. Install Murano
2. Log in WebUI
3. Navigate to Murano>Application Catalog>Environments
4. Create environment
5. Add app
6. Send environment to deploy
7. Wait until engine will execute "io.murano.system.Agent.call"
8. Corrupt connection between agent and engine

Actual result:
Environment can not be removed via dashboard, "delete environment" button doesn't appear
Also environment can not be removed via pythonclient
user@ubuntu:~/devstack$ murano environment-list
+----------------------------------+---------+---------------------+---------------------+
| ID | Name | Created | Updated |
+----------------------------------+---------+---------------------+---------------------+
| 24e5e25c4b204c8c857e996dfe25b38c | saewdgf | 2014-09-15T12:16:37 | 2014-09-15T12:16:37 |
+----------------------------------+---------+---------------------+---------------------+
user@ubuntu:~/devstack$ murano environment-delete 24e5e25c4b204c8c857e996dfe25b38c
<html>
<head>
  <title>403 Forbidden</title>
</head>
<body>
  <h1>403 Forbidden</h1>
  Could not open session for environment <EnvId: 24e5e25c4b204c8c857e996dfe25b38c>,environment has deploying status.<br /><br />

</body>
</html> (HTTP 403)

See original description

Tags:

Sergey Murashov (smurashov) on 2014-09-15

description:	updated
Changed in murano:
importance:	High → Critical

Stan Lagun (slagun) on 2014-09-18

Changed in murano:
milestone:	juno-rc1 → none

Serg Melikyan (smelikyan) on 2014-09-24

Changed in murano:
milestone:	none → juno-rc1
milestone:	juno-rc1 → none
status:	New → Confirmed
importance:	Critical → High

Ekaterina Chernova (efedorova) on 2014-12-25

Changed in murano:
status:	Confirmed → Won't Fix

Revision history for this message

Ekaterina Chernova (efedorova) wrote on 2014-12-25:

This bug will be fixed during "Add timeouts for murano-agent calls" blueprint implementtation [https://blueprints.launchpad.net/murano/+spec/murano-agent-timeouts]

Denis Ipatov (dipatov) on 2015-10-01

tags:

added: customer-found

Serg Melikyan (smelikyan) on 2015-10-01

affects:	murano → mos
Changed in mos:
milestone:	none → 6.0-updates
status:	Won't Fix → Confirmed
tags:	added: support

Roman Podoliaka (rpodolyaka) on 2015-10-01

Changed in mos:
assignee:	nobody → MOS Murano (mos-murano)

Revision history for this message

Roman Rufanov (rrufanov) wrote on 2015-10-23:

Customer found on MOS 6.0 - need fix and in all subsequent versions. Thanks !

Revision history for this message

Kirill Zaitsev (kzaitsev) wrote on 2015-10-27:

I believe that this is fixed in 7.0 and in 8.0 with the abandon "environment feature"

Serg, you've updated 6.0 from Won't Fix to Confirmed, are we backporting that feature to 6.0/6.1 ?

Revision history for this message

Serg Melikyan (smelikyan) wrote on 2015-11-02:

Many customers are stuck with hanged environments, I think to fix this behavior we should backport 2 features:
- https://blueprints.launchpad.net/murano/+spec/murano-agent-timeouts (already in 6.1 branch, but is not present in 6.0)
- https://blueprints.launchpad.net/murano/+spec/environment-abandon (added in MOS 8.0)

@Vitaly please confirm that this is feasible for your team.

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2015-11-06:

Typically we avoid to backport features. Is there a workaround for this issue, for example some manual steps to remove environment (like manually terminate VMs, etc)?

Revision history for this message

greg pryzby (gpryzby-k) wrote on 2015-11-08: Re: [Bug 1369551] Re: Can not remove hanged environment

I am ok w the decision. If it is fixed in 7, that works for me.

On Fri, Nov 6, 2015 at 4:25 PM Vitaly Sedelnik <email address hidden>
wrote:

> Typically we avoid to backport features. Is there a workaround for this
> issue, for example some manual steps to remove environment (like
> manually terminate VMs, etc)?
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1442910).
> https://bugs.launchpad.net/bugs/1369551
>
> Title:
> Can not remove hanged environment
>
> Status in Mirantis OpenStack:
> Invalid
> Status in Mirantis OpenStack 6.0.x series:
> Confirmed
> Status in Mirantis OpenStack 6.1.x series:
> New
> Status in Mirantis OpenStack 7.0.x series:
> Invalid
> Status in Mirantis OpenStack 8.0.x series:
> Invalid
>
> Bug description:
> Steps to reproduce:
> 1. Install Murano
> 2. Log in WebUI
> 3. Navigate to Murano>Application Catalog>Environments
> 4. Create environment
> 5. Add app
> 6. Send environment to deploy
> 7. Wait until engine will execute "io.murano.system.Agent.call"
> 8. Corrupt connection between agent and engine
>
> Actual result:
> Environment can not be removed via dashboard, "delete environment"
> button doesn't appear
> Also environment can not be removed via pythonclient
> user@ubuntu:~/devstack$ murano environment-list
>
> +----------------------------------+---------+---------------------+---------------------+
> | ID | Name | Created |
> Updated |
>
> +----------------------------------+---------+---------------------+---------------------+
> | 24e5e25c4b204c8c857e996dfe25b38c | saewdgf | 2014-09-15T12:16:37 |
> 2014-09-15T12:16:37 |
>
> +----------------------------------+---------+---------------------+---------------------+
> user@ubuntu:~/devstack$ murano environment-delete
> 24e5e25c4b204c8c857e996dfe25b38c
> <html>
> <head>
> <title>403 Forbidden</title>
> </head>
> <body>
> <h1>403 Forbidden</h1>
> Could not open session for environment <EnvId:
> 24e5e25c4b204c8c857e996dfe25b38c>,environment has deploying status.<br
> /><br />
>
> </body>
> </html> (HTTP 403)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mos/+bug/1369551/+subscriptions
>
--
greg pryzby <email address hidden>
+1-929-267-3676 +1-9292-OPENSOurce

I am ok w the decision. If it is fixed in 7, that works for me.

On Fri, Nov 6, 2015 at 4:25 PM Vitaly Sedelnik <1369551@bugs.launchpad.net>
wrote:

> Typically we avoid to backport features. Is there a workaround for this
> issue, for example some manual steps to remove environment (like
> manually terminate VMs, etc)?
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1442910).
> https://bugs.launchpad.net/bugs/1369551
>
> Title:
>   Can not remove hanged environment
>
> Status in Mirantis OpenStack:
>   Invalid
> Status in Mirantis OpenStack 6.0.x series:
>   Confirmed
> Status in Mirantis OpenStack 6.1.x series:
>   New
> Status in Mirantis OpenStack 7.0.x series:
>   Invalid
> Status in Mirantis OpenStack 8.0.x series:
>   Invalid
>
> Bug description:
>   Steps to reproduce:
>   1. Install Murano
>   2. Log in WebUI
>   3. Navigate to Murano>Application Catalog>Environments
>   4. Create environment
>   5. Add app
>   6. Send environment to deploy
>   7. Wait until engine will execute "io.murano.system.Agent.call"
>   8. Corrupt connection between agent and engine
>
>   Actual result:
>   Environment can not be removed via dashboard, "delete environment"
> button doesn't appear
>   Also environment can not be removed via pythonclient
>   user@ubuntu:~/devstack$ murano environment-list
>
> +----------------------------------+---------+---------------------+---------------------+
>   | ID                               | Name    | Created             |
> Updated             |
>
> +----------------------------------+---------+---------------------+---------------------+
>   | 24e5e25c4b204c8c857e996dfe25b38c | saewdgf | 2014-09-15T12:16:37 |
> 2014-09-15T12:16:37 |
>
> +----------------------------------+---------+---------------------+---------------------+
>   user@ubuntu:~/devstack$ murano environment-delete
> 24e5e25c4b204c8c857e996dfe25b38c
>   <html>
>    <head>
>     <title>403 Forbidden</title>
>    </head>
>    <body>
>     <h1>403 Forbidden</h1>
>     Could not open session for environment &lt;EnvId:
> 24e5e25c4b204c8c857e996dfe25b38c&gt;,environment has deploying status.<br
> /><br />
>
>    </body>
>   </html> (HTTP 403)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mos/+bug/1369551/+subscriptions
>
-- 
greg pryzby                                              greg@mirantis.com
+1-929-267-3676                               +1-9292-OPENSOurce

Revision history for this message

Nikolay Starodubtsev (starodubcevna) wrote on 2015-11-09:

I guess that we have a workaround:
1) Clear all resources manually
2) Remove environment from db
It should work.

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2015-11-24:

Won't Fix for 6.0-updates and 6.1-updates per comments #6 and #7. If this issue happens in customer environment the recommendation is to proceed with the above workaround.

Vitaly Sedelnik (vsedelnik) on 2016-02-12

tags:

added: wontfix-workaround

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.