Magnum

Swarm cluster should not be created if bay create gets failed for some reason

Bug #1479089 reported by Manjeet Singh Bhatia
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Magnum
Invalid
Low
Unassigned
Magnum liberty-2 "liberty-2"

Bug Description

I was creating bays and some of the bay_create got failed but still it was creating swarm manager and node.

steps to reproduce:

for some reason make create_bay fail
check bay-list if bay created has create_failed status.

then check nova list for manager and nodes of the cluster.

try ssh to nodes and manager of that cluster.

See original description
Manjeet Singh Bhatia (manjeet-s-bhatia)
summary: - Swarm cluster should be created if bay create gets failed for some
+ Swarm cluster should not be created if bay create gets failed for some
reason
Manjeet Singh Bhatia (manjeet-s-bhatia)
description: updated
Revision history for this message
Adrian Otto (aotto) wrote :

What are the steps to reproduce a bay failure?

Changed in magnum:
importance: Undecided → Low
milestone: none → liberty-2
Revision history for this message
Manjeet Singh Bhatia (manjeet-s-bhatia) wrote :

Use some sort of proxy and do not configure it in swarm cluster.
you will always see create_bay failure. Qiao has written a blog to set up proxy.

or do something wrong in /etc/heat/policy.json regarding stack_create role. or stack global_index.

Revision history for this message
Manjeet Singh Bhatia (manjeet-s-bhatia) wrote :

steps to reproduce this choose a system that has limited ram say 16gb. and try creating bays again and again.
at some point of time there may be the case that swarm manager get created successfully but one or
more node may not get completed due to limited resources.

You will see bay create status create_failed.

there will be a active swarm manager.

If bay_create fails there is no need to keep even database entry for cluster nodes. user can figure out from bay_status create_failed.

or there should be some sort of notification if there are not enough resources.

Revision history for this message
Eli Qiao (taget-9) wrote :

hi Manjeet
Magnum only talks to Heat to get the cluster's status, if we want to get more information if that bay is worth to keep or delete, then please investigate heat has such api which can be called.

Another thinking is if you delete the whole cluster when bay created failed, I think that is kinds of arbitrary.
I would propose that add a new option to allow user decide if they want to delete the heat cluster.

Revision history for this message
Eli Qiao (taget-9) wrote :

A new bp is registered to allow magnum to cleanup the cluster when bay created failed.
https://blueprints.launchpad.net/magnum/+spec/option-to-cleanup-cluster

Revision history for this message
Eli Qiao (taget-9) wrote :

discussed in Tokyo summit, Magnum won't do this cleanup work.

Changed in magnum:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.