database dying can result in FAILED stacks with IN_PROGRESS resources
Bug #1561214 reported by
Steve Baker
This bug affects 1 person
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| OpenStack Heat |
Fix Released
|
Medium
|
Thomas Herve | ||
Bug Description
Steps to Reproduce:
1. Deploy overcloud, mariadb runs out of file descriptors which causes the deployment to fail, and leaves heat in a bad state.
Running out of file descriptors will be difficult to reproduce. This particular state can be replicated by setting some resources to IN_PROGRESS while their stacks are in an UPDATE_FAILED state.
I'm suggesting a heat-manage command which acts on a single stack and traverses all nested stacks to put any IN_PROGRESS things to FAILED, and clear hooks.
Revision history for this message
| Steven Hardy (shardy) wrote | #1 |
| Changed in heat: | |
| assignee: | nobody → Thomas Herve (therve) |
| milestone: | none → newton-1 |
Revision history for this message
| OpenStack Infra (hudson-openstack) wrote Fix proposed to heat (master) | #2 |
| Changed in heat: | |
| status: | New → In Progress |
| Changed in heat: | |
| importance: | Undecided → Medium |
Revision history for this message
| OpenStack Infra (hudson-openstack) wrote Fix merged to heat (master) | #3 |
| Changed in heat: | |
| status: | In Progress → Fix Released |
Revision history for this message
| Doug Hellmann (doug-hellmann) wrote Fix included in openstack/heat 7.0.0.0b1 | #4 |
To post a comment you must log in.
Is there any less destructive way we can handle this, as all FAILED resources will be replaced, even if they are OK?
I'm thinking something which uses similar logic to stack-check so that it actually observes state rather than unconditionally replacing everything - possibly not enough state to do that safely tho I guess.