Tempest virtual env constraints env variable not reset when using master constraints

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/devstack/+/871942

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/devstack/+/871943

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/devstack/+/871944

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/devstack/+/871945

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/grenade/+/871946

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/grenade/+/871947

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/grenade/+/871948

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/grenade/+/871949

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/grenade/+/871955

Reviewed: https://review.opendev.org/c/openstack/devstack/+/871942
Committed: https://opendev.org/openstack/devstack/commit/7fe998109bda8cdd5cb5ba4a0e02c6c83cb0566d
Submitter: "Zuul (22348)"
Branch: master

commit 7fe998109bda8cdd5cb5ba4a0e02c6c83cb0566d
Author: Ghanshyam Mann <email address hidden>
Date: Thu Jan 26 22:28:07 2023 -0600

    Fix setting the tempest virtual env constraints env var

    Devstack set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
    which are used to use the constraints during Tempest virtual env installation.
    Those env var are set to non-master constraint when we need to use non-master
    constraints but when we need to use the master constraints we do not set/reset
    them point to master constraints. This create the issue when running the grenade
    job where we run Tempest on the old devstack as well as in the new devstack.
    When tempest is installed on old devstack then old tempest is used and it sets
    these env var to stable/<branch> constraints (this is the case when old devstack
    (the stable branch is in EM phase) uses the old tempest not the master tempest),
    all good till now. But the problem comes when in the same grenade script run
    upgrade-tempest install the master tempest (when new devstack branches are in
    the 'supported' phase and use the master tempest means) and are supposed to use
    the master constraints. But the TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE env
    var set by old tempest is used by the tempest and due to a mismatch in constraints
    it fails.

    This happened when we tried to pin the stable/wallaby with Tempest 29.0.0
    - https://review.opendev.org/c/openstack/devstack/+/871782

    and table/xena grenade job failed (stable/xena use master tempest and supposed
    to use master constraints)
    - https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641

    We should set/reset those constraint env var to master constraints if configuration
    tell devstack to use the master constraints.

    [1] https://github.com/openstack/devstack/blob/71c3c40c269a50303247855319d1d3a5d30f6773/lib/tempest#L124

    Closes-Bug: #2003993
    Change-Id: I5e938139b47f443a4c358415d0d4dcf6549cd085

Reviewed: https://review.opendev.org/c/openstack/devstack/+/871943
Committed: https://opendev.org/openstack/devstack/commit/30a7d790b6bf45bbcc6333008621b093c84055d1
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 30a7d790b6bf45bbcc6333008621b093c84055d1
Author: Ghanshyam Mann <email address hidden>
Date: Thu Jan 26 22:28:07 2023 -0600

    Fix setting the tempest virtual env constraints env var

    Devstack set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
    which are used to use the constraints during Tempest virtual env installation.
    Those env var are set to non-master constraint when we need to use non-master
    constraints but when we need to use the master constraints we do not set/reset
    them point to master constraints. This create the issue when running the grenade
    job where we run Tempest on the old devstack as well as in the new devstack.
    When tempest is installed on old devstack then old tempest is used and it sets
    these env var to stable/<branch> constraints (this is the case when old devstack
    (the stable branch is in EM phase) uses the old tempest not the master tempest),
    all good till now. But the problem comes when in the same grenade script run
    upgrade-tempest install the master tempest (when new devstack branches are in
    the 'supported' phase and use the master tempest means) and are supposed to use
    the master constraints. But the TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE env
    var set by old tempest is used by the tempest and due to a mismatch in constraints
    it fails.

    This happened when we tried to pin the stable/wallaby with Tempest 29.0.0
    - https://review.opendev.org/c/openstack/devstack/+/871782

    and table/xena grenade job failed (stable/xena use master tempest and supposed
    to use master constraints)
    - https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641

    We should set/reset those constraint env var to master constraints if configuration
    tell devstack to use the master constraints.

    [1] https://github.com/openstack/devstack/blob/71c3c40c269a50303247855319d1d3a5d30f6773/lib/tempest#L124

    Closes-Bug: #2003993
    Change-Id: I5e938139b47f443a4c358415d0d4dcf6549cd085
    (cherry picked from commit 7fe998109bda8cdd5cb5ba4a0e02c6c83cb0566d)

Reviewed: https://review.opendev.org/c/openstack/devstack/+/871944
Committed: https://opendev.org/openstack/devstack/commit/febdb122e46ad22b5481c03c33fcd4130593d116
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit febdb122e46ad22b5481c03c33fcd4130593d116
Author: Ghanshyam Mann <email address hidden>
Date: Thu Jan 26 22:28:07 2023 -0600

    Fix setting the tempest virtual env constraints env var

    Devstack set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
    which are used to use the constraints during Tempest virtual env installation.
    Those env var are set to non-master constraint when we need to use non-master
    constraints but when we need to use the master constraints we do not set/reset
    them point to master constraints. This create the issue when running the grenade
    job where we run Tempest on the old devstack as well as in the new devstack.
    When tempest is installed on old devstack then old tempest is used and it sets
    these env var to stable/<branch> constraints (this is the case when old devstack
    (the stable branch is in EM phase) uses the old tempest not the master tempest),
    all good till now. But the problem comes when in the same grenade script run
    upgrade-tempest install the master tempest (when new devstack branches are in
    the 'supported' phase and use the master tempest means) and are supposed to use
    the master constraints. But the TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE env
    var set by old tempest is used by the tempest and due to a mismatch in constraints
    it fails.

    This happened when we tried to pin the stable/wallaby with Tempest 29.0.0
    - https://review.opendev.org/c/openstack/devstack/+/871782

    and table/xena grenade job failed (stable/xena use master tempest and supposed
    to use master constraints)
    - https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641

    We should set/reset those constraint env var to master constraints if configuration
    tell devstack to use the master constraints.

    [1] https://github.com/openstack/devstack/blob/71c3c40c269a50303247855319d1d3a5d30f6773/lib/tempest#L124

    Closes-Bug: #2003993
    Change-Id: I5e938139b47f443a4c358415d0d4dcf6549cd085
    (cherry picked from commit 7fe998109bda8cdd5cb5ba4a0e02c6c83cb0566d)
    (cherry picked from commit 30a7d790b6bf45bbcc6333008621b093c84055d1)

Reviewed: https://review.opendev.org/c/openstack/devstack/+/871945
Committed: https://opendev.org/openstack/devstack/commit/cb891834beb0f3b63ece60c1735555c49b769d01
Submitter: "Zuul (22348)"
Branch: stable/xena

commit cb891834beb0f3b63ece60c1735555c49b769d01
Author: Ghanshyam Mann <email address hidden>
Date: Thu Jan 26 22:28:07 2023 -0600

    Fix setting the tempest virtual env constraints env var

    Devstack set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
    which are used to use the constraints during Tempest virtual env installation.
    Those env var are set to non-master constraint when we need to use non-master
    constraints but when we need to use the master constraints we do not set/reset
    them point to master constraints. This create the issue when running the grenade
    job where we run Tempest on the old devstack as well as in the new devstack.
    When tempest is installed on old devstack then old tempest is used and it sets
    these env var to stable/<branch> constraints (this is the case when old devstack
    (the stable branch is in EM phase) uses the old tempest not the master tempest),
    all good till now. But the problem comes when in the same grenade script run
    upgrade-tempest install the master tempest (when new devstack branches are in
    the 'supported' phase and use the master tempest means) and are supposed to use
    the master constraints. But the TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE env
    var set by old tempest is used by the tempest and due to a mismatch in constraints
    it fails.

    This happened when we tried to pin the stable/wallaby with Tempest 29.0.0
    - https://review.opendev.org/c/openstack/devstack/+/871782

    and table/xena grenade job failed (stable/xena use master tempest and supposed
    to use master constraints)
    - https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641

    We should set/reset those constraint env var to master constraints if configuration
    tell devstack to use the master constraints.

    Making openstacksdk-functional-devstack job non voting as this is
    failing 100% on stable/xena
    - https://zuul.openstack.org/builds?job_name=openstacksdk-functional-devstack&branch=stable%2Fxena&skip=0

    [1] https://github.com/openstack/devstack/blob/71c3c40c269a50303247855319d1d3a5d30f6773/lib/tempest#L124

    Closes-Bug: #2003993
    Change-Id: I5e938139b47f443a4c358415d0d4dcf6549cd085
    (cherry picked from commit 7fe998109bda8cdd5cb5ba4a0e02c6c83cb0566d)
    (cherry picked from commit 30a7d790b6bf45bbcc6333008621b093c84055d1)
    (cherry picked from commit febdb122e46ad22b5481c03c33fcd4130593d116)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/devstack/+/872285

Reviewed: https://review.opendev.org/c/openstack/devstack/+/872285
Committed: https://opendev.org/openstack/devstack/commit/f5e83ee14a36df11a3180f7f7c8b28d3aa3ed9fa
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f5e83ee14a36df11a3180f7f7c8b28d3aa3ed9fa
Author: Ghanshyam Mann <email address hidden>
Date: Thu Jan 26 22:28:07 2023 -0600

    Fix setting the tempest virtual env constraints env var

    Devstack set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
    which are used to use the constraints during Tempest virtual env installation.
    Those env var are set to non-master constraint when we need to use non-master
    constraints but when we need to use the master constraints we do not set/reset
    them point to master constraints. This create the issue when running the grenade
    job where we run Tempest on the old devstack as well as in the new devstack.
    When tempest is installed on old devstack then old tempest is used and it sets
    these env var to stable/<branch> constraints (this is the case when old devstack
    (the stable branch is in EM phase) uses the old tempest not the master tempest),
    all good till now. But the problem comes when in the same grenade script run
    upgrade-tempest install the master tempest (when new devstack branches are in
    the 'supported' phase and use the master tempest means) and are supposed to use
    the master constraints. But the TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE env
    var set by old tempest is used by the tempest and due to a mismatch in constraints
    it fails.

    This happened when we tried to pin the stable/wallaby with Tempest 29.0.0
    - https://review.opendev.org/c/openstack/devstack/+/871782

    and table/xena grenade job failed (stable/xena use master tempest and supposed
    to use master constraints)
    - https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641

    We should set/reset those constraint env var to master constraints if configuration
    tell devstack to use the master constraints.

    Making openstacksdk-functional-devstack job non voting as this is
    failing 100% on <=stable/xena, depends-On fix that

    Depends-On: https://review.opendev.org/c/openstack/python-openstackclient/+/872341

    [1] https://github.com/openstack/devstack/blob/71c3c40c269a50303247855319d1d3a5d30f6773/lib/tempest#L124

    Closes-Bug: #2003993
    Change-Id: I5e938139b47f443a4c358415d0d4dcf6549cd085
    (cherry picked from commit 7fe998109bda8cdd5cb5ba4a0e02c6c83cb0566d)
    (cherry picked from commit 30a7d790b6bf45bbcc6333008621b093c84055d1)
    (cherry picked from commit febdb122e46ad22b5481c03c33fcd4130593d116)
    (cherry picked from commit cb891834beb0f3b63ece60c1735555c49b769d01)

Reviewed: https://review.opendev.org/c/openstack/grenade/+/871946
Committed: https://opendev.org/openstack/grenade/commit/01e143de49182da1880af2fe77c3d1573011a82e
Submitter: "Zuul (22348)"
Branch: master

commit 01e143de49182da1880af2fe77c3d1573011a82e
Author: Ghanshyam Mann <email address hidden>
Date: Thu Jan 26 23:12:59 2023 -0600

    Fix setting the tempest virtual env constraints env var

    Grenade set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
    which are used to use the constraints during Tempest virtual env
    installation. Those env var are set to non-master constraint when
    we need to use non-master constraints but when we need to use the
    master constraints we do not set/reset them point to master constraints.

    This create the issue when grenade script install the tempest second time
    on new devstack where we need to use master constraints but env var
    TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE set previously in tempest
    installation in old devstack point to the stable constraints.

    One case where it started failing when we tried to pin the stable/wallaby with
    Tempest 29.0.0
    - https://review.opendev.org/c/openstack/devstack/+/871782

    and stable/xena grenade job failed (stable/xena use master tempest and supposed
    to use master constraints)
    - https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641

    We should set/reset those constraint env var to master constraints if configuration
    tell to use the master constraints.

    Closes-Bug: #2003993
    Change-Id: I8c623008292f3b2dd2cd6d596feea552194acd89

This issue was fixed in the openstack/heat 20.0.0.0rc1 release candidate.

Reviewed: https://review.opendev.org/c/openstack/grenade/+/871947
Committed: https://opendev.org/openstack/grenade/commit/e17b3a2119f55d7939a372c2bedfd55acd299a27
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e17b3a2119f55d7939a372c2bedfd55acd299a27
Author: Ghanshyam Mann <email address hidden>
Date: Thu Jan 26 23:12:59 2023 -0600

    Fix setting the tempest virtual env constraints env var

    Grenade set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
    which are used to use the constraints during Tempest virtual env
    installation. Those env var are set to non-master constraint when
    we need to use non-master constraints but when we need to use the
    master constraints we do not set/reset them point to master constraints.

    This create the issue when grenade script install the tempest second time
    on new devstack where we need to use master constraints but env var
    TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE set previously in tempest
    installation in old devstack point to the stable constraints.

    One case where it started failing when we tried to pin the stable/wallaby with
    Tempest 29.0.0
    - https://review.opendev.org/c/openstack/devstack/+/871782

    and stable/xena grenade job failed (stable/xena use master tempest and supposed
    to use master constraints)
    - https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641

    We should set/reset those constraint env var to master constraints if configuration
    tell to use the master constraints.

    Closes-Bug: #2003993
    Change-Id: I8c623008292f3b2dd2cd6d596feea552194acd89
    (cherry picked from commit 01e143de49182da1880af2fe77c3d1573011a82e)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/grenade/+/879113

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/grenade/+/879119

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/grenade/+/879123

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/grenade/+/879124

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/grenade/+/879125

Reviewed: https://review.opendev.org/c/openstack/grenade/+/879113
Committed: https://opendev.org/openstack/grenade/commit/4c9c059883158fd6bf3cb79ca89d52003a63b7fe
Submitter: "Zuul (22348)"
Branch: master

commit 4c9c059883158fd6bf3cb79ca89d52003a63b7fe
Author: Ghanshyam Mann <email address hidden>
Date: Thu Mar 30 20:58:04 2023 -0500

    Fix tempest venv constraints for target branch tempest run

    We fixed the tempest venv constraints setting for tempest test run
    on base branch but forgot to do the same for target branch test run.
    It cause issue when base and target branch have different Tempest
    and constraints to use (for example stable/wallaby use old tmepest
    and stable/wallaby constraints but stable/xena use tempest master
    and master constraints), in such cases we need to set proper constraints
    defined in devstack and then run tempest.

    We do not see this issue as none of the job run tempest on target except
    heat (https://review.opendev.org/c/openstack/heat/+/872055/5).

    I reproduced the issue in grenade stable/xena by enaling the tempest run
    on target:
    - https://review.opendev.org/c/openstack/grenade/+/878247/1
    - https://zuul.opendev.org/t/openstack/build/1b503d359717459c9c77010608068e27/log/controller/logs/grenade.sh_log.txt#17184

    Closes-Bug: 2003993
    Change-Id: I05fcdb5045a33997bd1a4c340a16458d88399f5f

Reviewed: https://review.opendev.org/c/openstack/grenade/+/879119
Committed: https://opendev.org/openstack/grenade/commit/62ea9b6ba35ae28c299a4366e81f516c503a518c
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 62ea9b6ba35ae28c299a4366e81f516c503a518c
Author: Ghanshyam Mann <email address hidden>
Date: Thu Mar 30 20:58:04 2023 -0500

    Fix tempest venv constraints for target branch tempest run

    We fixed the tempest venv constraints setting for tempest test run
    on base branch but forgot to do the same for target branch test run.
    It cause issue when base and target branch have different Tempest
    and constraints to use (for example stable/wallaby use old tmepest
    and stable/wallaby constraints but stable/xena use tempest master
    and master constraints), in such cases we need to set proper constraints
    defined in devstack and then run tempest.

    We do not see this issue as none of the job run tempest on target except
    heat (https://review.opendev.org/c/openstack/heat/+/872055/5).

    I reproduced the issue in grenade stable/xena by enaling the tempest run
    on target:
    - https://review.opendev.org/c/openstack/grenade/+/878247/1
    - https://zuul.opendev.org/t/openstack/build/1b503d359717459c9c77010608068e27/log/controller/logs/grenade.sh_log.txt#17184

    Closes-Bug: 2003993
    Change-Id: I05fcdb5045a33997bd1a4c340a16458d88399f5f
    (cherry picked from commit 4c9c059883158fd6bf3cb79ca89d52003a63b7fe)

Reviewed: https://review.opendev.org/c/openstack/grenade/+/879123
Committed: https://opendev.org/openstack/grenade/commit/ff45be406ae30193339d3e379f815a61c48e77c7
Submitter: "Zuul (22348)"
Branch: stable/zed

commit ff45be406ae30193339d3e379f815a61c48e77c7
Author: Ghanshyam Mann <email address hidden>
Date: Thu Mar 30 20:58:04 2023 -0500

    Fix tempest venv constraints for target branch tempest run

    We fixed the tempest venv constraints setting for tempest test run
    on base branch but forgot to do the same for target branch test run.
    It cause issue when base and target branch have different Tempest
    and constraints to use (for example stable/wallaby use old tmepest
    and stable/wallaby constraints but stable/xena use tempest master
    and master constraints), in such cases we need to set proper constraints
    defined in devstack and then run tempest.

    We do not see this issue as none of the job run tempest on target except
    heat (https://review.opendev.org/c/openstack/heat/+/872055/5).

    I reproduced the issue in grenade stable/xena by enaling the tempest run
    on target:
    - https://review.opendev.org/c/openstack/grenade/+/878247/1
    - https://zuul.opendev.org/t/openstack/build/1b503d359717459c9c77010608068e27/log/controller/logs/grenade.sh_log.txt#17184

    Closes-Bug: 2003993
    Change-Id: I05fcdb5045a33997bd1a4c340a16458d88399f5f
    (cherry picked from commit 4c9c059883158fd6bf3cb79ca89d52003a63b7fe)
    (cherry picked from commit 62ea9b6ba35ae28c299a4366e81f516c503a518c)

Reviewed: https://review.opendev.org/c/openstack/grenade/+/879124
Committed: https://opendev.org/openstack/grenade/commit/cd782d27d56b0d087d7806df4bad459f44a03656
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit cd782d27d56b0d087d7806df4bad459f44a03656
Author: Ghanshyam Mann <email address hidden>
Date: Thu Mar 30 20:58:04 2023 -0500

    Fix tempest venv constraints for target branch tempest run

    We fixed the tempest venv constraints setting for tempest test run
    on base branch but forgot to do the same for target branch test run.
    It cause issue when base and target branch have different Tempest
    and constraints to use (for example stable/wallaby use old tmepest
    and stable/wallaby constraints but stable/xena use tempest master
    and master constraints), in such cases we need to set proper constraints
    defined in devstack and then run tempest.

    We do not see this issue as none of the job run tempest on target except
    heat (https://review.opendev.org/c/openstack/heat/+/872055/5).

    I reproduced the issue in grenade stable/xena by enaling the tempest run
    on target:
    - https://review.opendev.org/c/openstack/grenade/+/878247/1
    - https://zuul.opendev.org/t/openstack/build/1b503d359717459c9c77010608068e27/log/controller/logs/grenade.sh_log.txt#17184

    Extra on backport:
    We need heat fixes to pass the heat multinode grenade job
    Depends-On: https://review.opendev.org/c/openstack/heat/+/872055
    Depends-On: https://review.opendev.org/c/openstack/heat/+/872054

    Closes-Bug: 2003993
    Change-Id: I05fcdb5045a33997bd1a4c340a16458d88399f5f
    (cherry picked from commit 4c9c059883158fd6bf3cb79ca89d52003a63b7fe)
    (cherry picked from commit 62ea9b6ba35ae28c299a4366e81f516c503a518c)
    (cherry picked from commit ff45be406ae30193339d3e379f815a61c48e77c7)

Reviewed: https://review.opendev.org/c/openstack/grenade/+/879125
Committed: https://opendev.org/openstack/grenade/commit/6c3167112866bd68d46d84736b6692adaf63498b
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 6c3167112866bd68d46d84736b6692adaf63498b
Author: Ghanshyam Mann <email address hidden>
Date: Thu Mar 30 20:58:04 2023 -0500

    Fix tempest venv constraints for target branch tempest run

    We fixed the tempest venv constraints setting for tempest test run
    on base branch but forgot to do the same for target branch test run.
    It cause issue when base and target branch have different Tempest
    and constraints to use (for example stable/wallaby use old tmepest
    and stable/wallaby constraints but stable/xena use tempest master
    and master constraints), in such cases we need to set proper constraints
    defined in devstack and then run tempest.

    We do not see this issue as none of the job run tempest on target except
    heat (https://review.opendev.org/c/openstack/heat/+/872055/5).

    I reproduced the issue in grenade stable/xena by enaling the tempest run
    on target:
    - https://review.opendev.org/c/openstack/grenade/+/878247/1
    - https://zuul.opendev.org/t/openstack/build/1b503d359717459c9c77010608068e27/log/controller/logs/grenade.sh_log.txt#17184

    Extra on backport:
    We need heat fixes to pass the heat multinode grenade job
    Depends-On: https://review.opendev.org/c/openstack/heat/+/872056/6
    Depends-On: https://review.opendev.org/c/openstack/heat/+/872055

    Closes-Bug: 2003993
    Change-Id: I05fcdb5045a33997bd1a4c340a16458d88399f5f
    (cherry picked from commit 4c9c059883158fd6bf3cb79ca89d52003a63b7fe)
    (cherry picked from commit 62ea9b6ba35ae28c299a4366e81f516c503a518c)
    (cherry picked from commit ff45be406ae30193339d3e379f815a61c48e77c7)
    (cherry picked from commit cd782d27d56b0d087d7806df4bad459f44a03656)

Change abandoned by "Ghanshyam <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/grenade/+/871948
Reason: fixed in https://review.opendev.org/c/openstack/grenade/+/879124

Change abandoned by "Ghanshyam <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/grenade/+/871949
Reason: fixed in https://review.opendev.org/c/openstack/grenade/+/879125

Change abandoned by "Ghanshyam <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/grenade/+/871955

This issue was fixed in the openstack/heat 17.0.2 release.

This issue was fixed in the openstack/heat 18.0.1 release.

This issue was fixed in the openstack/heat 19.0.1 release.