Undercloud upgrade forces container-tools:3.0 stream

container-tools:latest is available in current CentOS Stream 9 with
podman 3.3.1
buildah 1.22.33

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/826153
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/9aa90bd1a7d197c389190f47c912fe9dbf14d160
Submitter: "Zuul (22348)"
Branch: master

commit 9aa90bd1a7d197c389190f47c912fe9dbf14d160
Author: Douglas Viroel <email address hidden>
Date: Mon Jan 24 17:18:35 2022 -0300

    Skip 'ensure dnf modules' for CS9 on undercloud-upgrade

    This patch change the condition for installing modules when
    distro version is equal or greater than 9. This distro will
    provide this content through another packages.

    Related-Bug: #1943938

    Change-Id: I963133176456ec20fde8c033dca9630d89cb1e66

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836596

as discussed on irc with mciecierski we need this for stable/wallaby too.
we don't have the undercloud-upgrade job cos no 9/victoria
but we may want to just add a 9 'minor update' undercloud upgrade that would have caught it i think (i.e. just install 9/wallaby and run undercloud upgrade on it)...

mciecierski cherrypicked to https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836596

also discussed there https://bugzilla.redhat.com/show_bug.cgi?id=2071974

We are currently similar issue during undercloud upgrade at
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&project=openstack/tripleo-quickstart

It is a gate blocker failure with following error
```
 Ensure DNF modules have the right stream | undercloud | item={'module': 'container-tools', 'stream': '3.0'} | error={"ansible_loop_var": "item", "changed": false, "item": {"module": "container-tools", "stream": "3.0"}, "msg": "No group container-tools:3.0/common available.", "results": []}
```

Below is the relevant gate failure logs:
* https://2d86012f5b832b835a40-cf2e6d1d128a778849f5c48f17b8a34a.ssl.cf5.rackcdn.com/836104/21/gate/tripleo-ci-centos-9-undercloud-upgrade/ea8c76f/logs/undercloud/home/zuul/undercloud_upgrade.log

* https://adec4b54ecf11c859be8-ade91ca27312268e5fafbe848cc22b67.ssl.cf2.rackcdn.com/836104/21/gate/tripleo-ci-centos-9-undercloud-upgrade/f413ca7/logs/undercloud/home/zuul/undercloud_upgrade.log

* https://29f4219f553d662d0215-a0a2161f988b6356e48326da15450ffb.ssl.cf5.rackcdn.com/836104/21/gate/tripleo-ci-centos-9-undercloud-upgrade/abcd951/logs/undercloud/home/zuul/undercloud_upgrade.log

there is container-tools module in CentOS Stream /RHEL-9,
```
[root@02413b061ce1 /]# cat /etc/redhat-release
CentOS Stream release 9
[root@02413b061ce1 /]# dnf module enable container-tools:latest
CentOS Stream 9 - BaseOS 361 kB/s | 5.6 MB 00:15
CentOS Stream 9 - AppStream 1.6 MB/s | 13 MB 00:07
CentOS Stream 9 - Extras packages 6.8 kB/s | 6.8 kB 00:00
Error: Problems in request:
missing groups or modules: container-tools:latest
[root@02413b061ce1 /]#

```
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836596 is needed to fix the backport.

Adding promotion-blocker for more visibility.

By testing with dnm patch https://review.opendev.org/c/openstack/tripleo-quickstart/+/837455/ with depends on https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836596, the above issue is fixed but now undercloud-upgrade is failing with different error [1].
```
Failure running exec 'keystone_bootstrap'. rc=1, stdout=, stderr=
2022-04-12 07:18:43 | 2022-04-12 07:18:43.486121 | fa163e9f-c470-b3fd-9b14-000000001b51 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | undercloud | error={"changed": false, "msg": "Failed containers: ironic_db_sync, keystone_bootstrap"}
```
and while taking a look at failed container logs
https://6ef3897d43ca5b2f7eff-13a1490981ac67093979241f62e5a9df.ssl.cf2.rackcdn.com/837455/1/check/tripleo-ci-centos-9-undercloud-upgrade/181b185/logs/undercloud/var/log/extra/podman/containers/ironic_db_sync/stdout.log

```
+ echo 'Running command: '\''/usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c '\''ironic-dbsync --config-file /etc/ironic/ironic.conf'\'''\'''
Running command: '/usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c 'ironic-dbsync --config-file /etc/ironic/ironic.conf''
+ exec /usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c ''\''ironic-dbsync' --config-file '/etc/ironic/ironic.conf'\'''
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> 2581ebaf0cb2, initial migration
INFO [alembic.runtime.migration] Running upgrade 2581ebaf0cb2 -> 21b331f883ef, Add provision_updated_at
INFO [alembic.runtime.migration] Running upgrade 21b331f883ef -> 3cb628139ea4, Nodes add console enabled
INFO [alembic.runtime.migration] Running upgrade 3cb628139ea4 -> 31baaf680d2b, Add Node instance info
INFO [alembic.runtime.migration] Running upgrade 31baaf680d2b -> 3bea56f25597, add unique constraint to instance_uuid
CRITI [ironic] Unhandled error
Traceback (most recent call last):
  File "/usr/bin/ironic-dbsync", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.9/site-packages/ironic/cmd/dbsync.py", line 342, in main
    CONF.command.func()
  File "/usr/lib/python3.9/site-packages/ironic/cmd/dbsync.py", line 121, in upgrade
    migration.upgrade(CONF.command.revision)
  File "/usr/lib/python3.9/site-packages/ironic/db/migration.py", line 36, in upgrade
    return get_backend().upgrade(version)
  File "/usr/lib/python3.9/site-packages/ironic/db/sqlalchemy/migration.py", line 56, in upgrade
    alembic.command.upgrade(config, revision or 'head')
  File "/usr/lib/python3.9/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/usr/lib/python3.9/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module) # type: ignore
  File "<froze...

did some digging on the new issue @Chandan... i think we may need a new bug for that. question is though does it have anything to do with the fix in https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836596 (I don't think so)

found some relevant discussions (e.g. there [1] - old but has most recent comment from February) ... this may be because ironic uses sqlite instead of sql for these alembic migrations? Not sure if newer version of alembic would fix it?

[1] https://github.com/miguelgrinberg/Flask-Migrate/issues/61

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/837512

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836596
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/30566528b94f106d93f50aae399e888173d51b1b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 30566528b94f106d93f50aae399e888173d51b1b
Author: Douglas Viroel <email address hidden>
Date: Mon Jan 24 17:18:35 2022 -0300

    Skip 'ensure dnf modules' for CS9 on undercloud-upgrade

    This patch change the condition for installing modules when
    distro version is equal or greater than 9. This distro will
    provide this content through another packages.

    Related-Bug: #1943938

    Change-Id: I963133176456ec20fde8c033dca9630d89cb1e66
    (cherry picked from commit 9aa90bd1a7d197c389190f47c912fe9dbf14d160)

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&project=openstack/tripleo-quickstart jobs seems to be pretty much green/ I think we can mark this bug as a fixed release or we want to keep it as a tracker till we properly solve it?

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/837512
Reason: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&project=openstack/tripleo-quickstart job is green now