large amount of memory utilization slows down ansible execution as part of the deployment

> Additionally we should not use any of the package or service fact methods.

Are we saying that any kind of fact gathering should be avoided (or just service/package facts), as it's multiplied by number of nodes and adds to ansible controller memory usage (~32GB RAM is not enough)? I can't see anything in service_facts[1] module that can lead to high memory usage though.

[1] https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/service_facts.py

Yes. But the problem is not the amount of memory total but rather once you exceed like 1G, ansible comes to a crawl. I'm still looking into the details but I believe it's related to fork() and python's GIL as it relates to volume of large memory objects that must be copied when a task is spawned. The reality is that our code doesn't use a fraction of the facts that we gather and things like package facts and service facts pull in a bunch of information about unrelated items which causes more problems than simply doing a targeted check

How to find vars:
grep ansible_ * -r | egrep -v "(facts|become|user|host|limit|check_mode|python|ssh|connection|min_|managed|ceph_ansible|verbos|hieradata|job|_home|_log|loop|async|limit|filter|_version|calling_|_inventory|\.py|group_vars|diff_mode)" | grep ansible_

bulk fix vars:
# THT
for DIR in common deployment; do
# tripleo-ansible
#for DIR in tripleo_ansible/roles; do
  find $DIR -type f | xargs sed -i \
    -e "s/ansible_hostname\([\ |\.]\)/ansible_facts['hostname']\1/g" \
    -e "s/ansible_fqdn\([\ |\.]\)/ansible_facts['fqdn']\1/g" \
    -e "s/ansible_nodename\([\ |\.]\)/ansible_facts['nodename']\1/g" \
    -e "s/ansible_distribution\([\ |\.]\)/ansible_facts['distribution']\1/g" \
    -e "s/ansible_distribution_version\([\ |\.]\)/ansible_facts['distribution_version']\1/g" \
    -e "s/ansible_distribution_major_version\([\ |\.]\)/ansible_facts['distribution_major_version']\1/g" \
    -e "s/ansible_os_family\([\ |\.]\)/ansible_facts['os_family']\1/g" \
    -e "s/ansible_devices\([\ |\.]\)/ansible_facts['devices']\1/g"
done

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/786159

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/786219

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/786220

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/785918
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/e1998a8e58b68c67979d30e4213c772169624484
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit e1998a8e58b68c67979d30e4213c772169624484
Author: Dave Wilde (d34dh0r53) <email address hidden>
Date: Mon Apr 12 10:38:17 2021 -0500

    Ensure ansible_fqdn is set

    The ipaclient ansible role requires that ansible_fqdn is defined but
    due to [1] we don't have ansible_fqdn inside of ansible_facts. This
    uses the 'fqdn' ansible fact for ansible_fqdn which is equivalent.

    [1]: https://opendev.org/openstack/tripleo-heat-templates/commit/4e79336d69e
    6b7fa4b026922bac7953bafeee96d

    Related-Bug: 1915761
    Closes-Bug: 1923248
    Change-Id: I0a740e86588c96fff24fa09698c35e492d1c64db

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/786159
Committed: https://opendev.org/openstack/tripleo-ansible/commit/30df6f54375bb3ffa0cdd40ca219905b2fa92cf5
Submitter: "Zuul (22348)"
Branch: master

commit 30df6f54375bb3ffa0cdd40ca219905b2fa92cf5
Author: yatinkarel <email address hidden>
Date: Wed Apr 14 10:29:42 2021 +0530

    Use ansible_facts in tripleo_ssh_known_hosts role

    It was missed in original patch to switch to
    ansible_facts[1].

    For providing test value for 'ssh_host_key_rsa_public' overriding
    it in 'ansible_facts' with set_fact.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/776666

    Closes-Bug: #1923403
    Related-Bug: #1915761
    Change-Id: I9d5ae576c57eefe5496f9dda71e8eac23e45e89f

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/786321

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/786233

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/786234

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/786355

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/786233
Committed: https://opendev.org/openstack/tripleo-ansible/commit/c582b6584e78922403e296701c0ade252af7a8fe
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c582b6584e78922403e296701c0ade252af7a8fe
Author: yatinkarel <email address hidden>
Date: Wed Apr 14 10:29:42 2021 +0530

    Use ansible_facts in tripleo_ssh_known_hosts role

    It was missed in original patch to switch to
    ansible_facts[1].

    For providing test value for 'ssh_host_key_rsa_public' overriding
    it in 'ansible_facts' with set_fact.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/776666

    Closes-Bug: #1923403
    Related-Bug: #1915761
    Change-Id: I9d5ae576c57eefe5496f9dda71e8eac23e45e89f
    (cherry picked from commit 30df6f54375bb3ffa0cdd40ca219905b2fa92cf5)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/786234
Committed: https://opendev.org/openstack/tripleo-ansible/commit/b52dc1e7fd69fdc360f1281c53ac92abb7dddfee
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit b52dc1e7fd69fdc360f1281c53ac92abb7dddfee
Author: yatinkarel <email address hidden>
Date: Wed Apr 14 10:29:42 2021 +0530

    Use ansible_facts in tripleo_ssh_known_hosts role

    It was missed in original patch to switch to
    ansible_facts[1].

    For providing test value for 'ssh_host_key_rsa_public' overriding
    it in 'ansible_facts' with set_fact.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/776666

    Closes-Bug: #1923403
    Related-Bug: #1915761
    Change-Id: I9d5ae576c57eefe5496f9dda71e8eac23e45e89f
    (cherry picked from commit 30df6f54375bb3ffa0cdd40ca219905b2fa92cf5)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/786355
Committed: https://opendev.org/openstack/tripleo-ansible/commit/a29918a4aff5802f8c86d8f79b3846da755c4aa2
Submitter: "Zuul (22348)"
Branch: stable/train

commit a29918a4aff5802f8c86d8f79b3846da755c4aa2
Author: yatinkarel <email address hidden>
Date: Wed Apr 14 10:29:42 2021 +0530

    Use ansible_facts in tripleo_ssh_known_hosts role

    It was missed in original patch to switch to
    ansible_facts[1].

    For providing test value for 'ssh_host_key_rsa_public' overriding
    it in 'ansible_facts' with set_fact.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/776666

    Closes-Bug: #1923403
    Related-Bug: #1915761
    Change-Id: I9d5ae576c57eefe5496f9dda71e8eac23e45e89f
    (cherry picked from commit 30df6f54375bb3ffa0cdd40ca219905b2fa92cf5)
    (cherry picked from commit b52dc1e7fd69fdc360f1281c53ac92abb7dddfee)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/786219
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/1325566bad778b0d2d174833698ff41f46524948
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 1325566bad778b0d2d174833698ff41f46524948
Author: Dave Wilde (d34dh0r53) <email address hidden>
Date: Mon Apr 12 10:38:17 2021 -0500

    Ensure ansible_fqdn is set

    The ipaclient ansible role requires that ansible_fqdn is defined but
    due to [1] we don't have ansible_fqdn inside of ansible_facts. This
    uses the 'fqdn' ansible fact for ansible_fqdn which is equivalent.

    [1]: https://opendev.org/openstack/tripleo-heat-templates/commit/4e79336d69e
    6b7fa4b026922bac7953bafeee96d

    Related-Bug: 1915761
    Closes-Bug: 1923248
    Change-Id: I0a740e86588c96fff24fa09698c35e492d1c64db

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/786220
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/f24840a56133f8862a521391c1cf14b1f29a0211
Submitter: "Zuul (22348)"
Branch: stable/train

commit f24840a56133f8862a521391c1cf14b1f29a0211
Author: Dave Wilde (d34dh0r53) <email address hidden>
Date: Mon Apr 12 10:38:17 2021 -0500

    Ensure ansible_fqdn is set

    The ipaclient ansible role requires that ansible_fqdn is defined but
    due to [1] we don't have ansible_fqdn inside of ansible_facts. This
    uses the 'fqdn' ansible fact for ansible_fqdn which is equivalent.

    [1]: https://opendev.org/openstack/tripleo-heat-templates/commit/4e79336d69e
    6b7fa4b026922bac7953bafeee96d

    Related-Bug: 1915761
    Closes-Bug: 1923248
    Change-Id: I0a740e86588c96fff24fa09698c35e492d1c64db

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/786321
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/0de9ea84ff7a144c89c5a6cee0c68dabb571583a
Submitter: "Zuul (22348)"
Branch: stable/train

commit 0de9ea84ff7a144c89c5a6cee0c68dabb571583a
Author: Dave Wilde (d34dh0r53) <email address hidden>
Date: Wed Apr 14 15:21:21 2021 -0500

    [Train Only] Ensure novajoin code is setting ansible_fqdn

    Novajoin tls-e code is still available in train and is affected by
    the same removal of the fact passing as [1]. This uses
    ansible_facts['fqdn'] as the ansible_fqdn for the tripleo-ipa
    registration play.

    [1]: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/785909

    Related-Bug: 1915761
    Related-Bug: 1923248
    Change-Id: I5b1514b4ba9bb22bcef63e74b0400cd9332516ca