stable/yoga gate blocker: failures in py39 unit tests

Bug #1988482 reported by Jay Faulkner
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Balazs Gibizer
Yoga
Triaged
High
Balazs Gibizer

Bug Description

All jobs in nova.tests.unit.cmd.test_manage.DbCommandsTestCase, as well as test_nova.tests.unit.cmd.test_manage.UtilitiesTestCase.test_format_dict.

One specific example:
```
ft2.1: nova.tests.unit.cmd.test_manage.UtilitiesTestCase.test_format_dicttesttools.testresult.real._StringException: pythonlogging:'': {{{
2022-09-01 17:37:16,772 WARNING [oslo_policy.policy] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html.
2022-09-01 17:37:16,773 WARNING [oslo_policy.policy] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html.
2022-09-01 17:37:16,774 WARNING [oslo_policy.policy] Policy Rules ['os_compute_api:extensions', 'os_compute_api:os-floating-ip-pools', 'os_compute_api:os-quota-sets:defaults', 'os_compute_api:os-availability-zone:list', 'os_compute_api:limits', 'project_admin_api', 'project_member_api', 'project_reader_api', 'project_reader_or_admin', 'os_compute_api:limits:other_project', 'os_compute_api:os-lock-server:unlock:unlock_override', 'os_compute_api:servers:create:zero_disk_flavor', 'compute:servers:resize:cross_cell'] specified in policy files are the same as the defaults provided by the service. You can remove these rules from policy files which will make maintenance easier. You can detect these redundant rules by ``oslopolicy-list-redundant`` tool also.
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/unit/cmd/test_manage.py", line 55, in test_format_dict
    self.assertEqual(
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py39/lib/python3.9/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py39/lib/python3.9/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = '''\
+----------+----------------------+
| Property | Value |
+----------+----------------------+
| bing | bat |
| foo | bar |
| test | {'a nested': 'dict'} |
| wow | a multiline |
| | string |
+----------+----------------------+'''
actual = '''\
+----------+----------------------+
| Property | Value |
+----------+----------------------+
| bing | bat |
| foo | bar |
| test | {'a nested': 'dict'} |
| wow | a multiline |
| | string |
+----------+----------------------+'''

```

This is failing on at least two different changes that are completely unrelated and are unlikely to be causing these failures:
- https://review.opendev.org/c/openstack/nova/+/855025
- https://review.opendev.org/c/openstack/nova/+/854257

I was unable to create an opensearch query to find all these failures, but https://tinyurl.com/mr3pjzyn catches related failures in cross-nova-py38

I was able to reproduce the failure in test_format_dict locally, by downloading 854257 and running tox -e py39. Output from that reproduction is:

```
nova.tests.unit.cmd.test_manage.UtilitiesTestCase.test_format_dict
------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/jay/dev/nova/nova/tests/unit/cmd/test_manage.py", line 55, in test_format_dict
    self.assertEqual(

      File "/home/jay/dev/nova/.tox/py39/lib/python3.9/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)

      File "/home/jay/dev/nova/.tox/py39/lib/python3.9/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error

    testtools.matchers._impl.MismatchError: !=:
reference = '''\
+----------+----------------------+
| Property | Value |
+----------+----------------------+
| bing | bat |
| foo | bar |
| test | {'a nested': 'dict'} |
| wow | a multiline |
| | string |
+----------+----------------------+'''
actual = '''\
+----------+----------------------+
| Property | Value |
+----------+----------------------+
| bing | bat |
| foo | bar |
| test | {'a nested': 'dict'} |
| wow | a multiline |
| | string |
+----------+----------------------+'''
```

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote (last edit ):

I searched around and I can confirm it is only visible on stable/yoga and in the py39 job. Also it only appears in the last 4 days
https://paste.opendev.org/show/bxX18pfYNwDcIzq1e0H1/

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote (last edit ):

Now I realized that launchpad changed the pasted output of the bug report. The difference in the output is the placement of the "Value" header. The in the reference it is left aligned it the actual output it is centered.

So now I'm pretty sure it is related to the prettytable issue reported by flicker during the weekend:
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2022-08-27.log.html
https://zuul.opendev.org/t/openstack/build/2e4535d6aa2d4fd987ddfd0d0bc296d1

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

For some reason in py39 in stable/yoga PrettyTable==3.4.0 is installed which is very new, released on 25th of Aug and which has a breaking change about table formatting. So I looked into our global upper constraints file and actually it does not restrict PrettyTable for py39 on yoga[1]. I will push a patch soon to requirements to fix it.

[1]https://github.com/openstack/requirements/blob/52cd6721466b6af58f62ae8a16512b566db887dd/upper-constraints.txt#L154

Changed in nova:
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Balazs Gibizer (balazs-gibizer)
status: Triaged → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/855658

Changed in nova:
status: Invalid → In Progress
Changed in nova:
importance: Critical → High
Revision history for this message
sean mooney (sean-k-mooney) wrote :

with the new 3.4.1 release of PrettyTable we are no longer impacted by this however to make sure this does not break use going forward I'm going to continue with
https://review.opendev.org/c/openstack/nova/+/855658
and backport it to yoga so that we have captured our depency on the alignment.

if we want to then change the alignment in the future we can update the tests deliberately and not
worry about version difference for now that is out of scope so I'm just going to hardcode the left alignment of the table headers our tests depend on today.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/855658
Committed: https://opendev.org/openstack/nova/commit/6569de43a6482488afb620fd9dd43cb15b316cce
Submitter: "Zuul (22348)"
Branch: master

commit 6569de43a6482488afb620fd9dd43cb15b316cce
Author: Sean Mooney <email address hidden>
Date: Fri Sep 2 13:42:15 2022 +0100

    add header alingment for PrettyTable 3.4.0

    In 3.3.0 the align attirbute applied to both the header
    and the data. In 3.4.0 align only applies to the data.

    This change restores the previous left align behavior
    for both header and data.

    Closes-Bug: #1988482
    Change-Id: Ia77410b10c1706bc6561b11cf5d2ef72b936795e

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 27.0.0.0rc1

This issue was fixed in the openstack/nova 27.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.