[SRU] magnum ui can not delete the coe cluster

Thanks Felipe. I've sponsored the upload of your updates to python-magnumclient. It's in the unapproved queue now: https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=python-magnumclient

Checking the list of reverse dependencies of python3-magnumclient in focal:
$ apt-cache rdepends python3-magnumclient
python3-magnumclient
Reverse Depends:
  python3-mistral
  python3-mistral
  python3-heat
  python3-heat
  python3-magnum-ui

Assuming these are all openstack components and are not used outside of openstack, does the [test case] cover these components as well? Specifically mistral and heat?

Is there any sort of release notes available for 3.0.1, or any of the others that are compatible with what is in focal? And why did you pick 3.0.1, and not, say, 2.17.0 for example? Did you analyze the diff and conclude there are no backwards incompatible changes being introduced?

Hello Andreas,

Thank you so much for your thoughtful review, here are my replies to your questions/concerns inline:

> Assuming these are all openstack components and are not used outside of openstack, does the [test case] cover these components as well? Specifically mistral and heat?

They are all OpenStack components. python3-heat and python3-magnum-ui are indeed covered by tempest testing, because python3-magnum and python3-heat are tightly coupled, magnum needs heat to achieve its work done.

About python3-mistral, I wasn't planning on testing it, and it's not part of our default testing matrix, although I will add it, we have charm that could help us here - https://opendev.org/openstack/charm-mistral

> Is there any sort of release notes available for 3.0.1, or any of the others that are compatible with what is in focal?

Sadly the only release notes are the ones autogenerated which are empty - https://docs.openstack.org/releasenotes/python-magnumclient/ussuri.html

python clients released by OpenStack have a strong history of backwards compatibility, it's common for operators to use the latest release of the clients (e.g. via the openstackclients snap) to interact with any version of OpenStack, the clients are capable of downgrading based on the API (micro)versions exposed by each endpoint.

> And why did you pick 3.0.1, and not, say, 2.17.0 for example?

I could have used 2.17.0 in this case, since no bug fixes were merged (see below), although I had two concerns with pursuing that approach: 1) I wasn't sure if missing the commit "8fd3b04 Rename variables to address pep8 error" could break autopkgtest checks, 2) Incurring in technical debt without reason, since 3.0.1 is the last stable released from the stable/ussuri git branch.

$ git log --oneline 2.17.0..3.0.1
6d1a386 (tag: ussuri-em, tag: 3.0.1) Labels override
8fd3b04 Rename variables to address pep8 error
75fec05 Update TOX/UPPER_CONSTRAINTS_FILE for stable/ussuri
366893e Update .gitreview for stable/ussuri
11d2b72 (tag: 3.0.0) Merge "Update master for stable/train"
425fa49 Merge "Update links in README"
18f5928 Merge "Bugfix: Use fields option for cluster template list"
e589b6c Update master for stable/train
5d93b51 Update hacking for Python3
de2b8e8 Drop py27 tests
5b7a671 Bugfix: Use fields option for cluster template list
8473982 Update links in README

> Did you analyze the diff and conclude there are no backwards incompatible changes being introduced?

I did go through the diff[0], the commits that provide relevant changes to the functionality are:

934cf54 Add nodegroup CRUD commands
5ad8b7d Support network, subnet and FIP when creating cluster
d91d4c7 Display project_id for cluster show
6b756aa Support upgrade API
94380f9 Support resize api
74c5f22 Support health_status on client side
81b8480 Keystone auth support

While I was reading those patches I didn't detect any incompatible change, and the features those patches enable are already part of python3-magnum in Focal (10.0.0-0ubuntu0.20.04.2).

If there is any aspect I didn't cover, please let me know to elaborate with more detail.

Thanks in advance.

[0] https://github.com/openstack/python-magnumclient/compare/2.11...

I was asked to look at this as Andreas is out, so sorry to ask again but I have further questions.

I'm a bit unclear on versions and compatibility. Is the version of python-magnumclient shipped in Ubuntu 20.04 incompatible with the version of magnum shipped in Ubuntu 20.04? Or something else? Specifically you have a client version, a protocol version, and a server version, in your broken user story that needs fixing. Please could you state all three versions in your Impact statement, so I can understand exactly which versions are involved in the problem case?

If the problem lies entirely within the Ubuntu 20.04 archive, then how was it that we shipped it broken in the first place, given that we rely heavily on automated testing for Openstack point releases in stable Ubuntu releases? Is there a gap in testing somewhere?

> I'm a bit unclear on versions and compatibility.

The upstream project documents what versions are part of a given release, for example in the case of python-magnumclient on Focal (ussuri) they can be found at [0]

> Is the version of python-magnumclient shipped in Ubuntu 20.04 incompatible with the version of magnum shipped in Ubuntu 20.04?

correct. The version in Focal is 2.11.0 which corresponds to Stein[1] . I wasn't around at the time, but what I could infer from looking into the history of changelogs, it's that we got that version from Debian into Ubuntu, and during the 21.04 cycle the OpenStack team oboarded a Magnum charm that became part of "Charmed OpenStack"[2], so I assume that's the reason this incompatibility wasn't detected earlier.

> Specifically you have a client version, a protocol version, and a server version, in your broken user story that needs fixing.

Correct. The server side is being taken care by bug 2009966 [3], during the validation of it the incompatibility was detected.

I will update the impact to include the versions of magnum-ui, magnum and python-magnumclient.

> If the problem lies entirely within the Ubuntu 20.04 archive, then how was it that we shipped it broken in the first place, given that we rely heavily on automated testing for Openstack point releases in stable Ubuntu releases? Is there a gap in testing somewhere?

We added automated testing for Magnum as part of the MRE request[4], that was an identified gap that we closed back in February/March.

[0] https://releases.openstack.org/teams/magnum.html#team-ussuri-python-magnumclient
[1] - https://releases.openstack.org/teams/magnum.html#team-stein-python-magnumclient
[2] https://opendev.org/openstack/charm-magnum/commit/fb04f0f1fc87b42c4c8dfb202776717117ca1d93
[3] https://bugs.launchpad.net/ubuntu/+source/magnum/+bug/2009966
[4] https://lists.ubuntu.com/archives/ubuntu-release/2023-February/005543.html

Thank you for helping me understand that detail, and for the additional conversation which for reference is at https://irclogs.ubuntu.com/2023/06/22/%23ubuntu-devel.html#t14:13

SRU Review

Looking at Andreas' questions and your answers, it sounds like the test plan is reasonable then. Please add the python3-mistral you proposed to the Test Plan in the bug description so it doesn't get missed.

The number of functional changes upstream is not that big. Having reviewed them I would have preferred that you cherry-picked them only. I think that would have resulted in a simpler and more straightforward review.

You did say that you had difficulty in finding the right cherry-picks, I've reviewed all the changes now anyway, and it's true that the version is currently "wrong" against the magnum also in our Focal archive, so on balance I think it's OK to accept this approach now that we're here.

I found three CLI UI breaking changes in all:

cluster show adds a column for project_id
https://github.com/openstack/python-magnumclient/commit/d91d4c72a10493e57a664e18e1f98268e43dd537

cluster list adds a column for health_status
https://github.com/openstack/python-magnumclient/commit/74c5f22c6ffb23154f2e0412ba43371f3c02f3b7

nodegroup list *removes* some columns?
https://github.com/openstack/python-magnumclient/commit/934cf548540086268991dab47b5bcb85f65b693f

I accept that users are likely to have used `--format json | jq` and the likelihood of a user being regressed by this is low. But the removal of some columns rather than just the addition of them could also regress interactive users. What do you want to do with these? If you think it's necessary to keep these changes, please could you add to "Where problems could occur" with an explanation and justification?

Orig tarball method:

Upstream tag 2.11.0 is identical to what is in Focal except for the debian/ directory as expected. But upstream tag 3.0.1 is noticeably different from your upload, and the differences in setup.cfg may be functional. To avoid inadvertent changes, please re-upload generating the orig tarball in the same way that it was done before, so that the upstream 2.11.0..3.0.1 changes and related packaging changes are the only changes that will apply in this upload.

Given the above issue, do you have a build somewhere for binary debdiff purposes, please, so we can double-check that the build result is the same apart from the intended code changes?

Because I believe this needs a re-upload for the orig tarball issue, I'm rejecting the current upload from the queue.

An upload of python-magnumclient to focal-proposed has been rejected from the upload queue for the following reason: "See https://bugs.launchpad.net/ubuntu/+source/magnum-ui/+bug/1996229/comments/6".

> Looking at Andreas' questions and your answers, it sounds like the test plan
> is reasonable then. Please add the python3-mistral you proposed to the Test
> Plan in the bug description so it doesn't get missed.

Noted. Will do.

> nodegroup list *removes* some columns?
> https://github.com/openstack/python-magnumclient/commit/934cf548540086268991dab47b5bcb85f65b693f

$ git show 934cf548540086268991dab47b5bcb85f65b693f
[...]
@@ -72,7 +196,8 @@ class ListNodeGroup(command.Lister):
         self.log.debug("take_action(%s)", parsed_args)

         mag_client = self.app.client_manager.container_infra
- columns = ['uuid', 'name', 'flavor_id', 'node_count', 'role']
+ columns = ['uuid', 'name', 'flavor_id', 'image_id', 'node_count',
+ 'status', 'role']
         cluster_id = parsed_args.cluster
         nodegroups = mag_client.nodegroups.list(cluster_id,
                                                 limit=parsed_args.limit,
[...]

If this is the chunk you are referring to, I see `image_id` and `status` were
added, while `node_count` and `role` only changed position. Am I looking at the
wrong section of the patch?

> Orig tarball method:
>
> Upstream tag 2.11.0 is identical to what is in Focal except for the debian/
> directory as expected. But upstream tag 3.0.1 is noticeably different from
> your upload, and the differences in setup.cfg may be functional. To avoid
> inadvertent changes, please re-upload generating the orig tarball in the same
> way that it was done before, so that the upstream 2.11.0..3.0.1 changes and
> related packaging changes are the only changes that will apply in this upload.

OK, I will look into this, because I'm not familiar with this aspect. For this
upload the published tarball was used, and not a git tag, maybe the difference
come from there.

> Given the above issue, do you have a build somewhere for binary debdiff
> purposes, please, so we can double-check that the build result is the same
> apart from the intended code changes?

I was using this PPA for testing purposes, it's not the exact same package that
was uploaded in terms of debian/changelog, but all the rest is the same:

https://launchpad.net/~awake-marmot/+archive/ubuntu/focal-ussuri-staging/+packages?field.name_filter=python-magnumclient&field.status_filter=published&field.series_filter=

> Because I believe this needs a re-upload for the orig tarball issue, I'm
> rejecting the current upload from the queue.

ACK. We'll work on addressing the feedback.

> If this is the chunk you are referring to, I see `image_id` and `status` were
added, while `node_count` and `role` only changed position. Am I looking at the
wrong section of the patch?

Sorry, yes, I think this is my mistake. I didn't spot that they had just changed position. So the potential regression is less severe, but I think this request still applies:

> If you think it's necessary to keep these changes, please could you add to "Where problems could occur" with an explanation and justification?

To be clear, I'm saying that I think this is OK, but I think we should still document it as a deliberate decision.

> PPA

Thanks! Please could you update that against the new orig tarball, assuming it uses the old one? Once that's done, we can run debdiff against the archive and PPA binary debs to ensure that nothing unexpected will be changed. Or if you prefer we can just do that during SRU verification once the proposed binaries are built.

> For this upload the published tarball was used, and not a git tag, maybe the difference
come from there.

Yes this seems like the likely cause to me. "git archive" should help, assuming that's how it was done before, but this needs checking.

Hi Robie,

Thanks for the review. I've updated the [Test Plan] and [Where problems could occur] sections based on your feedback.

As for the orig tarball, in Ubuntu we use a different process from OpenStack in Debian. We use uscan to import upstream published release tarballs (via debian/watch) using pristine-tar, whereas in Debian imports the git source directly from the upstream git repository into their packages.

Instead of trying to do what is done in Debian I would prefer to provide "3.0.1" support via cherry-picks. I did an assessment to narrow down on the patches that we would need to pick up to get us there.

This is the entire delta between 2.11.0 and 3.0.1:

~/pkg/ussuri/upstream/python-magnumclient$ git-pretty 2.11.0..3.0.1
 - [6d1a386] Labels override
 - [8fd3b04] Rename variables to address pep8 error
 - [75fec05] Update TOX/UPPER_CONSTRAINTS_FILE for stable/ussuri
 - [366893e] Update .gitreview for stable/ussuri
 - [e589b6c] Update master for stable/train
 - [5d93b51] Update hacking for Python3
 - [de2b8e8] Drop py27 tests
 - [5b7a671] Bugfix: Use fields option for cluster template list
 - [c0d3683] Allow cluster config for any cluster state
 - [934cf54] Add nodegroup CRUD commands
 - [69be0ac] Replace git.openstack.org URLs with opendev.org URLs
 - [5ad8b7d] Support network, subnet and FIP when creating cluster
 - [a19d013] Add Python 3 Train unit tests
 - [c72512d] Conditional hidden arg for backward compatibility
 - [440f4b1] Fix coverage test
 - [3d6e9bd] Blacklist bandit 1.6.0 and cap Sphinx on Python2
 - [d91d4c7] Display project_id for cluster show
 - [d3d04b7] OpenDev Migration Patch
 - [9f5b8c7] Dropping the py35 testing
 - [ea7c571] Add nodegroup list/show commands
 - [6b756aa] Support upgrade API
 - [94380f9] Support resize api
 - [46e4ab1] Update master for stable/stein
 - [74c5f22] Support health_status on client side
 - [3f7b994] python3 fixes
 - [81b8480] Keystone auth support
 - [a3297b7] add python 3.7 unit test job
 - [ab8392c] Add hidden property to cluster template
 - [5deb538] Fix py37 compatibility
 - [b4120a1] Use oslo_serialization instead of the json module directly
 - [d0d08f0] Use template for lower-constraints
 - [d942ce5] Change openstack-dev to openstack-discuss
 - [11b7527] Add Python 3.6 classifier to setup.cfg
 - [07b83c2] add python 3.6 unit test job
 - [1ab2e77] Trivial: Update pypi url to new url
 - [8473982] Update links in README

I've narrowed down those commits to the following subset which are necessary to update the existing 2.11.0 package to 3.0.1 functionality:
 - [6d1a386] Labels override
 - [8fd3b04] Rename variables to address pep8 error
 - [5d93b51] Update hacking for Python3
 - [5b7a671] Bugfix: Use fields option for cluster template list
 - [c0d3683] Allow cluster config for any cluster state
 - [934cf54] Add nodegroup CRUD commands
 - [5ad8b7d] Support network, subnet and FIP when creating cluster
 - [c72512d] Conditional hidden arg for backward compatibility
 - [3d6e9bd] Blacklist bandit 1.6.0 and cap Sphinx on Python2
 - [d91d4c7] Display project_id for cluster show
 - [ea7c571] Add nodegroup list/show commands
 - [6b756aa] Support upgrade API
 - [94380f9] Sup...

Hi Robie,

I took a look at the diff of 2 other popular openstack clients between the upstream tag and the equivalent package source for stable/ussuri and they have similar differences to what we're seeing here with python-magnumclient [1]. I also took a closer look at the setup.cfg diff for python-magnumclient and it is just formatting changes (tabs and spaces differences) plus the addition of the [egg_info] section, nothing that would affect functionality.

I'm going to re-upload the same package version since this is a normal delta that is created as part of the upstream automated release tarball creation.

Please let me know if you need any more information.

[1]
python-openstackclient: https://paste.ubuntu.com/p/KdWrYNw3cG/
python-novaclient: https://paste.ubuntu.com/p/sgKzZ46HKf/

Thanks,
Corey

coreycb: python-magnumclient> can you not use the same method that was used for the current upload? SRUs are supposed to be minimal. Rather than requiring us to analyse and prove that differences aren't relevant, the expectation is that your workflows will ensure that those differences do not exist.

The issue is that this just adds SRU review workload which I don't think trades well with your convenience given that we accept that every change carries regression risk even when a convincing case is presented that it won't result in a regression. This is demonstrated by the constant flow of valid SRU regression reports.

Especially in this case where I'm being told by management that they need the fix quickly, I don't think it's helpful to just insist that it's fine and then re-upload the same thing again contrary to previous review feedback, handing the review burden back to the SRU team. The easy path is to upload the minimal fix required to fix the problem, as stated by SRU policy.

My position remains the same. The current upload needs fixing. Please fix it.

Please do not re-upload until the requested fix is provided, or you've agreed an alternative path with an SRU team member.

Hi Robie,

The method for importing the orig tarball actually didn't change since 2.11.0. (You can ignore my previous comments about Debian openstack packaging. I had mentioned that before I noticed 2.11.0 originated in Ubuntu.)

Anything that has changed is in the way the upstream tarballs are produced. I was hoping that providing examples of the same type of delta in other existing focal (stable/ussuri) packages would help support the case that the upstream orig tarball that we are using in the current upload doesn't include any inadvertent changes.

I can look into how to import a release straight from the upstream git repository but it doesn't seem like the right way to do things. I'm not sure of any other ways. I'm open to suggestions.

The other approach we can take is to cherry-pick the patches I mentiond in comment #11.

Let me know what your thoughts are.

Thanks,
Corey

I see the debian/watch file was changed from github to opendev so I think switching back to github might get back to where Robie was referencing in terms of using the original method. The reason it was changed is likely due to [1]. I'll look into getting the tarball from github.

[1] https://wiki.debian.org/debian/watch
"As of 2022-10-02, ?GitHub change the way its release/tags page is rendered. The links are dynamically loaded in ?JavaScript instead of static text. See thread for the discussions:".

With the tarball from github the delta is reduced in the way that Robie was asking for. I've just uploaded this new 3.0.1 version to the focal unapproved queue for review. Thank you for your thorough reviews.

The new upload is better thanks - the additional unnecessary changes are gone.

Hello Narinder, or anyone else affected,

Accepted python-magnumclient into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-magnumclient/3.0.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Narinder, or anyone else affected,

Accepted python-magnumclient into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

for the record, I'm running the verification tests for this SRU at the moment.

I went through the validation suite with tempest and it completed successfully. I will do Horizon verification now and capture screenshots as evidence.

```
======
Totals
======
Ran: 120 tests in 2223.9823 sec.
 - Passed: 111
 - Skipped: 9
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 1390.9734 sec.

==============
Worker Balance
==============
 - Worker 0 (120 tests) => 0:37:03.982314
```

Package used during the verification:

$ juju ssh magnum/leader apt policy python3-magnumclient
python3-magnumclient:
  Installed: 3.0.1-0ubuntu1
  Candidate: 3.0.1-0ubuntu1
  Version table:
 *** 3.0.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.11.0-0ubuntu6 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     2.11.0-0ubuntu4 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages
Connection to 10.5.2.15 closed.
$ juju ssh openstack-dashboard/leader apt policy python3-magnumclient
python3-magnumclient:
  Installed: 3.0.1-0ubuntu1
  Candidate: 3.0.1-0ubuntu1
  Version table:
 *** 3.0.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.11.0-0ubuntu6 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     2.11.0-0ubuntu4 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages
Connection to 10.5.0.154 closed.

Here I'm attaching screenshots of Horizon (OpenStack Dashboard) manipulating magnum objects (cluster template and cluster).

The verification of the Stable Release Update for python-magnumclient has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-magnumclient - 3.0.1-0ubuntu1

---------------
python-magnumclient (3.0.1-0ubuntu1) focal; urgency=medium

  [ Corey Bryant ]
  * New stable point release for OpenStack Ussuri (LP: #1996229).

  [ Felipe Reyes ]
  * d/p/fix-py37-compatibility.patch: Dropped. Fixed in new stable point
    release.
  * d/p/Fix-failing-to-parse-json-error-msg.patch: Dropped. Fixed in new
    stable point release.

 -- Corey Bryant <email address hidden> Thu, 29 Jun 2023 19:01:16 -0400