Magnum

Traceback when getting capacity in k8s_monitor

Bug #1659420 reported by Jason Dunsmore
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Magnum
Confirmed
Undecided
Unassigned
Ubuntu Cloud Archive
Fix Committed
High
Unassigned
Pike
Fix Committed
High
Unassigned
python-k8sclient (Ubuntu)
Fix Released
High
Unassigned
Artful
Fix Released
High
Unassigned

Bug Description

I'm seeing the following traceback in the magnum-conductor output:

2017-01-25 22:03:33.838 WARNING magnum.service.periodic [req-1950580a-0c7d-45bf-8dd6-74a488a55cac None None] Skip pulling data from cluster 487c7ad3-30b8-4571-bd09-6dd33b864107 due to error: 'object' object has no attribute '__getitem__'
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic Traceback (most recent call last):
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic File "/opt/stack/magnum/magnum/service/periodic.py", line 157, in _send_cluster_metrics
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic monitor.pull_data()
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic File "/opt/stack/magnum/magnum/drivers/common/k8s_monitor.py", line 44, in pull_data
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic self.data['nodes'] = self._parse_node_info(nodes)
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic File "/opt/stack/magnum/magnum/drivers/common/k8s_monitor.py", line 162, in _parse_node_info
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic memory = utils.get_k8s_quantity(capacity['memory'])
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic TypeError: 'object' object has no attribute '__getitem__'
2017-01-25 22:03:33.838 573 ERROR magnum.service.periodic

Maybe it's related to the fix for https://bugs.launchpad.net/magnum/+bug/1595373

Revision history for this message
Spyros Trigazis (strigazi) wrote :

The fix requires a fix in the k8sclient too. Check the version of your k8sclient. If you have version 0.3.0 it's expected that you'll have the bug.

Revision history for this message
Michal Nasiadka (mnasiadka) wrote :

I've had the same problem, upgrading to python-k8sclient does the trick.
Wouldn't it be easier for people to update the requirements.txt pointing to minimum supported version of python-k8sclient?

Changed in magnum:
status: New → Confirmed
Corey Bryant (corey.bryant)
Changed in cloud-archive:
status: New → Triaged
Changed in magnum (Ubuntu):
status: New → Triaged
Changed in cloud-archive:
importance: Undecided → High
Changed in magnum (Ubuntu):
importance: Undecided → High
Revision history for this message
Corey Bryant (corey.bryant) wrote :

I'd like to get a feature freeze exception in Ubuntu to upload python-k8sclient 0.4.0 to artful.

This issue is fixed by the 0.4.0 release of python-k8sclient. This will have a minimal impact on the archive as the only package that depends on python-k8sclient is python-magnum.

no longer affects: magnum (Ubuntu)
Changed in python-k8sclient (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-k8sclient - 0.4.0-0ubuntu1

---------------
python-k8sclient (0.4.0-0ubuntu1) artful; urgency=medium

  * d/gbp.conf: Update gbp configuration file.
  * d/control: Update Vcs-* links and maintainers.
  * New upstream release (LP: #1659420).

 -- Corey Bryant <email address hidden> Tue, 26 Sep 2017 14:04:35 -0400

Changed in python-k8sclient (Ubuntu Artful):
status: Triaged → Fix Released
Revision history for this message
Billy Olsen (billy-olsen) wrote :

Its been reported to me from a 3rd party that an update of the package to the latest 0.4.0 release of the python-k8sclient resolves this issue for them. They've been running for at least 24 hours and have not seen this traceback occur in any of their workloads.

tags: added: sts
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Jason, or anyone else affected,

Accepted python-k8sclient into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:pike-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-pike-needed
Revision history for this message
Spyros Trigazis (strigazi) wrote :

Hi Corey,

since the ocata release magnum is using python-kubernetes and not python-k8sclient

Magnum users and packager from ocata or later should use python-kubernetes.

[1] https://pypi.python.org/pypi/python-k8sclient/0.4.0
[2] https://pypi.python.org/pypi/kubernetes/4.0.0a1

Cheers,
Spyros

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hi Spyros,

Thanks. We didn't have a python-kubernetes package at one point so the Ubuntu package is still using python-k8sclient with a minor patch to do that. We'll get magnum switched to using python-kubernetes this cycle. I opened a bug to track it: https://bugs.launchpad.net/cloud-archive/+bug/1726883.

Thanks,
Core

Revision history for this message
Spyros Trigazis (strigazi) wrote :

Thanks Corey, I subscribed :)

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hi Jason or Billy, If the reported bug is fixed in pike-proposed, would one of you be able to tag this bug as verification-pike-done? Thanks.

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

Gentle ping again for Jason or Billy to confirm this fix

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.