OpenStackSDK refactoring caused various OSC networking commands to fail

Bug #1652317 reported by Valeriy Ponomaryov
88
This bug affects 16 people
Affects Status Importance Assigned to Milestone
OpenStack SDK
New
Undecided
Unassigned
OpenStack Shared File Systems Service (Manila)
Invalid
Critical
Unassigned
python-openstackclient
New
High
Unassigned
tacker
Fix Released
High
Unassigned

Bug Description

After merging upper constraint change for python-openstacksdk project (which is used by python-openstackclient) 0.9.10 -> 0.9.11
 [1] our devstack installation started failing where we try to create security group for manila.
here is logs:
http://logs.openstack.org/86/414286/2/gate/gate-manila-tempest-minimal-dsvm-lvm-ubuntu-xenial/a79dc50/logs/devstacklog.txt.gz#_2016-12-23_13_26_10_182

Where we see following error:

$ openstack security group create manila-service --description 'manila-service description'
'SecurityGroup' object has no attribute 'keys'

So, it can be said, that it is regression problem in sdk project, because it started returning class instances instead of dicts as was before.

[1] https://review.openstack.org/#/c/414366/

[UPD]
One more breakage:

$ openstack security group rule list manila-service
invalid literal for int() with base 10: '137:139'

Tags: devstack
tags: added: devstack
Changed in manila:
importance: Undecided → Critical
milestone: none → ocata-3
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to manila (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/414610

description: updated
Revision history for this message
Brian Curtin (brian.curtin) wrote : Re: Creation of security group in devstack is broken using openstackclient

Not an SDK bug

Changed in python-openstacksdk:
status: New → Invalid
Revision history for this message
Valeriy Ponomaryov (vponomaryov) wrote :

Brian, why do you think so? It is not enough just to say it is not bug of SDK project.
Why do you think it is not "regression" bug where was changed type of objects returned by SDK?

Revision history for this message
Brian Curtin (brian.curtin) wrote :

backwards compatibility isn't guaranteed pre-1.0, this was intentionally changed as part of a refactoring.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to manila (master)

Reviewed: https://review.openstack.org/414610
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=10b14bb50e95d2a30bba4e4c37a7de9c4b564691
Submitter: Jenkins
Branch: master

commit 10b14bb50e95d2a30bba4e4c37a7de9c4b564691
Author: vponomaryov <email address hidden>
Date: Fri Dec 23 17:07:39 2016 +0200

    [Devstack] Workaround osclient breakage

    Creation of security group using openstack client is broken. So,
    use nova client for it temporary while 'related' bug is not fixed.

    Change-Id: I51b5be5157bd49d5835b30eaa8b6b0e7185cabdf
    Related-Bug: #1652317

summary: - Creation of security group in devstack is broken using openstackclient
+ OpenStackSDK refactoring caused various OSC networking commands to fail
Changed in python-openstackclient:
importance: Undecided → High
Revision history for this message
Steve Martinelli (stevemar) wrote :

To summarize:

We're going to block SDK v0.9.11 for now, since the following commands will fail:

 - quota_set
 - quota show
 - network agent set
 - network agent show
 - network qos *
 - network qos rule *
 - port *
 - security group *
 - security group rule *

With SDK v0.9.10 installed, the following commands will fail:

 - qos_rule_type_list
 - network_service_provider_list
 - ip_availability_show

The latter seems more acceptable since it won't block any gates, and fewer impacted commands overall. We're going to re-work the commands to work with both versions of the SDK.

This was unfortunately not caught by our functional tests as they were not running (problem in the test hook), they are running now. See https://review.openstack.org/#/c/414652/.

Revision history for this message
Sridhar Ramaswamy (srics-r) wrote :

Affecting tacker gate functional tests

Changed in tacker:
importance: Undecided → High
Revision history for this message
Sridhar Ramaswamy (srics-r) wrote :

Is there an ETA for this bug? This is blocking tacker dsvm tests

Revision history for this message
Andrey Kurilin (andreykurilin) wrote :

@srics-r: after merging https://review.openstack.org/#/c/414621/ all gates should be unblocked. Can you check openstacksdk version in broken gates?

Revision history for this message
Dean Troyer (dtroyer) wrote :

https://review.openstack.org/#/c/415508/ reverts the skip for the commands that broke in SDK 0.9.11. This leaves the following still skipped for SDK 0.9.10:

 - qos_rule_type_list (https://bugs.launchpad.net/python-openstackclient/+bug/1653137)
 - network_service_provider_list (https://bugs.launchpad.net/python-openstackclient/+bug/1653138)
 - ip_availability_show (https://bugs.launchpad.net/python-openstackclient/+bug/1653139)

Moving forward, individual bugs for these have been opened (as listed above) to track their resulution.

Revision history for this message
Sridhar Ramaswamy (srics-r) wrote :

@andreykurilin: confirming the tacker gate tests are passing now, thanks!

Changed in tacker:
status: New → Fix Committed
Revision history for this message
Andrey Kurilin (andreykurilin) wrote :

Moved openstacksdk bug-report to New, since latest release 0.9.12 broke updating neutorn quotas again(it just doesn't fix issues of 0.9.11 release)

Changed in python-openstacksdk:
status: Invalid → New
Revision history for this message
Andrey Kurilin (andreykurilin) wrote :
Download full text (6.7 KiB)

Logs that can help for investigation:

openstack --debug --os-interface admin quota set --networks -1 --subnets -1 --routers -1 --floating-ips -1 --subnetpools -1 --secgroups -1 --secgroup-rules -1 --ports -1 rally-test-project-1

ESP BODY: {"NeutronError": {"message": "Unrecognized attribute(s) 'id'", "type": "HTTPBadRequest", "detail": ""}}
2017-01-09 13:34:17.292661 | 2017-01-09 13:34:17.291 |
2017-01-09 13:34:17.293790 | 2017-01-09 13:34:17.293 | Request returned failure status: 400
2017-01-09 13:34:17.294862 | 2017-01-09 13:34:17.294 | HttpException: Bad Request
2017-01-09 13:34:17.295931 | 2017-01-09 13:34:17.295 | Traceback (most recent call last):
2017-01-09 13:34:17.297009 | 2017-01-09 13:34:17.296 | File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
2017-01-09 13:34:17.298138 | 2017-01-09 13:34:17.297 | result = cmd.run(parsed_args)
2017-01-09 13:34:17.299359 | 2017-01-09 13:34:17.299 | File "/usr/local/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
2017-01-09 13:34:17.300660 | 2017-01-09 13:34:17.300 | return super(Command, self).run(parsed_args)
2017-01-09 13:34:17.301795 | 2017-01-09 13:34:17.301 | File "/usr/local/lib/python2.7/dist-packages/cliff/command.py", line 90, in run
2017-01-09 13:34:17.303381 | 2017-01-09 13:34:17.302 | return self.take_action(parsed_args) or 0
2017-01-09 13:34:17.304780 | 2017-01-09 13:34:17.304 | File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/quota.py", line 185, in take_action
2017-01-09 13:34:17.305888 | 2017-01-09 13:34:17.305 | **network_kwargs)
2017-01-09 13:34:17.307049 | 2017-01-09 13:34:17.306 | File "/usr/local/lib/python2.7/dist-packages/openstack/network/v2/_proxy.py", line 2025, in update_quota
2017-01-09 13:34:17.308530 | 2017-01-09 13:34:17.308 | return self._update(_quota.Quota, quota, **attrs)
2017-01-09 13:34:17.309591 | 2017-01-09 13:34:17.309 | File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 36, in check
2017-01-09 13:34:17.310703 | 2017-01-09 13:34:17.310 | return method(self, expected, actual, *args, **kwargs)
2017-01-09 13:34:17.311864 | 2017-01-09 13:34:17.311 | File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 173, in _update
2017-01-09 13:34:17.313098 | 2017-01-09 13:34:17.312 | return res.update(self.session)
2017-01-09 13:34:17.314221 | 2017-01-09 13:34:17.313 | File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 663, in update
2017-01-09 13:34:17.315444 | 2017-01-09 13:34:17.315 | json=request.body, headers=request.headers)
2017-01-09 13:34:17.316935 | 2017-01-09 13:34:17.316 | File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 726, in put
2017-01-09 13:34:17.318078 | 2017-01-09 13:34:17.317 | return self.request(url, 'PUT', **kwargs)
2017-01-09 13:34:17.319321 | 2017-01-09 13:34:17.318 | File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 77, in map_exceptions_wrapper
2017-01-09 13:34:17.320508 | 2017-01-09 13:34:17.320 | http_status=e.http_status, cause=e)
2017-01-09 13:34:17.321794 | 2017-01-09 13:34:17.321 | Htt...

Read more...

Revision history for this message
Clark Boylan (cboylan) wrote :

Related to this the nodepool/shade/ansible set of jobs began to fail. They rely the demo user that devstack sets up and expect things like networking to be preconfigured. It appears that these jobs started failing when OSC was updated and have begun passing now that that version is excluded again.

It appears that they failed because the demo user did not have a valid "private" network, but did have subnets for the private networks. These subnets were then attached to private networks "owned" by the admin user. So somewhere the client/sdk/api crossed the wires on that and ended up "wiring" up the networking incorrectly for the demo user.

Revision history for this message
Doug Wilmot (dwilmot) wrote :

Also impacts openstack subnet show:

openstack subnet show dec8c058-a578-4927-abd5-19780fd9dee1 (returns 'Subnet' object has no attribute 'keys')

However the older neutron cmd still works but rather not modify our scripts

[root@OpenStack-Launcher openstack-ns]# neutron subnet-show dec8c058-a578-4927-abd5-19780fd9dee1 <<<<
+-------------------+----------------------------------------------------+
| Field | Value |
+-------------------+----------------------------------------------------+
| allocation_pools | {"start": "11.255.255.2", "end": "11.255.255.254"} |
| cidr | 11.255.255.0/24 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 11.255.255.1 |
| host_routes | |
| id | dec8c058-a578-4927-abd5-19780fd9dee1 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | basic7-dwnew-subnet_dummy-or2jrteviwzr |
| network_id | f1c13eac-4a17-4c86-82a3-61a865195da4 |
| tenant_id | 8d33baa96b78444ab266538f6c4a23e0 |
+-------------------+----------------------------------------------------+

Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :

@Doug : I verified Openstack SDK 0.9.12 and subnet show seems to work fine

Successfully installed openstacksdk-0.9.12 python-openstacksdk-0.5.2
[outofmemory@localhost devstack]$ openstack subnet show private-subnet
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| allocation_pools | 10.0.0.2-10.0.0.62 |
| cidr | 10.0.0.0/26 |
| created_at | 2017-01-13T16:17:57Z |
| description | |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.0.1 |
| host_routes | |
| id | d8136226-acc7-4ab3-b808-fba25533cba7 |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | private-subnet |
| network_id | f77a1585-dd37-4072-b330-503b6eb2b043 |
| project_id | 25f495a77ad1450c90cb1b1ac1442b2f |
| revision_number | 2 |
| segment_id | None |
| service_types | |
| subnetpool_id | c1b01670-1559-4494-a708-47fcf10ffc3d |
| updated_at | 2017-01-13T16:17:57Z |
+-------------------+--------------------------------------+

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-openstacksdk (master)

Reviewed: https://review.openstack.org/419911
Committed: https://git.openstack.org/cgit/openstack/python-openstacksdk/commit/?id=0a7483ed952c82d006b0c55ce1ff77212a0c7729
Submitter: Jenkins
Branch: master

commit 0a7483ed952c82d006b0c55ce1ff77212a0c7729
Author: Reedip <email address hidden>
Date: Wed Jan 18 03:49:23 2017 -0500

    Fix Setting Quotas in Neutron

    Currently Quota Set command doenst work in SDK version 0.9.12
    as the request formed for the Neutron API is not correct.
    This patch attempts to fix the same.

    Change-Id: Id58b05bcdbfee73cb9b93dd5533b4c7d93dd03aa
    Partial-Bug:#1652317
    Closes-Bug:#1655445

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/420301
Committed: https://git.openstack.org/cgit/openstack/python-openstacksdk/commit/?id=6d185df4983ef824d629769c11dde5dd9a6297b9
Submitter: Jenkins
Branch: master

commit 6d185df4983ef824d629769c11dde5dd9a6297b9
Author: Reedip <email address hidden>
Date: Wed Jan 18 03:52:23 2017 -0500

    Add project ID in QuotaDefault requests

    Currently QuotaDefault doesnt need ID to work with,
    but it needs the URI to have the project ID so that it
    can reference the same in the Neutron API.
    This patch fixes this issue.

    Partial-Bug:#1652317
    Change-Id: I9288b1c9fc64c6326231d535d9c8574b24afeb3a

Changed in manila:
milestone: ocata-3 → ocata-rc1
Revision history for this message
Valeriy Ponomaryov (vponomaryov) wrote :

Manila project not affected anymore, fix in lib solved problem.

Changed in manila:
status: New → Invalid
Changed in manila:
milestone: ocata-rc1 → none
Changed in tacker:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.