odl client 404 on port update

Bug #1713487 reported by Michel Peterson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-odl
Confirmed
High
Unassigned

Bug Description

Relevant logs @ http://logs.openstack.org/38/496638/7/check/gate-rally-dsvm-networking-odl-carbon-snapshot/d1f32b9/logs/

q-svc log
=========

 INFO networking_odl.journal.journal [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] Processing - update port 5cd5a60a-06a2-4274-a7b0-b88ed937a49f
 DEBUG networking_odl.common.client [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] Sending METHOD (put) URL (http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f) JSON ({
   "port": {
     "allowed_address_pairs": [],
     "extra_dhcp_opts": [],
     "updated_at": "2017-08-27T11:58:14Z",
     "device_owner": "network:dhcp",
     "revision_number": 3,
     "port_security_enabled": false,
     "binding:profile": {},
     "fixed_ips": [
       {
         "subnet_id": "e272c54e-43be-47aa-acd0-c2aaab596862",
         "ip_address": "10.2.15.2"
       }
     ],
     "security_groups": [],
     "binding:vif_details": {
       "support_vhost_user": false,
       "has_datapath_type_netdev": false,
       "uuid": "9aafa204-dacc-4a42-80c6-a3a05de43393",
       "host_addresses": [
         "ubuntu-xenial-inap-mtl01-10650239"
       ]
     },
     "binding:vif_type": "ovs",
     "mac_address": "fa:16:3e:05:4e:41",
     "binding:host_id": "ubuntu-xenial-inap-mtl01-10650239",
     "description": "",
     "tags": [],
     "device_id": "dhcp8981c717-8b49-521d-af27-33139dcc5f8e-f3d90580-4bef-4d7c-b32b-af4576f42e67",
     "name": "",
     "admin_state_up": true,
     "created_at": "2017-08-27T11:58:06Z",
     "binding:vnic_type": "normal"
   }
 }) {{(pid=20895) request /opt/stack/new/networking-odl/networking_odl/common/client.py:89}}
 DEBUG neutron.scheduler.dhcp_agent_scheduler [None req-b815a515-fef2-4e46-8dce-99ab43cf0dd5 c_rally_f2ccc531_1M2EcU8x c_rally_f2ccc531_8fzScwr5] Network 0414e07d-aaf2-4abc-843b-3da23b8eefa1 is scheduled to be hosted by DHCP agent 30f28597-4e0a-4350-833e-7df8e1aec4e0 {{(pid=20892) bind /opt/stack/new/neutron/neutron/scheduler/dhcp_agent_scheduler.py:199}}
 DEBUG neutron.scheduler.dhcp_agent_scheduler [None req-fc80da8c-ad42-4dee-aa0e-aad6636f2f12 c_rally_f2ccc531_1M2EcU8x c_rally_f2ccc531_8fzScwr5] Network 39a52e67-9195-44ff-8d67-4d75c85fa703 is scheduled to be hosted by DHCP agent 30f28597-4e0a-4350-833e-7df8e1aec4e0 {{(pid=20892) bind /opt/stack/new/neutron/neutron/scheduler/dhcp_agent_scheduler.py:199}}
 DEBUG networking_odl.common.client [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] Exception from ODL: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f Port UUID does not exist. {{(pid=20895) _check_response /opt/stack/new/networking-odl/networking_odl/common/client.py:140}}
 ERROR networking_odl.common.client Traceback (most recent call last):
 ERROR networking_odl.common.client File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 136, in _check_response
 ERROR networking_odl.common.client response.raise_for_status()
 ERROR networking_odl.common.client File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 935, in raise_for_status
 ERROR networking_odl.common.client raise HTTPError(http_error_msg, response=self)
 ERROR networking_odl.common.client HTTPError: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f
 ERROR networking_odl.common.client
 ERROR networking_odl.common.client [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] REST request ( put ) to url ( ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f ) is failed. Request body : [{u'port': {'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'updated_at': '2017-08-27T11:58:14Z', 'device_owner': u'network:dhcp', 'revision_number': 3, 'port_security_enabled': False, 'binding:profile': {}, 'fixed_ips': [{'subnet_id': u'e272c54e-43be-47aa-acd0-c2aaab596862', 'ip_address': u'10.2.15.2'}], 'security_groups': [], 'binding:vif_details': {u'support_vhost_user': False, u'has_datapath_type_netdev': False, u'uuid': u'9aafa204-dacc-4a42-80c6-a3a05de43393', u'host_addresses': [u'ubuntu-xenial-inap-mtl01-10650239']}, 'binding:vif_type': u'ovs', 'mac_address': u'fa:16:3e:05:4e:41', 'binding:host_id': u'ubuntu-xenial-inap-mtl01-10650239', 'description': u'', 'tags': [], 'device_id': u'dhcp8981c717-8b49-521d-af27-33139dcc5f8e-f3d90580-4bef-4d7c-b32b-af4576f42e67', 'name': u'', 'admin_state_up': True, 'created_at': '2017-08-27T11:58:06Z', 'binding:vnic_type': u'normal'}}] service: HTTPError: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f
 ERROR networking_odl.journal.journal [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] Error while processing update port 5cd5a60a-06a2-4274-a7b0-b88ed937a49f: HTTPError: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f
 ERROR networking_odl.journal.journal Traceback (most recent call last):
 ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/journal/journal.py", line 245, in _sync_entry
 ERROR networking_odl.journal.journal self.client.sendjson(method, urlpath, to_send)
 ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 106, in sendjson
 ERROR networking_odl.journal.journal 'body': obj})
 ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
 ERROR networking_odl.journal.journal self.force_reraise()
 ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
 ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb)
 ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 98, in sendjson
 ERROR networking_odl.journal.journal self.request(method, urlpath, data))
 ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 140, in _check_response
 ERROR networking_odl.journal.journal {'e': error, 'text': response.text}, exc_info=1)
 ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
 ERROR networking_odl.journal.journal self.force_reraise()
 ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
 ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb)
 ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 136, in _check_response
 ERROR networking_odl.journal.journal response.raise_for_status()
 ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 935, in raise_for_status
 ERROR networking_odl.journal.journal raise HTTPError(http_error_msg, response=self)
 ERROR networking_odl.journal.journal HTTPError: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f
 ERROR networking_odl.journal.journal

Changed in networking-odl:
importance: Undecided → High
Mike Kolesnik (mkolesni)
Changed in networking-odl:
assignee: nobody → Mike Kolesnik (mkolesni)
Revision history for this message
Mike Kolesnik (mkolesni) wrote :

While indeed alerting, there isn't enough information in the logs to identify exactly what caused the bug (current suspicion is some problem in the dependency calculations, but without more info it's hard to know for sure).

I'll send a fix to add more information in the logging that will allow us to further understand such scenarios.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-odl (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/514354

Changed in networking-odl:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-odl (master)

Fix proposed to branch: master
Review: https://review.openstack.org/514355

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-odl (master)

Reviewed: https://review.openstack.org/514354
Committed: https://git.openstack.org/cgit/openstack/networking-odl/commit/?id=beeaf24dc8342c70c19baa72f8881e663ddd1973
Submitter: Zuul
Branch: master

commit beeaf24dc8342c70c19baa72f8881e663ddd1973
Author: Mike Kolesnik <email address hidden>
Date: Thu Oct 19 15:06:58 2017 +0300

    Have create_pending_row return the entry

    This change is needed in order to be able to print the entry's ID to the
    log once it's created, in order to have better debug-ability of the logs
    (we'll be able to correlate entries better if we know the ID and not
    only the object ID which can have multiple entries).

    A follow up patch makes use of this ability.

    Change-Id: I8eb82a5e5c0124c1eaf5fb3cbf32e12fd169e7ff
    Related-Bug: #1713487

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-odl (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/515998

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-odl (master)

Reviewed: https://review.openstack.org/514355
Committed: https://git.openstack.org/cgit/openstack/networking-odl/commit/?id=5df60d55a33a08a2e892ac6df84681e68a323adc
Submitter: Zuul
Branch: master

commit 5df60d55a33a08a2e892ac6df84681e68a323adc
Author: Mike Kolesnik <email address hidden>
Date: Mon Oct 23 18:03:11 2017 +0300

    Log additional info about entries

    In order to pinpoint bugs better it's necessary to log more details
    about the entries. The entry id will now be logged to better correlate
    the entry processing.
    Also dependencies will be logged to better understand if bugs in
    dependency calculations exist.

    Change-Id: I9078487bd4722132e92b48d7d333c93c82a002d9
    Partial-Bug: #1713487

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-odl (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/518532

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-odl (stable/pike)

Reviewed: https://review.openstack.org/515998
Committed: https://git.openstack.org/cgit/openstack/networking-odl/commit/?id=803a953031036279ad8bac6cd65f580c25a63482
Submitter: Zuul
Branch: stable/pike

commit 803a953031036279ad8bac6cd65f580c25a63482
Author: Mike Kolesnik <email address hidden>
Date: Thu Oct 19 15:06:58 2017 +0300

    Have create_pending_row return the entry

    This change is needed in order to be able to print the entry's ID to the
    log once it's created, in order to have better debug-ability of the logs
    (we'll be able to correlate entries better if we know the ID and not
    only the object ID which can have multiple entries).

    A follow up patch makes use of this ability.

    Change-Id: I8eb82a5e5c0124c1eaf5fb3cbf32e12fd169e7ff
    Related-Bug: #1713487
    (cherry picked from commit beeaf24dc8342c70c19baa72f8881e663ddd1973)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-odl (stable/pike)

Reviewed: https://review.openstack.org/518532
Committed: https://git.openstack.org/cgit/openstack/networking-odl/commit/?id=77cbc291fb689ce01e88963ce8e34e4df3dfe2bf
Submitter: Zuul
Branch: stable/pike

commit 77cbc291fb689ce01e88963ce8e34e4df3dfe2bf
Author: Mike Kolesnik <email address hidden>
Date: Mon Oct 23 18:03:11 2017 +0300

    Log additional info about entries

    In order to pinpoint bugs better it's necessary to log more details
    about the entries. The entry id will now be logged to better correlate
    the entry processing.
    Also dependencies will be logged to better understand if bugs in
    dependency calculations exist.

    Change-Id: I9078487bd4722132e92b48d7d333c93c82a002d9
    Partial-Bug: #1713487
    (cherry picked from commit 5df60d55a33a08a2e892ac6df84681e68a323adc)

Changed in networking-odl:
assignee: Mike Kolesnik (mkolesni) → nobody
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.