In those new logs, the destination port binding, that should be updated by Nova, is not sent [1]. We can see all events related to the OVS agents; for example when the port is deleted from the source agent:
11816:Jan 11 18:30:16.756385 ubuntu-jammy-inmotion-iad3-0032701749 neutron-server[55411]: DEBUG neutron.plugins.ml2.rpc [None req-f8d8130a-4979-4abc-b93b-bf722c7985de None None] Device 9f5bf1ec-480c-412b-9a4e-b2aa0356b4fa no longer exists at agent ovs-agent-ubuntu-jammy-inmotion-iad3-0032701750 {{(pid=55411) update_device_down /opt/stack/neutron/neutron/plugins/ml2/rpc.py:259}}
11823:Jan 11 18:30:16.836648 ubuntu-jammy-inmotion-iad3-0032701749 neutron-server[55411]: DEBUG neutron.plugins.ml2.plugin [None req-f8d8130a-4979-4abc-b93b-bf722c7985de None None] Current status of the port 9f5bf1ec-480c-412b-9a4e-b2aa0356b4fa is: ACTIVE; New status is: DOWN {{(pid=55411) _update_individual_port_db_status /opt/stack/neutron/neutron/plugins/ml2/plugin.py:2338}}
Or when the same port is bound to the destination OVS agent, just after the previous messages:
11825:Jan 11 18:30:16.854061 ubuntu-jammy-inmotion-iad3-0032701749 neutron-server[55411]: DEBUG neutron.plugins.ml2.rpc [None req-b9822962-cf31-4229-81ea-d1d4b008a142 None None] Device 9f5bf1ec-480c-412b-9a4e-b2aa0356b4fa up at agent ovs-agent-ubuntu-jammy-inmotion-iad3-0032701749 {{(pid=55411) update_device_up /opt/stack/neutron/neutron/plugins/ml2/rpc.py:296}}
11827:Jan 11 18:30:16.883051 ubuntu-jammy-inmotion-iad3-0032701749 neutron-server[55411]: DEBUG neutron.notifiers.nova [None req-f8d8130a-4979-4abc-b93b-bf722c7985de None None] device_id is not set on port 9f5bf1ec-480c-412b-9a4e-b2aa0356b4fa yet. {{(pid=55411) _can_notify /opt/stack/neutron/neutron/notifiers/nova.py:201}}
11829:Jan 11 18:30:16.892256 ubuntu-jammy-inmotion-iad3-0032701749 neutron-server[55411]: DEBUG neutron.plugins.ml2.plugin [None req-b9822962-cf31-4229-81ea-d1d4b008a142 None None] The host ubuntu-jammy-inmotion-iad3-0032701749 is not matching for port 9f5bf1ec-480c-412b-9a4e-b2aa0356b4fa host ubuntu-jammy-inmotion-iad3-0032701750! {{(pid=55411) port_bound_to_host /opt/stack/neutron/neutron/plugins/ml2/plugin.py:2436}}
11830:Jan 11 18:30:16.892256 ubuntu-jammy-inmotion-iad3-0032701749 neutron-server[55411]: DEBUG neutron.plugins.ml2.rpc [None req-b9822962-cf31-4229-81ea-d1d4b008a142 None None] Device 9f5bf1ec-480c-412b-9a4e-b2aa0356b4fa not bound to the agent host ubuntu-jammy-inmotion-iad3-0032701749 {{(pid=55411) update_device_up /opt/stack/neutron/neutron/plugins/ml2/rpc.py:304}}
11832:Jan 11 18:30:16.912446 ubuntu-jammy-inmotion-iad3-0032701749 neutron-server[55411]: DEBUG neutron.plugins.ml2.db [None req-f8d8130a-4979-4abc-b93b-bf722c7985de None None] For port 9f5bf1ec-480c-412b-9a4e-b2aa0356b4fa, host ubuntu-jammy-inmotion-iad3-0032701750, got binding levels [PortBindingLevel(driver='openvswitch',host='ubuntu-jammy-inmotion-iad3-0032701750',level=0,port_id=9f5bf1ec-480c-412b-9a4e-b2aa0356b4fa,segment=NetworkSegment(b71f2156-82ee-4e1c-ab74-d5cbdae3d781),segment_id=b71f2156-82ee-4e1c-ab74-d5cbdae3d781)] {{(pid=55411) get_binding_level_objs /opt/stack/neutron/neutron/plugins/ml2/db.py:74}}
However the port binding is not updated and the port is never set to up because the RPC update from the destination agent doesn't match with the port binding host information.
Hello Slawek:
In those new logs, the destination port binding, that should be updated by Nova, is not sent [1]. We can see all events related to the OVS agents; for example when the port is deleted from the source agent: jammy-inmotion- iad3-0032701749 neutron- server[ 55411]: DEBUG neutron. plugins. ml2.rpc [None req-f8d8130a- 4979-4abc- b93b-bf722c7985 de None None] Device 9f5bf1ec- 480c-412b- 9a4e-b2aa0356b4 fa no longer exists at agent ovs-agent- ubuntu- jammy-inmotion- iad3-0032701750 {{(pid=55411) update_device_down /opt/stack/ neutron/ neutron/ plugins/ ml2/rpc. py:259} } jammy-inmotion- iad3-0032701749 neutron- server[ 55411]: DEBUG neutron. plugins. ml2.plugin [None req-f8d8130a- 4979-4abc- b93b-bf722c7985 de None None] Current status of the port 9f5bf1ec- 480c-412b- 9a4e-b2aa0356b4 fa is: ACTIVE; New status is: DOWN {{(pid=55411) _update_ individual_ port_db_ status /opt/stack/ neutron/ neutron/ plugins/ ml2/plugin. py:2338} }
11816:Jan 11 18:30:16.756385 ubuntu-
11823:Jan 11 18:30:16.836648 ubuntu-
Or when the same port is bound to the destination OVS agent, just after the previous messages: jammy-inmotion- iad3-0032701749 neutron- server[ 55411]: DEBUG neutron. plugins. ml2.rpc [None req-b9822962- cf31-4229- 81ea-d1d4b008a1 42 None None] Device 9f5bf1ec- 480c-412b- 9a4e-b2aa0356b4 fa up at agent ovs-agent- ubuntu- jammy-inmotion- iad3-0032701749 {{(pid=55411) update_device_up /opt/stack/ neutron/ neutron/ plugins/ ml2/rpc. py:296} } jammy-inmotion- iad3-0032701749 neutron- server[ 55411]: DEBUG neutron. notifiers. nova [None req-f8d8130a- 4979-4abc- b93b-bf722c7985 de None None] device_id is not set on port 9f5bf1ec- 480c-412b- 9a4e-b2aa0356b4 fa yet. {{(pid=55411) _can_notify /opt/stack/ neutron/ neutron/ notifiers/ nova.py: 201}} jammy-inmotion- iad3-0032701749 neutron- server[ 55411]: DEBUG neutron. plugins. ml2.plugin [None req-b9822962- cf31-4229- 81ea-d1d4b008a1 42 None None] The host ubuntu- jammy-inmotion- iad3-0032701749 is not matching for port 9f5bf1ec- 480c-412b- 9a4e-b2aa0356b4 fa host ubuntu- jammy-inmotion- iad3-0032701750 ! {{(pid=55411) port_bound_to_host /opt/stack/ neutron/ neutron/ plugins/ ml2/plugin. py:2436} } jammy-inmotion- iad3-0032701749 neutron- server[ 55411]: DEBUG neutron. plugins. ml2.rpc [None req-b9822962- cf31-4229- 81ea-d1d4b008a1 42 None None] Device 9f5bf1ec- 480c-412b- 9a4e-b2aa0356b4 fa not bound to the agent host ubuntu- jammy-inmotion- iad3-0032701749 {{(pid=55411) update_device_up /opt/stack/ neutron/ neutron/ plugins/ ml2/rpc. py:304} } jammy-inmotion- iad3-0032701749 neutron- server[ 55411]: DEBUG neutron. plugins. ml2.db [None req-f8d8130a- 4979-4abc- b93b-bf722c7985 de None None] For port 9f5bf1ec- 480c-412b- 9a4e-b2aa0356b4 fa, host ubuntu- jammy-inmotion- iad3-0032701750 , got binding levels [PortBindingLev el(driver= 'openvswitch' ,host=' ubuntu- jammy-inmotion- iad3-0032701750 ',level= 0,port_ id=9f5bf1ec- 480c-412b- 9a4e-b2aa0356b4 fa,segment= NetworkSegment( b71f2156- 82ee-4e1c- ab74-d5cbdae3d7 81),segment_ id=b71f2156- 82ee-4e1c- ab74-d5cbdae3d7 81)] {{(pid=55411) get_binding_ level_objs /opt/stack/ neutron/ neutron/ plugins/ ml2/db. py:74}}
11825:Jan 11 18:30:16.854061 ubuntu-
11827:Jan 11 18:30:16.883051 ubuntu-
11829:Jan 11 18:30:16.892256 ubuntu-
11830:Jan 11 18:30:16.892256 ubuntu-
11832:Jan 11 18:30:16.912446 ubuntu-
However the port binding is not updated and the port is never set to up because the RPC update from the destination agent doesn't match with the port binding host information.
This is not, if I'm not wrong, a Neutron error.
[1]https:/ /storage. gra.cloud. ovh.net/ v1/AUTH_ dcaab5e32b234d5 6b626f72581e364 4c/zuul_ opendev_ logs_558/ 867769/ 11/gate/ neutron- ovs-tempest- multinode- full/558cfa3/ controller/ logs/screen- q-svc.txt