OVN metadata use the same IP as LB health monitor

Recently I ran into an issue that instances are unable to get metadata, after a few debugs I found out the issue to be the LB health monitor is replying the arp request for OVN metadata agent's IP causing the request going to metadata agent actually being sent to another MAC address.

Form sb_db logical flow dump I found the bellow flow

_uuid : df93e919-c7f5-4d04-be6f-796e175e266f
actions : "eth.dst = eth.src; eth.src = 5e:69:e7:47:38:7b; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = 5e:69:e7:47:38:7b; arp.tpa = arp.spa; arp.spa = 10.0.0.2; outport = inport; flags.loopback = 1; output;"
controller_meter : []
external_ids : {source="northd.c:7971", stage-hint="5b11468a", stage-name=ls_in_arp_rsp}
logical_datapath : f714c506-3043-4366-8065-d6af4d2a08de
logical_dp_group : []
match : "arp.tpa == 10.0.0.2 && arp.op == 1"
pipeline : ingress
priority : 110
table_id : 18
tags : {}
hash : 0

Priority 110 flow with this action should be from the LB health monitor according to ovn-northd doc [1]

Where as ovn-metadata agent also uses the same IP (10.0.0.2)

+-------------------------+---------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+---------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | |
| binding_profile | |
| binding_vif_details | |
| binding_vif_type | unbound |
| binding_vnic_type | normal |
| created_at | 2023-01-17T08:04:35Z |
| data_plane_status | None |
| description | |
| device_id | ovnmeta-5640d643-e490-46b3-a724-0478aef3722e |
| device_owner | network:distributed |
| device_profile | None |
| dns_assignment | fqdn='host-10-0-0-2.infra.cloudnative.tw.', hostname='host-10-0-0-2', ip_address='10.0.0.2' |
| dns_domain | |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.0.0.2', subnet_id='3000b8bf-f687-406d-b000-55d38f365e33' |
| id | c1ba8ceb-6f02-48e6-9da3-3cb9b628b771 |
| ip_allocation | None |
| mac_address | fa:16:3e:7c:b8:1c |
| name | |
| network_id | 5640d643-e490-46b3-a724-0478aef3722e |
| numa_affinity_policy | None |
| port_security_enabled | False |
| project_id | 2fbc86f895ed4ef1ad036b7e4a068b50 |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 2 |
| security_group_ids | |
| status | DOWN |
| tags | |
| trunk_details | None |
| updated_at | 2023-01-17T08:04:36Z |
+-------------------------+---------------------------------------------------------------------------------------------+

This causes all packet that should be going to 10.0.0.2 (metadata agent) actually being sent to 5e:69:e7:47:38:7b MAC address which is incorrect.

Expected result:

OVN LB health monitor sending arp replies to an unused IP, or reserved a port with an IP specifically for health monitor.

[1] https://www.ovn.org/support/dist-docs/ovn-northd.8.html