[ovn] metadata route missing on the guest

Hi Przemysław Lal,

What you mean by "affected" network? Do you mean there are multiple networks in your setup, and out of those only one such network is misbehaving in terms of routes?

If above is true, is the "affected" network misbehaving since when it's created, or it used to work earlier and stopped working later? What other differences are there in affected and unaffected networks/subnets?

Following information would be good to collect for affected network, by dropping the workarounds:-
- openstack network show <network id>
- openstack subnet show <subnet id>
- openstack port list --device-owner network:distributed --network <affected network>
- ovn-nbctl find DHCP_Options external_ids:subnet_id=<subnet-id>

And also same info from unaffected network.

Hi Przemysław:

Apart from what Yatin requested, that is necessary to debug this issue, can you confirm "ovn_metadata_enabled" is True? I guess it is because other subnets have this port, but doesn't harm to double check.

Just in case, please check if you have [1] and [2] in your code. If you changed the subnet parameters ("dhcp_enable") without those patches, you could be in an undefined state now.

Can you check also in the server logs, during the network creation, if the metadata port was created too? [3].

Regards.

[1]https://review.opendev.org/q/I05394e49077a72199bbc80c8cb622ec2b17f2fa7
[2]https://review.opendev.org/q/I09cc14dff6933aae63cbd43a29f9221f405ecede
[3]https://github.com/openstack/neutron/blob/e7b70521d0e230143a80974e7e4795a2acafcc9b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1753

Status changed to 'Confirmed' because the bug affects multiple users.

We hit this issue today as well. Same symptoms:

* Failed to get metadata during VM launch - consistently and only on the "affected" network. Other networks like "unaffected" are OK.
* Missing metadata route inside VM
* After adding the route manually to the .2 IP we can ping/curl the metadata endpoint with no issues, so it seems the route is the only thing missing.
* The workaround of adding the metadata route explicitly to the relevant router allows new VMs in the affected network to get metadata without problems.

These are the current packages:

ii neutron-common 2:16.4.2-0ubuntu1
ii neutron-ovn-metadata-agent 2:16.4.2-0ubuntu1
ii python3-neutron 2:16.4.2-0ubuntu1
ii python3-neutron-lib 2.3.0-0ubuntu1
ii python3-neutronclient 1:7.1.1-0ubuntu1

I am attaching the information requested above for an "affected" and "unaffected" network. The main difference I see is that the "unaffected" subnet has the following option in the ovn-nb that is missing from the "affected" subnet:

classless_static_route="{169.254.169.254/32,10.131.83.2, 0.0.0.0/0,10.131.83.1}"

The two patches you mention are indeed included in python3-neutron 2:16.4.2-0ubuntu1. I additionally confirmed by checking /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py

Regarding "ovn_metadata_enabled", I didn't find it set to "true" in any config under /etc/neutron. I can only see the default commented out and no mention in neutron_ovn_metadata_agent.ini, which has the ovs/ovn config in it (but I am no expert)

/etc/neutron# grep -r ovn_metadata
ovn.ini:#ovn_metadata_enabled = false

The creation logs are no longer available. The ports for the .2 IPs are created in the subnet, and they do have a device_id of ovnmeta-<networkid>, but the device_owner is network:dhcp and not network:distributed as you seem to be expecting. I added the output of `port show` for them as well. Note that other networks on the same compute nodes have no issues providing metadata, including the "unaffected" network (data attached).

unaffected network/subnet information

affected network/subnet information

Actually I can confirm ovn_metadata_enabled is set to "True". I was looking in the wrong place (compute/metadata agent server) and this seems to be set in the API server node:

/etc/neutron# grep -r ovn_metadata
plugins/ml2/ml2_conf.ini:ovn_metadata_enabled = True
ovn.ini:#ovn_metadata_enabled = false

Looks like you are using Ussuri.

In case of OVN the route is not supposed to be used: https://docs.openstack.org/networking-ovn/latest/contributor/design/metadata_api.html

Instead OVN does all the proxying in OVN layer (see above)