Original default route in DVR SNAT namespace is not recovered after creating and deleting /0 extraroute

Hi,

Thx for reporting that bug. I think we had similar discussion regarding e.g allowed_address_pairs and security groups. Basically it's up to the user what routes he will configure in own router.
If we will forbid route to 0.0.0.0/0 but allow e.g. 1.0.0.0/8 we will basically still allow them to break connectivity to significant part of the Internet, right?
So IMHO this should be documented but not disallowed in the code.

Fix proposed to branch: master
Review: https://review.opendev.org/760314

Hello:

As Slawek commented, this is a user input. We can warn about it but never prevent it.

Regards.

OK, seems we have the consensus that the API should allow 0.0.0.0/0. We recently meet such error. But the bad thing is that even user removed the 0.0.0.0/0 routes, the connectivity is not recovered. So IMO, we should fix the bug in L3 agent side, when user remove the intended 0.0.0.0/0 routes, the DVR related default route should be recovered automatically. This should be the main issue of this bug. The default route is not recovered after the API removal of the route 0.0.0.0/0.

I guess that default route must be only present in a dvr environment, right?

Because I tried to reproduce it, but couldn't. My current devstack was built with no dvr and that way there's no problem:

$ openstack router create router2
$ router_id="$( openstack router show router2 -f value -c id )"

$ openstack network create net2
$ openstack subnet create --network net2 --use-default-subnet-pool subnet2
$ openstack router add subnet router2 subnet2

# no default route in the first place
$ sudo ip netns exec "qrouter-$router_id" ip route
10.0.0.64/26 dev qr-283bcf49-c2 proto kernel scope link src 10.0.0.65

$ openstack subnet show subnet2
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| allocation_pools | 10.0.0.66-10.0.0.126 |
| cidr | 10.0.0.64/26 |
| created_at | 2020-10-29T15:17:57Z |
| description | |
| dns_nameservers | |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 10.0.0.65 |
| host_routes | |
| id | 7bfb1bc6-e5d9-4f57-a4f1-118cfbc2eb0b |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | subnet2 |
| network_id | e46a48c2-4458-48c6-ad21-bdf47a9638b5 |
| prefix_length | None |
| project_id | 928bf5e466e245039d17bd2f5c3139bd |
| revision_number | 0 |
| segment_id | None |
| service_types | |
| subnetpool_id | 4d5019e3-fe19-4433-b491-bc40cb07fe04 |
| tags | |
| updated_at | 2020-10-29T15:17:57Z |
+----------------------+--------------------------------------+

$ openstack router set router2 --no-route --route destination=0.0.0.0/0,gateway=10.0.0.70
$ sudo ip netns exec "qrouter-$router_id" ip route
default via 10.0.0.70 dev qr-283bcf49-c2 proto static·
10.0.0.64/26 dev qr-283bcf49-c2 proto kernel scope link src 10.0.0.65

$ openstack router set router2 --no-route
# no default route as before
$ sudo ip netns exec "qrouter-$router_id" ip route
10.0.0.64/26 dev qr-283bcf49-c2 proto kernel scope link src 10.0.0.65

Would this be the way to reproduce it if I had dvr? I guess I should rebuild my devstack with dvr.

Answering my own question: The way I tried to reproduce the issue is missing setting an external gw network on the router, that's why there's no default route in the first place. Will add that and re-try in a dvr environment.

In a DVR environment with an external gateway I was able to reproduce the part of the bug Liu wrote about in comment #4. Please see below:

$ openstack router create router2
$ router_id="$( openstack router show router2 -f value -c id )"

$ openstack network create net2
$ openstack subnet create --network net2 --use-default-subnet-pool subnet2
$ openstack router set --external-gateway public router2
$ openstack router add subnet router2 subnet2

# we have a default route to start with
$ sudo ip netns exec "snat-$router_id" ip route
default via 172.24.4.1 dev qg-9235492c-11 proto static
10.0.0.64/26 dev sg-7ef787ed-d0 proto kernel scope link src 10.0.0.80
172.24.4.0/24 dev qg-9235492c-11 proto kernel scope link src 172.24.4.40

$ openstack subnet show subnet2
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| allocation_pools | 10.0.0.66-10.0.0.126 |
| cidr | 10.0.0.64/26 |
| created_at | 2020-10-29T15:17:57Z |
| description | |
| dns_nameservers | |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 10.0.0.65 |
| host_routes | |
| id | 7bfb1bc6-e5d9-4f57-a4f1-118cfbc2eb0b |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | subnet2 |
| network_id | e46a48c2-4458-48c6-ad21-bdf47a9638b5 |
| prefix_length | None |
| project_id | 928bf5e466e245039d17bd2f5c3139bd |
| revision_number | 0 |
| segment_id | None |
| service_types | |
| subnetpool_id | 4d5019e3-fe19-4433-b491-bc40cb07fe04 |
| tags | |
| updated_at | 2020-10-29T15:17:57Z |
+----------------------+--------------------------------------+

$ openstack router set router2 --no-route --route destination=0.0.0.0/0,gateway=10.0.0.70

# default route is overriden by /0 extraroute
$ sudo ip netns exec "snat-$router_id" ip route
default via 10.0.0.70 dev sg-7ef787ed-d0 proto static
10.0.0.64/26 dev sg-7ef787ed-d0 proto kernel scope link src 10.0.0.80
172.24.4.0/24 dev qg-9235492c-11 proto kernel scope link src 172.24.4.40

$ openstack router set router2 --no-route

# original default route is not recovered after deleting /0 extraroute
$ sudo ip netns exec "snat-$router_id" ip route
10.0.0.64/26 dev sg-7ef787ed-d0 proto kernel scope link src 10.0.0.80
172.24.4.0/24 dev qg-9235492c-11 proto kernel scope link src 172.24.4.40

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/760314
Reason: Feel free to restore this patch if needed.