kolla-ansible

vpnaas not working on centos8-stream on xena

Bug #1988574 reported by Jacolex on 2022-09-02

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Incomplete	Undecided	Unassigned
	neutron	New	Undecided	Unassigned

Bug Description

Hello

After configuring VPN Endpoint, the l3 agent has problem with start the vpn service:

2022-09-02 13:54:02.390 654 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec [-] Failed to enable vpn process on router 3659d2d3-5c2e-4097-92dc-08f1567524f5: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'qrouter-3659d2d3-5c2e-4097-92dc-08f1567524f5', '/var/lib/kolla/venv/bin/neutron-vpn-netns-wrapper', '--mount_paths=/etc:/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc,/var/run:/var/lib/neutron/ipsec/3659d2d3-5c2e-409
7-92dc-08f1567524f5/var/run', '--rootwrap_config=/etc/neutron/rootwrap.conf', '--cmd=ipsec,_stackmanager,start']; Stdin: ; Stdout: 2022-09-02 13:54:01.673 88268 INFO neutron.common.config [-] Logging enabled!ESC[00m
2022-09-02 13:54:01.674 88268 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-vpn-netns-wrapper version 19.3.1.dev44ESC[00m
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc', '/etc'] Exit code: 0 Stdout: Stderr: 2022-09-02 13:54:01.693 88268 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc has been bind-mounted in /etcESC[00m
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run', '/var/run'] Exit code: 0 Stdout: Stderr: 2022-09-02 13:54:01.714 88268 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run has been bind-mounted in /var/runESC[00m
Command: ['ipsec', '_stackmanager', 'start'] Exit code: 1 Stdout: Stderr: cannot load config '/etc/ipsec.conf': /etc/ipsec.conf:3: syntax error, unexpected STRING [nat_traversal]
cannot load config '/etc/ipsec.conf': /etc/ipsec.conf:3: syntax error, unexpected STRING [nat_traversal]

So I did the workaround putting into
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_vpnaas/services/vpn/device_drivers/template/openswan/ipsec.conf.template

config setup
#nat_traversal=yes

After that the second problem appeared:

2022-09-02 13:41:35.252 35 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec [req-aa8d3095-578e-4747-a708-d55d3a4ff889 7a8ec6fc4ec12049bb7f243a354430b4b5ecc5a3fedcdc1c555f1f1a5ce70eb5 715cf7f57a6f47119161fe0654ed8a1c - - -] Failed to enable vpn process on router 3659d2d3-5c2e-4097-92dc-08f1567524f5: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'qrouter-3659d2d3-5c2e-4097-92dc-08f1567524f5', '/var/lib/kolla/venv/bin/neutron-vpn-netns-w
rapper', '--mount_paths=/etc:/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc,/var/run:/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run', '--rootwrap_config=/etc/neutron/rootwrap.conf', '--cmd=ipsec,pluto,--use-netkey,--uniqueids']; Stdin: ; Stdout: 2022-09-02 13:41:34.832 14537 INFO neutron.common.config [-] Logging enabled!ESC[00m
2022-09-02 13:41:34.834 14537 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-vpn-netns-wrapper version 19.3.1.dev44ESC[00m
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc', '/etc'] Exit code: 0 Stdout: Stderr: 2022-09-02 13:41:34.845 14537 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc has been bind-mounted in /etcESC[00m
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run', '/var/run'] Exit code: 0 Stdout: Stderr: 2022-09-02 13:41:34.856 14537 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run has been bind-mounted in /var/runESC[00m
Command: ['ipsec', 'pluto', '--use-netkey', '--uniqueids'] Exit code: 1 Stdout: Stderr: /usr/libexec/ipsec/pluto: unrecognized option '--use-netkey'
For usage information: /usr/libexec/ipsec/pluto --help
Libreswan 4.5

So I deployed the second workaround in
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py

    def start_pluto(self):
        cmd = ['pluto',
               '--use-netkey',
               '--uniqueids']

And removed --use-netkey:
    def start_pluto(self):
        cmd = ['pluto',
               '--uniqueids']

After that the vpn endpoint starts working correctly.
Seems there is some problems with libreswan version.
Image version:
quay.io/openstack.kolla/centos-source-neutron-l3-agent
"build-date": "20220726",

Tags:

Revision history for this message

Jeffrey Zhang (jeffrey4l) wrote on 2022-11-28 (last edit on 2022-11-28):

the same issue with https://bugs.launchpad.net/neutron/+bug/1938571

Revision history for this message

Ian Kumlien (pomac) wrote on 2023-01-09:

So, from what i can see only openswan has support for --use-netkey - which means that if you support free/libre-swan then it should be removed.

Revision history for this message

Jacolex (jacolex) wrote on 2023-01-09 (last edit on 2023-01-09):

Based on kolla scripts - libreswan should be supported on centos installation

/usr/local/share/kolla-ansible/ansible/roles/neutron/templates/l3_agent.ini.j2

#jinja2: trim_blocks: False
{% set vpn_device_driver = 'neutron_vpnaas.services.vpn.device_drivers.strongswan_ipsec.StrongSwanDriver' if kolla_base_distro in ['ubuntu', 'debian'] else 'neutron_vpnaas.services.vpn.device_drivers.libreswan_ipsec.LibreSwanDriver'%}

[...]

{% if enable_neutron_vpnaas | bool %}
[ipsec]
enable_detailed_logging = {{ neutron_logging_debug }}

[vpnagent]
vpn_device_driver = {{ vpn_device_driver }}
{% endif %}

[...]

Revision history for this message

Ian Kumlien (pomac) wrote on 2023-01-10:

Ok, so this should work with libreswan but not with freeswan.

Since the official documentation is useless:
https://libreswan.org/man/ipsec_pluto.8.html
(linked from: https://libreswan.org/man/)

Lets use this instead:
https://manpages.debian.org/unstable/libreswan/pluto.8.en.html

Which clearly states that it should be available at least in 4.3 -- if it's not in 4.5 then something has changed... And since it's in mainline and was added 2014-04-15... I don't know.

However is the netkey behavior something that neutron expects or can it just be ignored?

Revision history for this message

Ian Kumlien (pomac) wrote on 2023-01-10:

As a side note:
libreswan-4.5-1.el8.x86_64

ipsec pluto --help |grep netkey |wc -l
0

ipsec --version
Linux Libreswan 4.5 (XFRM) [cut kernel version]

Revision history for this message

Ian Kumlien (pomac) wrote on 2023-01-10:

So a colleague looked around a bit and it will not work with centos-*-stream (both 8 and 9)

It will however work in Ubuntu

Revision history for this message

Ian Kumlien (pomac) wrote on 2023-01-10:

Apparently it's been removed from Ubuntu 22.10 ;)

Revision history for this message

Michal Nasiadka (mnasiadka) wrote on 2023-01-31:

I believe it's a neutron-vpnaas bug, not kolla-ansible bug.

Changed in kolla-ansible:
status:	New → Incomplete

Revision history for this message

Lajos Katona (lajos-katona) wrote on 2023-02-06:

Am I understand right that on some distros has different swan libs like openswan libreswan etc..., and they differ in what options they support?
If my understanding is right, (without looking into the code) it should be handled in the driver itself.

Revision history for this message

Ian Kumlien (pomac) wrote on 2023-03-15:

#10

Some older versions of ipsec had a command you could use, it's gone in the packaged versions of ipsec, so yes, it could be handled - it could also just be removed.

Revision history for this message

Ian Kumlien (pomac) wrote on 2023-04-17:

#11

Also:
nat_traversal

OBSOLETE. Support for NAT Traversal is always enabled.

https://manpages.debian.org/experimental/libreswan/ipsec.conf.5.en.html

Error log:
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec Command: ['ipsec', '_stackmanager', 'start'] Exit code: 1 Stdout: Stderr: cannot load config '/etc/ipsec.conf': /etc/ipsec.conf:3: syntax error, unexpected STRING [nat_traversal]
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec cannot load config '/etc/ipsec.conf': /etc/ipsec.conf:3: syntax error, unexpected STRING [nat_traversal]
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec unknown stack
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec ; Stderr:
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.