Ubuntu
octavia package

amphora-agent package, its init script and configuration file

Bug #1831642 reported by Frode Nordahl
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
octavia (Ubuntu)
Triaged
High
Unassigned

Bug Description

1)
At present the ``amphora-agent`` package has a init script that points the amphora-agent at ``/etc/octavia/octavia.conf``. However, the upstream Octavia project will tell cloud-init to write the agent configuration to ``/etc/octavia/amphora-agent.conf``.

Upstream reference:
https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/tasks/compute_tasks.py#L89

Config-drive contents:
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:~# mount /dev/sr0 /mnt
mount: /mnt: WARNING: device write-protected, mounted read-only.
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:~# cd /mnt
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt# ls -l
total 4
dr-xr-xr-x 4 root root 2048 Jun 4 13:03 ec2
dr-xr-xr-x 12 root root 2048 Jun 4 13:03 openstack
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt# cd openstack/
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack# ls -l
total 20
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2012-08-10
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2013-04-04
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2013-10-17
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2015-10-15
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2016-06-30
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2016-10-06
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2017-02-22
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 2018-08-27
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 content
dr-xr-xr-x 2 root root 2048 Jun 4 13:03 latest
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack# cd latest
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/latest# ls -l
total 4
-r--r--r-- 1 root root 2136 Jun 4 13:03 meta_data.json
-r--r--r-- 1 root root 506 Jun 4 13:03 network_data.json
-r--r--r-- 1 root root 2 Jun 4 13:03 vendor_data.json
-r--r--r-- 1 root root 14 Jun 4 13:03 vendor_data2.json
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/latest# cat meta_data.json
{"uuid": "0aedb1b1-46a1-4734-aa1c-f1aed62f427f", "files": [{"path": "/etc/octavia/amphora-agent.conf", "content_path": "/content/0000"}, {"path": "/etc/octavia/certs/client_ca.pem", "content_path": "/content/0001"}, {"path": "/etc/octavia/certs/server.pem", "content_path": "/content/0002"}], "admin_pass": "boHNY7N32jAb", "public_keys": {"test": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi1Mwv77hDyu/68syJflcPO6oKGkUBzjBIcPwC1hNdWvVHyaBI2lm6millJIBVQdLYHh/MJe6FRz6iOOZ7eBnnzd63D8GOTS6UnpX+F7BmT8ICH1CBqbnVAn+LJxCfzs8+Q706jTXqkiU3Rgl7A6v8+Tqqu6VsI+4/lODj43LXhlN4+tt3jvbNCXXo0d4jRBEysEnINjsmxcNce3sCeAaDY1trC8DsGAFUJxoPi8We2qYPVvHbWj7yLgDzJULf05TEIxR1IvbaU/2K+eT2ITyPdTgsSCP7E2DAK8WtKiOWJRBiOwE46yT6Os/rAjoIrzv9IqPDnp97uxf1Ia1jD4Ah ubuntu@test\n"}, "keys": [{"name": "test", "type": "ssh", "data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCi1Mwv77hDyu/68syJflcPO6oKGkUBzjBIcPwC1hNdWvVHyaBI2lm6millJIBVQdLYHh/MJe6FRz6iOOZ7eBnnzd63D8GOTS6UnpX+F7BmT8ICH1CBqbnVAn+LJxCfzs8+Q706jTXqkiU3Rgl7A6v8+Tqqu6VsI+4/lODj43LXhlN4+tt3jvbNCXXo0d4jRBEysEnINjsmxcNce3sCeAaDY1trC8DsGAFUJxoPi8We2qYPVvHbWj7yLgDzJULf05TEIxR1IvbaU/2K+eT2ITyPdTgsSCP7E2DAK8WtKiOWJRBiOwE46yT6Os/rAjoIrzv9IqPDnp97uxf1Ia1jD4Ah ubuntu@test\n"}], "hostname": "amphora-82c4877a-d932-49af-b590-7859c6edb84c.novalocal", "name": "amphora-82c4877a-d932-49af-b590-7859c6edb84c", "launch_index": 0, "availability_zone": "nova", "random_seed": "TbGbfew/Ie7ZwKL2CIOYTiJ5mMAEddIsbzAK3Tg8aa6pitSZiY67oLXgadY7TUhNGon5ZsW7hnUCmAweHbNMXsYOs5jkMljURsuh55ihJb7+Uz57kNTQaMaLi6mLy0PmHXdhpzWs3PJRxddQGtzLa4k1j7dX4O4otViD4kHwVp8K/ZfDORVd8Ot+xOP9oYkvTCf76kjGl/z8Jnwhm83YlOGvjcYxKVEKAy87QAb7k0qxzcai8as1zwck3nyHe6Frl/lHelywoTadSelXkWUCqxfbN/594bTjQobMZPp2O4iixGjeQMzU0utOGJM+AYG9VjZvI9uZ9T3WW+eERn9vg+TOk0mF7bCTNgtzdd2K5zBthTf+g1+3rBNWp4/iSqbHrA+PHK2CEKwiM/5SIarW4/raoBrsDMep47P2ax4XMcZKuTXKZjcnht1hOblDRKtW65fgELkv5/aixCJmD4k5ks9u4FDJgWxdGRZzTNSpWbmnz+qPOk2R+/n3KkIeevQHR3U341Xq4Oyxff1d/9+unRNU6ujC7o7HyDynlnTYcjOHlUcg1KW5Wl7PZ7qredWVmNgwTdotNPJJQe6E9TZjjeVxXBzTxHGoZuvzAKkew5r60Ky2LyUAm+sZiv5Y9pp/P55Hejntg5U9JvVq870m+dqVbLeqdRWYT3G3ysLX/HI=", "project_id": "964a97ca99c6489b817dd8fb8b6ee740", "devices": []}root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/latest# cd ..
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack# cd content/
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/content# ls -l
total 6
-r--r--r-- 1 root root 727 Jun 4 13:03 0000
-r--r--r-- 1 root root 1135 Jun 4 13:03 0001
-r--r--r-- 1 root root 2875 Jun 4 13:03 0002
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/content# cat 0000

[DEFAULT]
debug = True

[haproxy_amphora]
base_cert_dir = /var/lib/octavia/certs
base_path = /var/lib/octavia
bind_host = ::
bind_port = 9443
haproxy_cmd = /usr/sbin/haproxy
respawn_count = 2
respawn_interval = 2
use_upstart = True

[health_manager]
controller_ip_port_list = fc00:4f4a:d87f:9a6c:f816:3eff:fe58:3dd0:5555, fc00:4f4a:d87f:9a6c:f816:3eff:fee0:a2d9:5555, fc00:4f4a:d87f:9a6c:f816:3eff:fee4:669b:5555
heartbeat_interval = 10
heartbeat_key = e211b7a6-f98a-42be-a4c6-b0d5893fb566

[amphora_agent]
agent_server_ca = /etc/octavia/certs/client_ca.pem
agent_server_cert = /etc/octavia/certs/server.pem
agent_request_read_timeout = 120
amphora_id = 82c4877a-d932-49af-b590-7859c6edb84c
amphora_udp_driver = keepalived_lvsroot@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/mnt/openstack/content# cd /etc/octavia/
root@amphora-82c4877a-d932-49af-b590-7859c6edb84c:/etc/octavia# ls -l
total 84
-rw-rw---- 1 root root 727 Jun 4 13:03 amphora-agent.conf
drwxr-xr-x 2 root root 4096 Jun 4 13:03 certs
-rw-r----- 1 root octavia 68324 Jun 4 12:05 octavia.conf
-rw-r----- 1 root octavia 5290 Jun 4 12:05 policy.json

Due to how the init script is laid out it is not simple to override this through ``/etc/default/amphora-agent`` or ``/etc/default/openstack`` as those files are read in too late.

Can we either fix the location of the configuration file in the init script or expose the ``CONFIG_FILE`` variable from ``/etc/default/amphora-agent``?

2)
The package creates the ``/etc/octavia`` directory with 0o750 permissions, subsequently when cloud-init writes out the ``/etc/octavia/amphora-agent.conf`` from the config drive the Octavia service provides it will get 0o640 permissions.

Given the file is owned by root:root the amphora-agent service does not have access to read this configuration file.

Could we either make ``/etc/octavia`` mode 0o755 or 2o750 (group sticky bit) to alleviate this?

3)
For some reason the amphora-agent attempts to write to ``/var/log/amphora-agent.log`` although the init script does pass --log-file=/ver/log/octavia/amphora-agent.log.

Still investigating this, some bits:
# systemctl status amphora-agent
● amphora-agent.service - OpenStack Octavia Amphora Agent (amphora-agent)
   Loaded: loaded (/lib/systemd/system/amphora-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-06-04 16:26:53 UTC; 324ms ago
 Main PID: 1591 (/usr/bin/python)
    Tasks: 1 (limit: 614)
   CGroup: /system.slice/amphora-agent.service
           └─1591 /usr/bin/python3.6 /usr/bin/amphora-agent --config-file=/etc/octavia/amphora-agent.conf --log-file=/var/log/octavia/amphora-agent.log

-- The start-up result is RESULT.
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f amphora-agent[1649]: tcmsg: [Errno 2] No such file or directory: '/proc/net/psched'
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f amphora-agent[1649]: the tc subsystem functionality is limited
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f amphora-agent[1649]: Error: Error: '/var/log/amphora-agent.log' isn't writable [PermissionError(13, 'Permission denied')]
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f systemd[1]: amphora-agent.service: Main process exited, code=exited, status=1/FAILURE
Jun 04 16:26:56 amphora-f4d05447-0603-4f51-9d35-9f302dc4665f systemd[1]: amphora-agent.service: Failed with result 'exit-code'.

For some of these we have limited control over how the Octavia service instructs the amphora instance to run things as it is in full control over config-drive contents etc. If not change our defaults, we should at least make it easily configurable so a Octavia Amphora can be built using packages from UCA.

I'll keep investigating this and put forward a patch as soon as all the pieces necessary for a working amphora from packages is in place.

Frode Nordahl (fnordahl)
summary: - amphora-agent package and configuration file
+ amphora-agent package, it's init script and configuration file
summary: - amphora-agent package, it's init script and configuration file
+ amphora-agent package, its init script and configuration file
Revision history for this message
Frode Nordahl (fnordahl) wrote :

For 3) the upstream project has the ``/var/log/amphora-agent.log`` path hard-coded, ref: https://opendev.org/openstack/octavia/src/branch/master/octavia/cmd/agent.py#L80

Revision history for this message
Frode Nordahl (fnordahl) wrote :

The ``amphora-agent`` run in the load-balancer instances expects to run as root. This is what the images built with octavia from source does.

This also explains how the hard coded path to ``/var/log/amphora-agent.log`` works with the built-from-source images.

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

Hi Frode,

1) This should be a simple change in the package for the amphora-agent init to use /etc/octavia/amphora-agent.conf.

2) These permissions have changed since this bug was reported. /etc/octavia now gets:

root@g1:~# ls -al /etc/octavia
total 36
drwxr-x--- 2 root octavia 4 Jul 17 17:51 .
drwxr-xr-x 100 root root 191 Jul 17 17:51 ..
-rw-r----- 1 root octavia 61659 Jul 17 17:51 octavia.conf
-rw-r----- 1 root octavia 7184 Jul 17 17:51 policy.json

So perhaps that is now solved?

3) Use of /var/log/amphora-agent.log feels non-standard but I'm not finding proof to back that argument up. Packages generally put log files in /var/log/<package>/, ie. /var/log/octavia/amphora-agent.log. I did a quick search of debian standards and filesystem hierarchy standards but didn't find anything. I'm wondering if this should get changed upstream or if we could carry a patch in ubuntu to patch the code to use /var/log/octavia/amphora-agent.log (unless a patch upstream makes sense).

Thanks,
Corey

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

Will switch status while waiting for further info.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Sorry was logged in as the wrong user, it's me!

Changed in octavia (Ubuntu):
importance: Undecided → High
status: New → Triaged
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Actually, for 1), I was thinking the upstream source might ship with a sample or have the ability to generate amphora-agent.conf but it doesn't appear to. So I don't know that the package can install in /etc/octavia/amphora-agent.conf. Maybe the current permissions will allow the project source to write what it needs?

Revision history for this message
Corey Bryant (corey.bryant) wrote :

But even in that case, it seems odd for the package to ship an init file that uses a conf file that isn't shipped by the package.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.