maas: 1:3.3.2-13177-g.a73a6e2bd-0ubuntu1~22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840
MAAS 3.3.2 (and 3.3.1 at least) returns PTR records in an inconsistent way for LXD containers created by Juju. Those LXD interfaces are managed as "devices" in MAAAS.
This behavior doesn't happen with 3.2.7.
$ juju run --all 'hostname -I'
- MachineId: "0"
Stdout: "10.206.50.1 192.168.151.120 \n"
- MachineId: 0/lxd/0
Stdout: "192.168.151.121 \n"
- MachineId: 0/lxd/1
Stdout: "192.168.151.123 \n"
- MachineId: 0/lxd/2
Stdout: "192.168.151.122 \n"
$ juju run --all 'dig +short -x $(hostname -I)'
- MachineId: "0"
Stdout: |
large-wolf.
large-wolf.local.
- MachineId: 0/lxd/0
Stdout: |
juju-f8d90a-0-lxd-0.maas.
- MachineId: 0/lxd/1
Stdout: |
juju-f8d90a-0-lxd-1.maas.
eth0.juju-f8d90a-0-lxd-1.maas.
- MachineId: 0/lxd/2
Stdout: |
juju-f8d90a-0-lxd-2.maas.
eth0.juju-f8d90a-0-lxd-2.maas.
^^^ MAAS DNS returns two PTR records with and without "eth0" for one IP address for some LXD containers (not all).
How to reproduce:
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
https://launchpadlibrarian.net/657634345/keystone-ha_vault_edge.yaml
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status
[original description]
There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>
> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname().
https://github.com/juju/charm-helpers/blob/6e302bab63e22e356d77d76a5c6d90d9d24c6390/charmhelpers/contrib/network/ip.py#L497
In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file.
https://github.com/juju/charm-helpers/blob/b5725ac546372e7d4004d15095f79cdd5e7da687/charmhelpers/contrib/openstack/cert_utils.py#L105
[requested and written cert]
/etc/apache2/ssl/keystone/cert_eth0.juju-043209-2-lxd-0.maas (w/ eth0.)
-> exists
[the patch trying to create a symlink to]
/etc/apache2/ssl/keystone/cert_juju-043209-2-lxd-0.maas (w/o eth0.)
-> does not exist
unit-keystone-2: 12:45:41 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'eth0.juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': '{"eth0.juju-043209-2-lxd-0.maas": {"sans": ["192.168.151.131", "192.168.151.99"]}}', 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': {'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}}, 'unit_name': 'keystone_2'}
This is due to how MAAS DNS works.
$ dig +short @192.168.151.1 -x 192.168.151.131
eth0.juju-043209-2-lxd-0.maas.
juju-043209-2-lxd-0.maas.
$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/
/var/lib/bind/maas/zone.maas-$ORIGIN maas.
/var/lib/bind/maas/zone.maas:juju-043209-2-lxd-0 A 192.168.151.131
/var/lib/bind/maas/zone.maas:$ORIGIN juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.maas-eth0 A 192.168.151.131
--
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-129 PTR juju-fe03b8-2-lxd-8.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa:131 PTR eth0.juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa: PTR juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-134 PTR eth0.juju-043209-0-lxd-0.maas.
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
The step 3&4 can be unattended as follows.
====
juju destroy-model keystone-test --no-wait --force -y; \
juju add-model keystone-test maas && \
juju deploy ./keystone-
VAULT_ADDR="http://
export VAULT_ADDR
vault_init_
vault operator unseal "$(echo "$vault_
VAULT_TOKEN="$(echo "$vault_
export VAULT_TOKEN
juju run-action --wait vault/leader authorize-charm \
token="$(vault token create -ttl=10m -format json | jq -r .auth.client_
juju run-action vault/leader --wait generate-root-ca
====