There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>
> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status
The step 3&4 can be unattended as follows.
====
juju destroy-model keystone-test --no-wait --force -y; \
juju add-model keystone-test maas && \
juju deploy ./keystone-ha_vault_edge.yaml && time juju-wait -w --exclude vault
VAULT_ADDR="http://$(juju run --unit vault/leader -- network-get certificates --ingress-address):8200"
export VAULT_ADDR
maas: 1:3.3.1- 13169-g. 94920eb1e- 0ubuntu1~ 22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840
There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition: ssl/*/cert_ <fqdn> ssl/*/cert_ <vip> to /etc/apache2/ ssl/*/cert_ <hostname> ssl/*/cert_ <vip>
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/
- symlink creation fails from /etc/apache2/
- apache2 fails to start because of missing /etc/apache2/
> $ sudo systemctl status apache2 system/ apache2. service; enabled; vendor preset: enabled) /httpd. apache. org/docs/ 2.4/ sites-enabled/ openstack_ https_frontend. conf: ssl/keystone/ cert_192. 168.151. 99' does not exist or is empty
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https:/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname(). /github. com/juju/ charm-helpers/ blob/6e302bab63 e22e356d77d76a5 c6d90d9d24c6390 /charmhelpers/ contrib/ network/ ip.py#L497
https:/
In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file. /github. com/juju/ charm-helpers/ blob/b5725ac546 372e7d4004d1509 5f79cdd5e7da687 /charmhelpers/ contrib/ openstack/ cert_utils. py#L105
https:/
[requested and written cert] ssl/keystone/ cert_eth0. juju-043209- 2-lxd-0. maas (w/ eth0.)
/etc/apache2/
-> exists
[the patch trying to create a symlink to] ssl/keystone/ cert_juju- 043209- 2-lxd-0. maas (w/o eth0.)
/etc/apache2/
-> does not exist
unit-keystone-2: 12:45:41 WARNING unit.keystone/ 2.juju- log certificates:10: get_request: self.hostname_ entry={ 'cn': 'eth0.juju- 043209- 2-lxd-0. maas', 'addresses': ['192.168.151.131', '192.168. 151.99' ]},self. entries= [{'cn': 'eth0.juju- 043209- 2-lxd-0. maas', 'addresses': ['192.168.151.131', '192.168. 151.99' ]}],sans= ['192.168. 151.131' , '192.168. 151.99' ],request= {'eth0. juju-043209- 2-lxd-0. maas': {'sans': ['192.168.151.131', '192.168. 151.99' ]}},req= {'cert_ requests' : '{"eth0. juju-043209- 2-lxd-0. maas": {"sans": ["192.168.151.131", "192.168. 151.99" ]}}', 'unit_name': 'keystone_2'} 2.juju- log certificates:10: get_request: self.hostname_ entry={ 'cn': 'juju-043209- 2-lxd-0. maas', 'addresses': ['192.168.151.131', '192.168. 151.99' ]},self. entries= [{'cn': 'juju-043209- 2-lxd-0. maas', 'addresses': ['192.168.151.131', '192.168. 151.99' ]}],sans= ['192.168. 151.131' , '192.168. 151.99' ],request= {'juju- 043209- 2-lxd-0. maas': {'sans': ['192.168.151.131', '192.168. 151.99' ]}},req= {'cert_ requests' : {'juju- 043209- 2-lxd-0. maas': {'sans': ['192.168.151.131', '192.168. 151.99' ]}}, 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/
This is due to how MAAS DNS works.
$ dig +short @192.168.151.1 -x 192.168.151.131 043209- 2-lxd-0. maas. 2-lxd-0. maas.
eth0.juju-
juju-043209-
$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/ bind/maas/ zone.maas- $ORIGIN maas. bind/maas/ zone.maas: juju-043209- 2-lxd-0 A 192.168.151.131 bind/maas/ zone.maas: $ORIGIN juju-043209- 2-lxd-0. maas. bind/maas/ zone.maas- eth0 A 192.168.151.131 bind/maas/ zone.151. 168.192. in-addr. arpa-129 PTR juju-fe03b8- 2-lxd-8. maas. bind/maas/ zone.151. 168.192. in-addr. arpa:131 PTR eth0.juju- 043209- 2-lxd-0. maas. bind/maas/ zone.151. 168.192. in-addr. arpa: PTR juju-043209- 2-lxd-0. maas. bind/maas/ zone.151. 168.192. in-addr. arpa-134 PTR eth0.juju- 043209- 0-lxd-0. maas.
/var/lib/
/var/lib/
/var/lib/
/var/lib/
--
/var/lib/
/var/lib/
/var/lib/
/var/lib/
>>> str(dns. resolver. query(dns. reversename. from_address( "192.168. 151.131" ), "PTR")[0]) 2-lxd-0. maas.' resolver. query(dns. reversename. from_address( "192.168. 151.131" ), "PTR")[0]) 043209- 2-lxd-0. maas.' resolver. query(dns. reversename. from_address( "192.168. 151.131" ), "PTR")[0]) 2-lxd-0. maas.' resolver. query(dns. reversename. from_address( "192.168. 151.131" ), "PTR")[0]) 043209- 2-lxd-0. maas.' resolver. query(dns. reversename. from_address( "192.168. 151.131" ), "PTR")[0]) 2-lxd-0. maas.'
'juju-043209-
>>> str(dns.
'eth0.juju-
>>> str(dns.
'juju-043209-
>>> str(dns.
'eth0.juju-
>>> str(dns.
'juju-043209-
How to reproduce:
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status
The step 3&4 can be unattended as follows. ha_vault_ edge.yaml && time juju-wait -w --exclude vault
====
juju destroy-model keystone-test --no-wait --force -y; \
juju add-model keystone-test maas && \
juju deploy ./keystone-
VAULT_ADDR="http:// $(juju run --unit vault/leader -- network-get certificates --ingress- address) :8200"
export VAULT_ADDR
vault_init_ output= "$(vault operator init -key-shares=1 -key-threshold=1 -format json)" init_output" | jq -r .unseal_ keys_b64[ ])"
vault operator unseal "$(echo "$vault_
VAULT_TOKEN="$(echo "$vault_ init_output" | jq -r .root_token)"
export VAULT_TOKEN
juju run-action --wait vault/leader authorize-charm \ token)"
token="$(vault token create -ttl=10m -format json | jq -r .auth.client_
juju run-action vault/leader --wait generate-root-ca
====