| 2023-03-26 02:54:56 |
Nobuto Murata |
bug |
|
|
added bug |
| 2023-03-26 02:56:31 |
Nobuto Murata |
attachment added |
|
keystone-ha_vault_edge.yaml https://bugs.launchpad.net/charm-keystone/+bug/2012801/+attachment/5657438/+files/keystone-ha_vault_edge.yaml |
|
| 2023-03-26 02:57:16 |
Nobuto Murata |
description |
maas: 1:3.3.1-13169-g.94920eb1e-0ubuntu1~22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840
There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>
> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname().
https://github.com/juju/charm-helpers/blob/6e302bab63e22e356d77d76a5c6d90d9d24c6390/charmhelpers/contrib/network/ip.py#L497
In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file.
https://github.com/juju/charm-helpers/blob/b5725ac546372e7d4004d15095f79cdd5e7da687/charmhelpers/contrib/openstack/cert_utils.py#L105
[requested and written cert]
/etc/apache2/ssl/keystone/cert_eth0.juju-043209-2-lxd-0.maas (w/ eth0.)
-> exists
[the patch trying to create a symlink to]
/etc/apache2/ssl/keystone/cert_juju-043209-2-lxd-0.maas (w/o eth0.)
-> does not exist
unit-keystone-2: 12:45:41 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'eth0.juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': '{"eth0.juju-043209-2-lxd-0.maas": {"sans": ["192.168.151.131", "192.168.151.99"]}}', 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': {'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}}, 'unit_name': 'keystone_2'}
This is due to how MAAS DNS works.
$ dig +short @192.168.151.1 -x 192.168.151.131
eth0.juju-043209-2-lxd-0.maas.
juju-043209-2-lxd-0.maas.
$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/
/var/lib/bind/maas/zone.maas-$ORIGIN maas.
/var/lib/bind/maas/zone.maas:juju-043209-2-lxd-0 A 192.168.151.131
/var/lib/bind/maas/zone.maas:$ORIGIN juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.maas-eth0 A 192.168.151.131
--
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-129 PTR juju-fe03b8-2-lxd-8.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa:131 PTR eth0.juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa: PTR juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-134 PTR eth0.juju-043209-0-lxd-0.maas.
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
How to reproduce:
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status
The step 3&4 can be unattended as follows.
====
juju destroy-model keystone-test --no-wait --force -y; \
juju add-model keystone-test maas && \
juju deploy ./keystone-ha_vault_edge.yaml && time juju-wait -w --exclude vault
VAULT_ADDR="http://$(juju run --unit vault/leader -- network-get certificates --ingress-address):8200"
export VAULT_ADDR
vault_init_output="$(vault operator init -key-shares=1 -key-threshold=1 -format json)"
vault operator unseal "$(echo "$vault_init_output" | jq -r .unseal_keys_b64[])"
VAULT_TOKEN="$(echo "$vault_init_output" | jq -r .root_token)"
export VAULT_TOKEN
juju run-action --wait vault/leader authorize-charm \
token="$(vault token create -ttl=10m -format json | jq -r .auth.client_token)"
juju run-action vault/leader --wait generate-root-ca
==== |
maas: 1:3.3.1-13169-g.94920eb1e-0ubuntu1~22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840
There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>
> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname().
https://github.com/juju/charm-helpers/blob/6e302bab63e22e356d77d76a5c6d90d9d24c6390/charmhelpers/contrib/network/ip.py#L497
In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file.
https://github.com/juju/charm-helpers/blob/b5725ac546372e7d4004d15095f79cdd5e7da687/charmhelpers/contrib/openstack/cert_utils.py#L105
[requested and written cert]
/etc/apache2/ssl/keystone/cert_eth0.juju-043209-2-lxd-0.maas (w/ eth0.)
-> exists
[the patch trying to create a symlink to]
/etc/apache2/ssl/keystone/cert_juju-043209-2-lxd-0.maas (w/o eth0.)
-> does not exist
unit-keystone-2: 12:45:41 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'eth0.juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': '{"eth0.juju-043209-2-lxd-0.maas": {"sans": ["192.168.151.131", "192.168.151.99"]}}', 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': {'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}}, 'unit_name': 'keystone_2'}
This is due to how MAAS DNS works.
$ dig +short @192.168.151.1 -x 192.168.151.131
eth0.juju-043209-2-lxd-0.maas.
juju-043209-2-lxd-0.maas.
$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/
/var/lib/bind/maas/zone.maas-$ORIGIN maas.
/var/lib/bind/maas/zone.maas:juju-043209-2-lxd-0 A 192.168.151.131
/var/lib/bind/maas/zone.maas:$ORIGIN juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.maas-eth0 A 192.168.151.131
--
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-129 PTR juju-fe03b8-2-lxd-8.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa:131 PTR eth0.juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa: PTR juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-134 PTR eth0.juju-043209-0-lxd-0.maas.
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
How to reproduce:
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status |
|
| 2023-03-26 02:57:26 |
Nobuto Murata |
bug task added |
|
charm-helpers |
|
| 2023-03-26 02:59:34 |
Nobuto Murata |
attachment added |
|
juju-crashdump-cc0a9308-2db7-4a3d-9acd-7c0cdfd68952.tar.xz https://bugs.launchpad.net/charm-helpers/+bug/2012801/+attachment/5657439/+files/juju-crashdump-cc0a9308-2db7-4a3d-9acd-7c0cdfd68952.tar.xz |
|
| 2023-03-26 03:05:42 |
Nobuto Murata |
bug |
|
|
added subscriber Canonical Field High |
| 2023-04-15 13:51:15 |
Nobuto Murata |
bug task added |
|
maas |
|
| 2023-04-15 13:52:04 |
Nobuto Murata |
summary |
Services not running that should be: apache2, SSLCertificateFile: file '/etc/apache2/ssl/*/cert_* does not exist or is empty |
MAAS rDNS returns two hostnames that lead to Services not running that should be: apache2, SSLCertificateFile: file '/etc/apache2/ssl/*/cert_* does not exist or is empty |
|
| 2023-04-15 13:57:24 |
Nobuto Murata |
description |
maas: 1:3.3.1-13169-g.94920eb1e-0ubuntu1~22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840
There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>
> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname().
https://github.com/juju/charm-helpers/blob/6e302bab63e22e356d77d76a5c6d90d9d24c6390/charmhelpers/contrib/network/ip.py#L497
In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file.
https://github.com/juju/charm-helpers/blob/b5725ac546372e7d4004d15095f79cdd5e7da687/charmhelpers/contrib/openstack/cert_utils.py#L105
[requested and written cert]
/etc/apache2/ssl/keystone/cert_eth0.juju-043209-2-lxd-0.maas (w/ eth0.)
-> exists
[the patch trying to create a symlink to]
/etc/apache2/ssl/keystone/cert_juju-043209-2-lxd-0.maas (w/o eth0.)
-> does not exist
unit-keystone-2: 12:45:41 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'eth0.juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': '{"eth0.juju-043209-2-lxd-0.maas": {"sans": ["192.168.151.131", "192.168.151.99"]}}', 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': {'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}}, 'unit_name': 'keystone_2'}
This is due to how MAAS DNS works.
$ dig +short @192.168.151.1 -x 192.168.151.131
eth0.juju-043209-2-lxd-0.maas.
juju-043209-2-lxd-0.maas.
$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/
/var/lib/bind/maas/zone.maas-$ORIGIN maas.
/var/lib/bind/maas/zone.maas:juju-043209-2-lxd-0 A 192.168.151.131
/var/lib/bind/maas/zone.maas:$ORIGIN juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.maas-eth0 A 192.168.151.131
--
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-129 PTR juju-fe03b8-2-lxd-8.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa:131 PTR eth0.juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa: PTR juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-134 PTR eth0.juju-043209-0-lxd-0.maas.
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
How to reproduce:
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status |
maas: 1:3.3.2-13177-g.a73a6e2bd-0ubuntu1~22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840
MAAS 3.3.2 (and 3.3.1 at least) returns PTR records in an inconsistent way for LXD containers created by MAAS. Those LXD interfaces are managed as "devices" in MAAAS.
This behavior doesn't happen with 3.2.7.
$ juju run --all 'hostname -I'
- MachineId: "0"
Stdout: "10.206.50.1 192.168.151.120 \n"
- MachineId: 0/lxd/0
Stdout: "192.168.151.121 \n"
- MachineId: 0/lxd/1
Stdout: "192.168.151.123 \n"
- MachineId: 0/lxd/2
Stdout: "192.168.151.122 \n"
$ juju run --all 'dig +short -x $(hostname -I)'
- MachineId: "0"
Stdout: |
large-wolf.
large-wolf.local.
- MachineId: 0/lxd/0
Stdout: |
juju-f8d90a-0-lxd-0.maas.
- MachineId: 0/lxd/1
Stdout: |
juju-f8d90a-0-lxd-1.maas.
eth0.juju-f8d90a-0-lxd-1.maas.
- MachineId: 0/lxd/2
Stdout: |
juju-f8d90a-0-lxd-2.maas.
eth0.juju-f8d90a-0-lxd-2.maas.
^^^ MAAS DNS returns two PTR records with and without "eth0" for one IP address for some LXD containers (not all).
How to reproduce:
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
https://launchpadlibrarian.net/657634345/keystone-ha_vault_edge.yaml
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status
[original description]
There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>
> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname().
https://github.com/juju/charm-helpers/blob/6e302bab63e22e356d77d76a5c6d90d9d24c6390/charmhelpers/contrib/network/ip.py#L497
In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file.
https://github.com/juju/charm-helpers/blob/b5725ac546372e7d4004d15095f79cdd5e7da687/charmhelpers/contrib/openstack/cert_utils.py#L105
[requested and written cert]
/etc/apache2/ssl/keystone/cert_eth0.juju-043209-2-lxd-0.maas (w/ eth0.)
-> exists
[the patch trying to create a symlink to]
/etc/apache2/ssl/keystone/cert_juju-043209-2-lxd-0.maas (w/o eth0.)
-> does not exist
unit-keystone-2: 12:45:41 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'eth0.juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': '{"eth0.juju-043209-2-lxd-0.maas": {"sans": ["192.168.151.131", "192.168.151.99"]}}', 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': {'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}}, 'unit_name': 'keystone_2'}
This is due to how MAAS DNS works.
$ dig +short @192.168.151.1 -x 192.168.151.131
eth0.juju-043209-2-lxd-0.maas.
juju-043209-2-lxd-0.maas.
$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/
/var/lib/bind/maas/zone.maas-$ORIGIN maas.
/var/lib/bind/maas/zone.maas:juju-043209-2-lxd-0 A 192.168.151.131
/var/lib/bind/maas/zone.maas:$ORIGIN juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.maas-eth0 A 192.168.151.131
--
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-129 PTR juju-fe03b8-2-lxd-8.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa:131 PTR eth0.juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa: PTR juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-134 PTR eth0.juju-043209-0-lxd-0.maas.
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.' |
|
| 2023-04-15 13:58:49 |
Nobuto Murata |
charm-helpers: status |
New |
Invalid |
|
| 2023-04-15 13:58:53 |
Nobuto Murata |
charm-keystone: status |
New |
Invalid |
|
| 2023-04-15 14:00:04 |
Nobuto Murata |
description |
maas: 1:3.3.2-13177-g.a73a6e2bd-0ubuntu1~22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840
MAAS 3.3.2 (and 3.3.1 at least) returns PTR records in an inconsistent way for LXD containers created by MAAS. Those LXD interfaces are managed as "devices" in MAAAS.
This behavior doesn't happen with 3.2.7.
$ juju run --all 'hostname -I'
- MachineId: "0"
Stdout: "10.206.50.1 192.168.151.120 \n"
- MachineId: 0/lxd/0
Stdout: "192.168.151.121 \n"
- MachineId: 0/lxd/1
Stdout: "192.168.151.123 \n"
- MachineId: 0/lxd/2
Stdout: "192.168.151.122 \n"
$ juju run --all 'dig +short -x $(hostname -I)'
- MachineId: "0"
Stdout: |
large-wolf.
large-wolf.local.
- MachineId: 0/lxd/0
Stdout: |
juju-f8d90a-0-lxd-0.maas.
- MachineId: 0/lxd/1
Stdout: |
juju-f8d90a-0-lxd-1.maas.
eth0.juju-f8d90a-0-lxd-1.maas.
- MachineId: 0/lxd/2
Stdout: |
juju-f8d90a-0-lxd-2.maas.
eth0.juju-f8d90a-0-lxd-2.maas.
^^^ MAAS DNS returns two PTR records with and without "eth0" for one IP address for some LXD containers (not all).
How to reproduce:
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
https://launchpadlibrarian.net/657634345/keystone-ha_vault_edge.yaml
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status
[original description]
There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>
> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname().
https://github.com/juju/charm-helpers/blob/6e302bab63e22e356d77d76a5c6d90d9d24c6390/charmhelpers/contrib/network/ip.py#L497
In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file.
https://github.com/juju/charm-helpers/blob/b5725ac546372e7d4004d15095f79cdd5e7da687/charmhelpers/contrib/openstack/cert_utils.py#L105
[requested and written cert]
/etc/apache2/ssl/keystone/cert_eth0.juju-043209-2-lxd-0.maas (w/ eth0.)
-> exists
[the patch trying to create a symlink to]
/etc/apache2/ssl/keystone/cert_juju-043209-2-lxd-0.maas (w/o eth0.)
-> does not exist
unit-keystone-2: 12:45:41 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'eth0.juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': '{"eth0.juju-043209-2-lxd-0.maas": {"sans": ["192.168.151.131", "192.168.151.99"]}}', 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': {'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}}, 'unit_name': 'keystone_2'}
This is due to how MAAS DNS works.
$ dig +short @192.168.151.1 -x 192.168.151.131
eth0.juju-043209-2-lxd-0.maas.
juju-043209-2-lxd-0.maas.
$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/
/var/lib/bind/maas/zone.maas-$ORIGIN maas.
/var/lib/bind/maas/zone.maas:juju-043209-2-lxd-0 A 192.168.151.131
/var/lib/bind/maas/zone.maas:$ORIGIN juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.maas-eth0 A 192.168.151.131
--
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-129 PTR juju-fe03b8-2-lxd-8.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa:131 PTR eth0.juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa: PTR juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-134 PTR eth0.juju-043209-0-lxd-0.maas.
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.' |
maas: 1:3.3.2-13177-g.a73a6e2bd-0ubuntu1~22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840
MAAS 3.3.2 (and 3.3.1 at least) returns PTR records in an inconsistent way for LXD containers created by Juju. Those LXD interfaces are managed as "devices" in MAAAS.
This behavior doesn't happen with 3.2.7.
$ juju run --all 'hostname -I'
- MachineId: "0"
Stdout: "10.206.50.1 192.168.151.120 \n"
- MachineId: 0/lxd/0
Stdout: "192.168.151.121 \n"
- MachineId: 0/lxd/1
Stdout: "192.168.151.123 \n"
- MachineId: 0/lxd/2
Stdout: "192.168.151.122 \n"
$ juju run --all 'dig +short -x $(hostname -I)'
- MachineId: "0"
Stdout: |
large-wolf.
large-wolf.local.
- MachineId: 0/lxd/0
Stdout: |
juju-f8d90a-0-lxd-0.maas.
- MachineId: 0/lxd/1
Stdout: |
juju-f8d90a-0-lxd-1.maas.
eth0.juju-f8d90a-0-lxd-1.maas.
- MachineId: 0/lxd/2
Stdout: |
juju-f8d90a-0-lxd-2.maas.
eth0.juju-f8d90a-0-lxd-2.maas.
^^^ MAAS DNS returns two PTR records with and without "eth0" for one IP address for some LXD containers (not all).
How to reproduce:
1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
https://launchpadlibrarian.net/657634345/keystone-ha_vault_edge.yaml
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status
[original description]
There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>
> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.
Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname().
https://github.com/juju/charm-helpers/blob/6e302bab63e22e356d77d76a5c6d90d9d24c6390/charmhelpers/contrib/network/ip.py#L497
In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file.
https://github.com/juju/charm-helpers/blob/b5725ac546372e7d4004d15095f79cdd5e7da687/charmhelpers/contrib/openstack/cert_utils.py#L105
[requested and written cert]
/etc/apache2/ssl/keystone/cert_eth0.juju-043209-2-lxd-0.maas (w/ eth0.)
-> exists
[the patch trying to create a symlink to]
/etc/apache2/ssl/keystone/cert_juju-043209-2-lxd-0.maas (w/o eth0.)
-> does not exist
unit-keystone-2: 12:45:41 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'eth0.juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': '{"eth0.juju-043209-2-lxd-0.maas": {"sans": ["192.168.151.131", "192.168.151.99"]}}', 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': {'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}}, 'unit_name': 'keystone_2'}
This is due to how MAAS DNS works.
$ dig +short @192.168.151.1 -x 192.168.151.131
eth0.juju-043209-2-lxd-0.maas.
juju-043209-2-lxd-0.maas.
$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/
/var/lib/bind/maas/zone.maas-$ORIGIN maas.
/var/lib/bind/maas/zone.maas:juju-043209-2-lxd-0 A 192.168.151.131
/var/lib/bind/maas/zone.maas:$ORIGIN juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.maas-eth0 A 192.168.151.131
--
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-129 PTR juju-fe03b8-2-lxd-8.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa:131 PTR eth0.juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa: PTR juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-134 PTR eth0.juju-043209-0-lxd-0.maas.
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.' |
|
| 2023-04-17 00:46:44 |
DUFOUR Olivier |
bug |
|
|
added subscriber DUFOUR Olivier |
| 2023-04-20 19:49:29 |
Alexsander de Souza |
maas: status |
New |
Triaged |
|
| 2023-04-20 19:49:38 |
Alexsander de Souza |
maas: milestone |
|
3.4.0 |
|
| 2023-04-20 19:49:43 |
Alexsander de Souza |
maas: importance |
Undecided |
Critical |
|
| 2023-04-21 09:21:02 |
Adam Collard |
charm-keystone: status |
Invalid |
New |
|
| 2023-04-21 09:21:18 |
Adam Collard |
charm-helpers: status |
Invalid |
New |
|
| 2023-05-11 08:34:22 |
Jerzy Husakowski |
maas: assignee |
|
Christian Grabowski (cgrabowski) |
|
| 2023-05-22 04:22:30 |
Nobuto Murata |
attachment added |
|
sosreport-quick-maas-2023-05-22-xowucij.tar.xz https://bugs.launchpad.net/maas/+bug/2012801/+attachment/5674554/+files/sosreport-quick-maas-2023-05-22-xowucij.tar.xz |
|
| 2023-05-22 04:42:56 |
Nobuto Murata |
attachment added |
|
maas_devices_read.json https://bugs.launchpad.net/maas/+bug/2012801/+attachment/5674568/+files/maas_devices_read.json |
|
| 2023-06-02 18:14:31 |
Christian Grabowski |
maas: status |
Triaged |
New |
|
| 2023-06-02 18:14:35 |
Christian Grabowski |
maas: milestone |
3.4.0 |
|
|
| 2023-06-07 08:59:52 |
Adam Collard |
maas: status |
New |
In Progress |
|
| 2023-06-07 08:59:57 |
Adam Collard |
maas: milestone |
|
3.4.0 |
|
| 2023-06-09 03:33:47 |
Christian Grabowski |
merge proposal linked |
|
https://code.launchpad.net/~cgrabowski/maas/+git/maas/+merge/444471 |
|
| 2023-06-09 08:58:30 |
MAAS Lander |
maas: status |
In Progress |
Fix Committed |
|
| 2023-06-12 10:09:15 |
Jerzy Husakowski |
maas: status |
Fix Committed |
In Progress |
|
| 2023-06-15 16:46:34 |
Jeffrey Chang |
bug |
|
|
added subscriber Jeffrey Chang |
| 2023-06-20 14:20:58 |
Thorsten Merten |
maas: status |
In Progress |
Fix Committed |
|
| 2023-06-22 05:00:04 |
Nobuto Murata |
maas: status |
Fix Committed |
Confirmed |
|
| 2023-06-22 08:13:46 |
Adam Collard |
maas: status |
Confirmed |
Triaged |
|
| 2023-06-26 16:46:31 |
Christian Grabowski |
maas: status |
Triaged |
In Progress |
|
| 2023-06-28 10:11:30 |
Alberto Donato |
merge proposal linked |
|
https://code.launchpad.net/~cgrabowski/maas/+git/maas/+merge/445513 |
|
| 2023-06-28 13:39:23 |
Alberto Donato |
nominated for series |
|
maas/3.4 |
|
| 2023-06-28 13:39:23 |
Alberto Donato |
bug task added |
|
maas/3.4 |
|
| 2023-06-28 13:39:23 |
Alberto Donato |
nominated for series |
|
maas/3.5 |
|
| 2023-06-28 13:39:23 |
Alberto Donato |
bug task added |
|
maas/3.5 |
|
| 2023-06-28 13:39:34 |
Alberto Donato |
maas/3.5: milestone |
3.4.0 |
3.5.0 |
|
| 2023-06-28 13:39:37 |
Alberto Donato |
maas/3.5: importance |
Critical |
High |
|
| 2023-06-28 13:39:40 |
Alberto Donato |
maas/3.4: status |
New |
In Progress |
|
| 2023-06-28 13:39:42 |
Alberto Donato |
maas/3.4: importance |
Undecided |
High |
|
| 2023-06-28 13:39:48 |
Alberto Donato |
maas/3.4: assignee |
|
Christian Grabowski (cgrabowski) |
|
| 2023-06-28 13:39:52 |
Alberto Donato |
maas/3.4: milestone |
|
3.4.0 |
|
| 2023-06-28 13:39:56 |
Alberto Donato |
maas/3.5: status |
In Progress |
New |
|
| 2023-06-28 13:39:59 |
Alberto Donato |
maas/3.5: status |
New |
In Progress |
|
| 2023-06-28 15:58:26 |
MAAS Lander |
maas/3.5: status |
In Progress |
Fix Committed |
|
| 2023-06-29 07:07:54 |
Alberto Donato |
merge proposal linked |
|
https://code.launchpad.net/~ack/maas/+git/maas/+merge/445605 |
|
| 2023-06-29 08:20:40 |
MAAS Lander |
maas/3.4: status |
In Progress |
Fix Committed |
|
| 2023-07-03 10:49:29 |
Alberto Donato |
nominated for series |
|
maas/3.3 |
|
| 2023-07-03 10:49:29 |
Alberto Donato |
bug task added |
|
maas/3.3 |
|
| 2023-07-03 10:49:34 |
Alberto Donato |
maas/3.3: status |
New |
Triaged |
|
| 2023-07-03 10:49:37 |
Alberto Donato |
maas/3.3: importance |
Undecided |
High |
|
| 2023-07-03 10:52:20 |
Alberto Donato |
maas/3.3: assignee |
|
Christian Grabowski (cgrabowski) |
|
| 2023-07-03 10:52:25 |
Alberto Donato |
maas/3.3: milestone |
|
3.3.x |
|
| 2023-07-03 15:22:20 |
Christian Grabowski |
merge proposal linked |
|
https://code.launchpad.net/~cgrabowski/maas/+git/maas/+merge/445877 |
|
| 2023-07-03 15:58:24 |
MAAS Lander |
maas/3.3: status |
Triaged |
Fix Committed |
|
| 2023-08-04 13:14:56 |
Felipe Reyes |
charm-keystone: status |
New |
Invalid |
|
| 2023-08-04 13:14:59 |
Felipe Reyes |
charm-helpers: status |
New |
Invalid |
|
| 2023-08-07 14:18:55 |
Alex Kavanagh |
charm-helpers: importance |
Undecided |
High |
|
| 2023-08-07 14:18:55 |
Alex Kavanagh |
charm-helpers: status |
Invalid |
Triaged |
|
| 2023-08-15 12:42:49 |
Adam Collard |
bug task added |
|
charm-magpie |
|
| 2023-08-15 12:44:00 |
Adam Collard |
charm-magpie: status |
New |
Incomplete |
|
| 2023-08-15 12:44:03 |
Adam Collard |
charm-magpie: status |
Incomplete |
In Progress |
|
| 2023-08-23 15:24:55 |
Trent Lloyd |
bug |
|
|
added subscriber Trent Lloyd |
| 2023-09-11 09:20:26 |
Adam Collard |
charm-magpie: status |
In Progress |
Fix Committed |
|
| 2023-09-11 09:20:30 |
Adam Collard |
charm-magpie: assignee |
|
Adam Collard (adam-collard) |
|
| 2023-09-27 20:15:27 |
Alexsander de Souza |
maas/3.4: status |
Fix Committed |
Fix Released |
|
