So I wanted to add ceph (via cephadm) in my spine/leaf tls-e environment. It all worked without ceph, but as soon as I add ceph it fails on all controllers with (Note that I patched heat on the UC with the fixes for https://bugs.launchpad.net/tripleo/+bug/1925373):
021-04-30 07:47:10.308998 | 525400d9-461e-459c-3980-00000000e4ac | FATAL | Ensure certificate requests | ctrl-3-0 | item={'ca': 'ipa', 'dns': 'ctrl-3-0.mainnetwork.bgp.ftw', 'key_size': '2048', 'name': 'ceph_dashboard', 'principal': '<email address hidden>', 'run_after': '# Get mgr systemd unit\nmgr_unit=$(systemctl list-units | awk \'/ceph-mgr/ {print $1}\')\n# Restart the mgr systemd unit\nif [ -n "$mgr_unit" ]; then\n systemctl restart "$mgr_unit"\nfi\n'} | error={"ansible_loop_var": "item", "changed": false, "cmd": "/bin/getcert request -N CN=ctrl-3-0.mainnetwork.bgp.ftw -c IPA -w -k /etc/pki/tls/private/ceph_dashboard.key -f /etc/pki/tls/certs/ceph_dashboard.crt -D ctrl-3-0.mainnetwork.bgp.ftw -D '' -A '' -E '' -r -g 2048 -K '' -K '' -u keyEncipherment -u digitalSignature -U 1.3.6.1.5.5.7.3.1 -U 1.3.6.1.5.5.7.3.2 -U '' -B '' -C /etc/certmonger/post-scripts/ceph_dashboard-838da8a.sh", "item": {"ca": "ipa", "dns": "ctrl-3-0.mainnetwork.bgp.ftw", "key_size": "2048", "name": "ceph_dashboard", "principal": "<email address hidden>", "run_after": "# Get mgr systemd unit\nmgr_unit=$(systemctl list-units | awk '/ceph-mgr/ {print $1}')\n# Restart the mgr systemd unit\nif [ -n \"$mgr_unit\" ]; then\n systemctl restart \"$mgr_unit\"\nfi\n"}, "msg": "", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "New signing request \"20210430074709\" added.\n", "stdout_lines": ["New signing request \"20210430074709\" added."]}
If I hop on the failing node ctrl-1-0 I indeed see the request being rejected by freeipa:
Apr 29 18:36:34 ctrl-1-0.bgp.ftw certmonger[38657]: 2021-04-29 18:36:34 [39400] Running enrollment helper "/usr/libexec/certmonger/ipa-submit".
Apr 29 18:36:34 ctrl-1-0.bgp.ftw certmonger[38657]: Submitting request to "https://freeipa-0.bgp.ftw/ipa/json".
Apr 29 18:36:34 ctrl-1-0.bgp.ftw ipa-submit[39400]: JSON-RPC error: 2100: Insufficient access: Insufficient 'add' privilege to add the entry '<email address hidden>,cn=services,cn=accounts,dc=bgp,dc=ftw'.
On freeipa I see:
httpd/error_log:[Thu Apr 29 18:36:33.250445 2021] [wsgi:error] [pid 31608:tid 140062130665216] [remote 172.30.3.1:39142] ipa: INFO: [jsonserver_kerb] host/ctrl-3-0.bgp.ft
<email address hidden>: cert_request('MIID3DCCAsQCAQAwJzElMCMGA1UEAxMcY3RybC0zLTAubWFpbm5ldHdvcmsuYmdwLmZ0dzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMuRHXy71Dtxesg24trSlNe92k9k3CYTo
9n/VOFZgDCtXIr1JD4qAM9I+3j4EBHixPW0wUF2qxthk9oCJX4c0JqPvPWCylzaL37FyovLtYOAm0IABou65GajLlO9IsJI9GfCJZ69aM/1q0n1N1X/wWO3GwM1ngJJI9OEpOvwzA7si8Y+1R/UBRp+Jgcx9HZDux2Usv2bfO1
x33Zqy/OEHMbN5oCx/rhgX3zfZ+B8NkVlbUh7dAV75R0kWSqhhdnWPcrErxgmiOHpBvT9pZMoKzAMKlnpml2Y7Sz0JCXu7hsbVdEUbVNFiIInwQ5SclYqNWaHSyc6XfzHPvmRRu6esv0CAwEAAaCCAW4wKwYJKoZIhvcNAQkUM
R4eHAAyADAAMgAxADAANAAyADkAMQA4ADMANgAzADIwggE9BgkqhkiG9w0BCQ4xggEuMIIBKjAOBgNVHQ8BAQAEBAMCBaAwgcUGA1UdEQEBAASBujCBt4IcY3RybC0zLTAubWFpbm5ldHdvcmsuYmdwLmZ0d6BDBgorBgEEAYI
3FAIDoDUMM2NlcGhfZGFzaGJvYXJkL2N0cmwtMy0wLm1haW5uZXR3b3JrLmJncC5mdHdAQkdQLkZUV6BSBgYrBgEFAgKgSDBGoAkbB0JHUC5GVFehOTA3oAMCAQGhMDAuGw5jZXBoX2Rhc2hib2FyZBscY3RybC0zLTAubWFpb
m5ldHdvcmsuYmdwLmZ0dzAgBgNVHSUBAQAEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAgBgNVHQ4BAQAEFgQU0udMTy6T3f4+Bl0GNAK3e90GE+YwDQYJKoZIhvcNAQELBQADggEBACmKUWuXeFvd/ug
PdBhMIDGzrjpWfmh6YKVKgfWiLKSIW3BolfK/+t8ZUQyuHNFbJEh4T9te65lzc990CcYAl39tuLD5JFK8xdvMUAPOPcbPE9f9tctXLlp5OInqUzNVE/X1klS2tTZ83ieMUgsp7YrJCNO6gqUTyT+IcS+D54WlronTedKrGN9n6
/Zpysop6yi+fHX/60hTsM2uu7qIFuIihxel2ST4jVOXux14Sp4D+x38v8NHAmc1DBEsSxPYVKucVe8y9yCqd0mOLVOWmtMHgi4ChPar2ehxvD3Y5JQVPjo71YtkkVL9IC3FG8SIoi8eQCIsPodOwTITox2Wf9I=', principa
<email address hidden>', add=True): ACIError
Whereas a working cert request (from a compute for example) succeeds without issues:
[Thu Apr 29 18:36:38.101465 2021] [wsgi:error] [pid 31601:tid 140062130665216] [remote 172.30.2.2:58244] ipa: INFO: [jsonserver_kerb] <email address hidden>: cert_r
equest('MIIDyjCCArICAQAwJjEkMCIGA1UEAxMbY21wLTItMC5tYWlubmV0d29yay5iZ3AuZnR3MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAr0yKzGSE+26igS1tNGr7l5u2I3owQdJspq+/cvX9tb5pS06wku
cGixHYS9dvnFrAxc/bLoi8VoUNLLGOGMVjlXu/OaTeLiKeWlG525o6BGm15JzO87QtfQCvOBkH/bVMj6+XFKFp4DLvKSyxT4ObrjoVLo6g/vbtw8it3dCoclA/FRv1/Lc96JlXeZ/cKmr+o1kZJscXvQV7a3u829olsAMoTc1j
DeqPmzTJUXwqIoo7BlLgHMxHALI5mV/EYhTyZWgMCDNCij/vmw4OwVj03bedsbT+UitcrRqNiBCY3m0tq4Ip67OeZq6h9C1XKIg3v1ju7uy/Y9bgLUFq7vRaJwIDAQABoIIBXTArBgkqhkiG9w0BCRQxHh4cADIAMAAyADEAMA
A0ADIAOQAxADgAMwA2ADMANzCCASwGCSqGSIb3DQEJDjGCAR0wggEZMA4GA1UdDwEBAAQEAwIFoDCBtAYDVR0RAQEABIGpMIGmghtjbXAtMi0wLm1haW5uZXR3b3JrLmJncC5mdHegOwYKKwYBBAGCNxQCA6AtDCtsaWJ2aXJ0
L2NtcC0yLTAubWFpbm5ldHdvcmsuYmdwLmZ0d0BCR1AuRlRXoEoGBisGAQUCAqBAMD6gCRsHQkdQLkZUV6ExMC+gAwIBAaEoMCYbB2xpYnZpcnQbG2NtcC0yLTAubWFpbm5ldHdvcmsuYmdwLmZ0dzAgBgNVHSUBAQAEFjAUBg
grBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAgBgNVHQ4BAQAEFgQUEbHrlQMdy2teYcIsuwONqgrrIOMwDQYJKoZIhvcNAQELBQADggEBAGigc6mWQ+272buGsFRMbsQjPm2Nq/g76tDbjTDMGDRPC5VK5JLJ30i9
pcVY/F1vJj61tmaw2GgdbaDLmHVqe3E6bfUENbjcsYDT9coc6HQB63/20nGO0Y52FycTdZM1LQUQNQeJA6iR68MixlQhZ1NWpKL5ISL0wHsCMSmVRHxLYTQND+hQOmAYGNCj4ZnSQGJffUo1VNm8X1D7GSAMqPqd9YUd/FAWZt
8znvBBwvtlcNiHmfCkrucnxdijxOWlt9VrtCWPTkl/XRip/BteIWhitCjeb10lzL4+GRpaWOc3sCYBz9sHm3eLZNVBw0VpVio2saD2LNt+neKAwrCnqW0=', principal='libvirt/cmp-2-0.mainnetwork.bgp.ftw@BG
P.FTW', add=True): SUCCESS
Note that without ceph I deploy this tls-e environment just fine, so something is messing up things when ceph is involved (and I did patch heat for the list_unique LP).
sosreports of undercloud, freeipa, ctrl-1-0 (broken), cmp-1-0 (working) are here:
https://acksyn.org/files/tripleo/ceph-master-tlse/
Thanks @Michele for the info provideded.
Just a few preliminary considerations: the Ceph bits for the cert
request in TripleO are [1] [2] [3] but looks like they're the pretty
much the same code for the other OSP components.
The certificates are generated per-network using the service_net_map,
I'm wondering if we need to look for more clues on the ipa side.
[1] https:/ /github. com/openstack/ tripleo- heat-templates/ blob/master/ deployment/ cephadm/ ceph-mgr. yaml
[2] https:/ /github. com/openstack/ tripleo- heat-templates/ blob/master/ deployment/ cephadm/ ceph-rgw. yaml
[3] https:/ /github. com/openstack/ tripleo- heat-templates/ blob/master/ deployment/ cephadm/ ceph-grafana. yaml