ceph dashboard fails to get a certificate with tls-e

Bug #1926746 reported by Michele Baldessari
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Medium
Unassigned

Bug Description

So I wanted to add ceph (via cephadm) in my spine/leaf tls-e environment. It all worked without ceph, but as soon as I add ceph it fails on all controllers with (Note that I patched heat on the UC with the fixes for https://bugs.launchpad.net/tripleo/+bug/1925373):
021-04-30 07:47:10.308998 | 525400d9-461e-459c-3980-00000000e4ac | FATAL | Ensure certificate requests | ctrl-3-0 | item={'ca': 'ipa', 'dns': 'ctrl-3-0.mainnetwork.bgp.ftw', 'key_size': '2048', 'name': 'ceph_dashboard', 'principal': '<email address hidden>', 'run_after': '# Get mgr systemd unit\nmgr_unit=$(systemctl list-units | awk \'/ceph-mgr/ {print $1}\')\n# Restart the mgr systemd unit\nif [ -n "$mgr_unit" ]; then\n systemctl restart "$mgr_unit"\nfi\n'} | error={"ansible_loop_var": "item", "changed": false, "cmd": "/bin/getcert request -N CN=ctrl-3-0.mainnetwork.bgp.ftw -c IPA -w -k /etc/pki/tls/private/ceph_dashboard.key -f /etc/pki/tls/certs/ceph_dashboard.crt -D ctrl-3-0.mainnetwork.bgp.ftw -D '' -A '' -E '' -r -g 2048 -K '' -K '' -u keyEncipherment -u digitalSignature -U 1.3.6.1.5.5.7.3.1 -U 1.3.6.1.5.5.7.3.2 -U '' -B '' -C /etc/certmonger/post-scripts/ceph_dashboard-838da8a.sh", "item": {"ca": "ipa", "dns": "ctrl-3-0.mainnetwork.bgp.ftw", "key_size": "2048", "name": "ceph_dashboard", "principal": "<email address hidden>", "run_after": "# Get mgr systemd unit\nmgr_unit=$(systemctl list-units | awk '/ceph-mgr/ {print $1}')\n# Restart the mgr systemd unit\nif [ -n \"$mgr_unit\" ]; then\n systemctl restart \"$mgr_unit\"\nfi\n"}, "msg": "", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "New signing request \"20210430074709\" added.\n", "stdout_lines": ["New signing request \"20210430074709\" added."]}

If I hop on the failing node ctrl-1-0 I indeed see the request being rejected by freeipa:
Apr 29 18:36:34 ctrl-1-0.bgp.ftw certmonger[38657]: 2021-04-29 18:36:34 [39400] Running enrollment helper "/usr/libexec/certmonger/ipa-submit".
Apr 29 18:36:34 ctrl-1-0.bgp.ftw certmonger[38657]: Submitting request to "https://freeipa-0.bgp.ftw/ipa/json".
Apr 29 18:36:34 ctrl-1-0.bgp.ftw ipa-submit[39400]: JSON-RPC error: 2100: Insufficient access: Insufficient 'add' privilege to add the entry '<email address hidden>,cn=services,cn=accounts,dc=bgp,dc=ftw'.

On freeipa I see:
httpd/error_log:[Thu Apr 29 18:36:33.250445 2021] [wsgi:error] [pid 31608:tid 140062130665216] [remote 172.30.3.1:39142] ipa: INFO: [jsonserver_kerb] host/ctrl-3-0.bgp.ft
<email address hidden>: cert_request('MIID3DCCAsQCAQAwJzElMCMGA1UEAxMcY3RybC0zLTAubWFpbm5ldHdvcmsuYmdwLmZ0dzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMuRHXy71Dtxesg24trSlNe92k9k3CYTo
9n/VOFZgDCtXIr1JD4qAM9I+3j4EBHixPW0wUF2qxthk9oCJX4c0JqPvPWCylzaL37FyovLtYOAm0IABou65GajLlO9IsJI9GfCJZ69aM/1q0n1N1X/wWO3GwM1ngJJI9OEpOvwzA7si8Y+1R/UBRp+Jgcx9HZDux2Usv2bfO1
x33Zqy/OEHMbN5oCx/rhgX3zfZ+B8NkVlbUh7dAV75R0kWSqhhdnWPcrErxgmiOHpBvT9pZMoKzAMKlnpml2Y7Sz0JCXu7hsbVdEUbVNFiIInwQ5SclYqNWaHSyc6XfzHPvmRRu6esv0CAwEAAaCCAW4wKwYJKoZIhvcNAQkUM
R4eHAAyADAAMgAxADAANAAyADkAMQA4ADMANgAzADIwggE9BgkqhkiG9w0BCQ4xggEuMIIBKjAOBgNVHQ8BAQAEBAMCBaAwgcUGA1UdEQEBAASBujCBt4IcY3RybC0zLTAubWFpbm5ldHdvcmsuYmdwLmZ0d6BDBgorBgEEAYI
3FAIDoDUMM2NlcGhfZGFzaGJvYXJkL2N0cmwtMy0wLm1haW5uZXR3b3JrLmJncC5mdHdAQkdQLkZUV6BSBgYrBgEFAgKgSDBGoAkbB0JHUC5GVFehOTA3oAMCAQGhMDAuGw5jZXBoX2Rhc2hib2FyZBscY3RybC0zLTAubWFpb
m5ldHdvcmsuYmdwLmZ0dzAgBgNVHSUBAQAEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAgBgNVHQ4BAQAEFgQU0udMTy6T3f4+Bl0GNAK3e90GE+YwDQYJKoZIhvcNAQELBQADggEBACmKUWuXeFvd/ug
PdBhMIDGzrjpWfmh6YKVKgfWiLKSIW3BolfK/+t8ZUQyuHNFbJEh4T9te65lzc990CcYAl39tuLD5JFK8xdvMUAPOPcbPE9f9tctXLlp5OInqUzNVE/X1klS2tTZ83ieMUgsp7YrJCNO6gqUTyT+IcS+D54WlronTedKrGN9n6
/Zpysop6yi+fHX/60hTsM2uu7qIFuIihxel2ST4jVOXux14Sp4D+x38v8NHAmc1DBEsSxPYVKucVe8y9yCqd0mOLVOWmtMHgi4ChPar2ehxvD3Y5JQVPjo71YtkkVL9IC3FG8SIoi8eQCIsPodOwTITox2Wf9I=', principa
<email address hidden>', add=True): ACIError

Whereas a working cert request (from a compute for example) succeeds without issues:
[Thu Apr 29 18:36:38.101465 2021] [wsgi:error] [pid 31601:tid 140062130665216] [remote 172.30.2.2:58244] ipa: INFO: [jsonserver_kerb] <email address hidden>: cert_r
equest('MIIDyjCCArICAQAwJjEkMCIGA1UEAxMbY21wLTItMC5tYWlubmV0d29yay5iZ3AuZnR3MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAr0yKzGSE+26igS1tNGr7l5u2I3owQdJspq+/cvX9tb5pS06wku
cGixHYS9dvnFrAxc/bLoi8VoUNLLGOGMVjlXu/OaTeLiKeWlG525o6BGm15JzO87QtfQCvOBkH/bVMj6+XFKFp4DLvKSyxT4ObrjoVLo6g/vbtw8it3dCoclA/FRv1/Lc96JlXeZ/cKmr+o1kZJscXvQV7a3u829olsAMoTc1j
DeqPmzTJUXwqIoo7BlLgHMxHALI5mV/EYhTyZWgMCDNCij/vmw4OwVj03bedsbT+UitcrRqNiBCY3m0tq4Ip67OeZq6h9C1XKIg3v1ju7uy/Y9bgLUFq7vRaJwIDAQABoIIBXTArBgkqhkiG9w0BCRQxHh4cADIAMAAyADEAMA
A0ADIAOQAxADgAMwA2ADMANzCCASwGCSqGSIb3DQEJDjGCAR0wggEZMA4GA1UdDwEBAAQEAwIFoDCBtAYDVR0RAQEABIGpMIGmghtjbXAtMi0wLm1haW5uZXR3b3JrLmJncC5mdHegOwYKKwYBBAGCNxQCA6AtDCtsaWJ2aXJ0
L2NtcC0yLTAubWFpbm5ldHdvcmsuYmdwLmZ0d0BCR1AuRlRXoEoGBisGAQUCAqBAMD6gCRsHQkdQLkZUV6ExMC+gAwIBAaEoMCYbB2xpYnZpcnQbG2NtcC0yLTAubWFpbm5ldHdvcmsuYmdwLmZ0dzAgBgNVHSUBAQAEFjAUBg
grBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAgBgNVHQ4BAQAEFgQUEbHrlQMdy2teYcIsuwONqgrrIOMwDQYJKoZIhvcNAQELBQADggEBAGigc6mWQ+272buGsFRMbsQjPm2Nq/g76tDbjTDMGDRPC5VK5JLJ30i9
pcVY/F1vJj61tmaw2GgdbaDLmHVqe3E6bfUENbjcsYDT9coc6HQB63/20nGO0Y52FycTdZM1LQUQNQeJA6iR68MixlQhZ1NWpKL5ISL0wHsCMSmVRHxLYTQND+hQOmAYGNCj4ZnSQGJffUo1VNm8X1D7GSAMqPqd9YUd/FAWZt
8znvBBwvtlcNiHmfCkrucnxdijxOWlt9VrtCWPTkl/XRip/BteIWhitCjeb10lzL4+GRpaWOc3sCYBz9sHm3eLZNVBw0VpVio2saD2LNt+neKAwrCnqW0=', principal='libvirt/cmp-2-0.mainnetwork.bgp.ftw@BG
P.FTW', add=True): SUCCESS

Note that without ceph I deploy this tls-e environment just fine, so something is messing up things when ceph is involved (and I did patch heat for the list_unique LP).

sosreports of undercloud, freeipa, ctrl-1-0 (broken), cmp-1-0 (working) are here:
https://acksyn.org/files/tripleo/ceph-master-tlse/

tags: removed: sec
Revision history for this message
Francesco Pantano (fmount) wrote :

Thanks @Michele for the info provideded.
Just a few preliminary considerations: the Ceph bits for the cert
request in TripleO are [1] [2] [3] but looks like they're the pretty
much the same code for the other OSP components.
The certificates are generated per-network using the service_net_map,
I'm wondering if we need to look for more clues on the ipa side.

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-mgr.yaml

[2] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-rgw.yaml

[3] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-grafana.yaml

Revision history for this message
Francesco Pantano (fmount) wrote :

In addition I see:

Apr 29 18:36:34 ctrl-1-0.bgp.ftw ipa-submit[39400]: JSON-RPC error: 2100: Insufficient access: Insufficient 'add' privilege to add the entry '<email address hidden>,cn=services,cn=accounts,dc=bgp,dc=ftw'.

and I'm wondering if something is missing on the ipa side (maybe related to the ceph_dashboard name used to submit the request while the service name specified in tht is ceph_mgr?

Revision history for this message
Michele Baldessari (michele) wrote :

Also note that the failing deployment we care about start at 2021-04-30 07:18:56.771. Any error messages before that we likely do not care about

Revision history for this message
Michele Baldessari (michele) wrote :

Note: I tried with replacing ceph_mgr with ceph_dashboard in the ceph-mgr.yaml file to no avail. Same issue

Revision history for this message
Michele Baldessari (michele) wrote :

So it fails even with cephenableddashboard set to false. That is a separate bug to be tracked in https://bugs.launchpad.net/tripleo/+bug/1927093

Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Changed in tripleo:
milestone: xena-1 → xena-2
Changed in tripleo:
milestone: xena-2 → xena-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.