OpenStack API Layer

The default AMQP handler does not handle TLS

Bug #1841912 reported by Frode Nordahl on 2019-08-29

This bug affects 1 person

	Status	Importance	Assigned to
OpenStack API Layer	Triaged	High	Unassigned
OpenStack Base Layer	Triaged	High	Unassigned
OpenStack Octavia Charm	Triaged	High	Unassigned
charms.openstack	Triaged	High	Unassigned

Bug Description

When the rabbitmq-server charm is configured with TLS it will announce this fact along with the CA certificate used over the relation.

The layer does not contain a handler for this and it should have one.

charms.openstack does contain code to handle the RabbitMQ cert that piggy backs on the handling of other TLS related events.

Without layer handling code it will not be called on changes to the relation and will then lead to situations where a charm using our stack will no longer be able to communicate with RabbitMQ when the certificate changes.

There is example in the ``neutron-dynamic-routing`` [0] charm for layer handling code for this, and this also raises the need for completing the move of the default AMQP handlers down to the ``openstack`` layer from the ``openstack-api`` layer.

0: https://review.opendev.org/#/q/topic:bug/1807233+(status:open+OR+status:merged)

Frode Nordahl (fnordahl) on 2019-08-29

Changed in charm-octavia:
status:	New → Triaged
importance:	Undecided → High

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2019-08-29:

Download full text (9.8 KiB)

While validating framework changes for bug 1840899 I caught this during a charm-octavia test run in a TLS enabled deployment with ``ssl=only`` configured for ``rabbitmq-server``.

From ``/var/log/octavia/octavia-worker.log``:
2019-08-29 09:53:40.572 2483 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852) (retrying in 32.0 seconds): ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

And surely enough the data represented in ``/var/lib/charm/octavia/rabbit-client-ca.pem`` differ from what is on the relation.

While validating framework changes for bug 1840899 I caught this during a charm-octavia test run in a TLS enabled deployment with ``ssl=only`` configured for ``rabbitmq-server``.

And surely enough the data represented in ``/var/lib/charm/octavia/rabbit-client-ca.pem`` differ from what is on the relation.

root@juju-77088e-zaza-fdbdfb4a2723-8:/var/lib/charm/octavia# cat /var/lib/charm/octavia/rabbit-client-ca.pem 
-----BEGIN CERTIFICATE-----
MIICdzCCAeCgAwIBAgIUAXtKq1MuovVzE6TsykohOdN0gigwDQYJKoZIhvcNAQEF
BQAwRzENMAsGA1UECgwEanVqdTE2MDQGA1UECwwtcmFiYml0bXEtc2VydmVyIHNl
cnZpY2UgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTE5MDgyOTA2MjYzOVoXDTI1
MDgyNzA2MjYzOVowRzENMAsGA1UECgwEanVqdTE2MDQGA1UECwwtcmFiYml0bXEt
c2VydmVyIHNlcnZpY2UgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQC0skntKENl4jhOw8TRMWVHMBsWaulwVr51sDGNAfIV
+n92mzZEVXBNZ34JvYmXck5woZLwOh8k10vnheVM3TB1UALW0itDkQSmmrGymLGH
HdwM29r8gKYYHiyL+sg/QQmQtXbb/LBEXzD3wDXXSJZKWpZs4ClRljUHwItQh/gi
PwIDAQABo2AwXjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRcRSzZ3Cgb/Cub
xmYfCPAGP0KfpDAfBgNVHSMEGDAWgBRcRSzZ3Cgb/CubxmYfCPAGP0KfpDALBgNV
HQ8EBAMCAQYwDQYJKoZIhvcNAQEFBQADgYEAGs/kT+XqC48IvoiZ5s9SwlT9tngK
ZHAdAUSjvOpjutJl5/RUCn/OU5gSDdvjf6nhxV34DCufIeIi1wi9FIY3xfXw1deM
5R2XZsjCifSPc8gq8hPKuvJ4Xf/WwO/eoIAwi3iEaoxlNStVw6P9kv1m8tawcOXe
VDH7zZdwPJ31dA4=
-----END CERTIFICATE-----
ubuntu@test:~$ juju run --unit octavia/0 'relation-ids amqp'
amqp:26
ubuntu@test:~$ juju run --unit octavia/0 'relation-get -r amqp:26 - rabbitmq-server/0'
egress-subnets: 10.5.0.52/32
hostname: 10.5.0.52
ingress-address: 10.5.0.52
password: sMPG4nw5KC4RGz3TZNTqgpjtH6kRWL5wRXG5ZVNHSrKSW6FcF5mYSqBHC7dWwHCr
private-address: 10.5.0.52
ssl_ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4VENDQWRtZ0F3SUJBZ0lVV0xjM2NRbEJ3enFEdWhzS091cmNxYi9EUDlNd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dqRVlNQllHQTFVRUF3d1BSR2wyYVc1bFFYVjBhRzl5YVhSNU1CNFhEVEU1TURneU9UQTJOVGMwTmxvWApEVEU1TURreU9EQTJOVGMwTjFvd0dqRVlNQllHQTFVRUF3d1BSR2wyYVc1bFFYVjBhRzl5YVhSNU1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTVId2pXMkdRUlljcGo5bFBvM05EMEo5RGdZbU8KR1RaU0grSjZvdENKMGxhUENoaDlNSzFoZTd5Rmk5b09QNkpDMUIxamNaRWxVUEdpRm1uK3JRL2MvTGVScUJrSQozS2U4R0FKcFlZT042WGU4cnkyRHR6WjJVNzNTOWNLRnZqWS8vQnkxM1JSL3kzenB0NnRNZHhSWEdyUXlaS1VPCk9nUWVXMkViMXBnMU1WekRZTE91My96RHorT1NoSFF5bUV2TEZQUFRjY1luWTFEckRFTDNYZGtuVUJwWWduQ3cKUzR6Q0thZ1czVzVTSjJJSnZmbmVGenpqbCsrVDE4aWYzSG1lcmpNVENtS3RxS3VxMnpBc0VIdlNJWHZGWElMaQpsUk9tL2NnS2dEV1llY1lyTVhxbHhCdzZkTVg0Z24xUm9PMEc2aEltWktyT1ZRdjI5TTBlRTY2ZE5RSURBUUFCCm95OHdMVEFhQmdOVkhSRUVFekFSZ2c5RWFYWnBibVZCZFhSb2IzSnBkSGt3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUZPTTB2bWp2d3MwNzczaVZ3elNyMlQ5b3Q1ZnAxbkthNXkvTQpiYzhEdDcwc1dVZnptS3RQYVUzcVN3U1kyZi9tVGNEblNRVFp4MHhsL2dMeG9KL2NMOTlKMXlrc0J6a0tNMCtuCnkxU0xSVmdta2o1QmFWZVdLU2U0Qi9XWXppbVBvRDVJWENWUm5XV21NMHdiejVCcUFpQ1ZRdlNScWpZb0lHQysKendibTJKdjdydThLbUcrTWNOOTZFMGJGekRVYXNBa3VVTllNcW1lenQ2dHVRK01ZdDBaWUdjYkNWNnJGYU9lNQozNmVtcUU2bXRoNE5sUHB0c2tCaGJEbjEzeTFkSEQ1blZNR0FQTGFZYU1iTWIvY1F0M0xVbm92RndscXNtOHY4Cm91em50c2VVOUhQZnMyTWxybXZ2RWY1dnFmV045SUliNVZCUTZNaVdVQm9GWkRpclpnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJREFEQ0NBZWlnQXdJQkFnSVVYQVNNM202d0dyYVptbU5UOS9TTmVvVUtUbUl3RFFZSktvWklodmNOQVFFTApCUUF3R2pFWU1CWUdBMVVFQXd3UFJHbDJhVzVsUVhWMGFHOXlhWFI1TUI0WERURTVNRGd5T0RBMk5UYzBOMW9YCkRURTVNVEV4TnpBMk5UYzBOMW93UlRGRE1FRUdBMVVFQXhNNlZtRjFiSFFnU1c1MFpYSnRaV1JwWVhSbElFTmwKY25ScFptbGpZWFJsSUVGMWRHaHZjbWwwZVNBb1kyaGhjbTB0Y0d0cExXeHZZMkZzS1RDQ0FTSXdEUVlKS29aSQpodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUtudEpCSnlxc0pPYXFjRmtyU3Rjbmg0czVINXA5VHdLN29jCnNDck03ZXpVTHZyWFFKVE1KYlFSclBDK2l5NTRsMEZtYzVSb0tzRE9qb2dSMXRtOURpYTBuQk8zRDFzd1NDYjYKMTl1bnRxRzVsMEdBTnFMdFdMWFpvajd4anJoYkVZOGZqdkRqMjF3RVFnYkMyL3ZFejNBMjBlZDFmVG9oYjdMaQovd1E0UncvRHFoZWZydXRTTytwU0grMkZnZ21MTWJSeVdVeStVRS8zZ0JXRkJoYVo4SDN2K0hya29lL1pjbWd6ClFiMFI4Y2s1N1FNZHFxNDlYQkhoaGREdGc3TWRnR3VWbjBsNU4zRXRGOTRMT1ByK3ZPL25MOE1QMEtydVdSVlkKakcxQ1VLTE1BeU9hU2pRM0NOd05WZWI1Z2tYVWxCMUNFQ1lqVFhVa1lCM3Y2UG9rbEpVQ0F3RUFBYU1UTUJFdwpEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFxbXc4amlrOURyQlkxb013CjV5SXN2U0VQVHNKZVF0YTM4Y1I5WFVQSnZJUmNKNU5ObHJ0Wm5NTWxBR3MwRGN6RHlWaDd0dkxzYVB3QmZIbmYKeVJFZzhueVA3NHdLU3k4cXF3WXUrSlV6NlNZRVlQaCtPQy85MXhDd1dIMmxFV1BIL3RoYmRLbmRuczc0N1dKSwpSZ1ZBOCt0clkrOFE3ZHM4QTF0UWhzUmNKc0t0V3VEVFp4UHFmbytTMldMbW1pczN6TC9ibG1oNVJjb0pKQkp0CkgyOFhTVnFDZ1pmY2NkRTF4WlkxeFNQWjFJU1VsWCtlQ2lUcEtNVGdpUzBlUThRMmVnQUNRcHd5c1dvOGlySjgKYmppZ1AzM0dtcjNJSWxMZ2hMTCtLSnVTRWQzaFlMY1hnYmxRNnYxU1E5UDBobWR3VHRWUStHUHhJUlVJSGpMawpVazdKdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
ssl_port: "5671"
ubuntu@test:~$ base64 -d
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4VENDQWRtZ0F3SUJBZ0lVV0xjM2NRbEJ3enFEdWhzS091cmNxYi9EUDlNd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dqRVlNQllHQTFVRUF3d1BSR2wyYVc1bFFYVjBhRzl5YVhSNU1CNFhEVEU1TURneU9UQTJOVGMwTmxvWApEVEU1TURreU9EQTJOVGMwTjFvd0dqRVlNQllHQTFVRUF3d1BSR2wyYVc1bFFYVjBhRzl5YVhSNU1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTVId2pXMkdRUlljcGo5bFBvM05EMEo5RGdZbU8KR1RaU0grSjZvdENKMGxhUENoaDlNSzFoZTd5Rmk5b09QNkpDMUIxamNaRWxVUEdpRm1uK3JRL2MvTGVScUJrSQozS2U4R0FKcFlZT042WGU4cnkyRHR6WjJVNzNTOWNLRnZqWS8vQnkxM1JSL3kzenB0NnRNZHhSWEdyUXlaS1VPCk9nUWVXMkViMXBnMU1WekRZTE91My96RHorT1NoSFF5bUV2TEZQUFRjY1luWTFEckRFTDNYZGtuVUJwWWduQ3cKUzR6Q0thZ1czVzVTSjJJSnZmbmVGenpqbCsrVDE4aWYzSG1lcmpNVENtS3RxS3VxMnpBc0VIdlNJWHZGWElMaQpsUk9tL2NnS2dEV1llY1lyTVhxbHhCdzZkTVg0Z24xUm9PMEc2aEltWktyT1ZRdjI5TTBlRTY2ZE5RSURBUUFCCm95OHdMVEFhQmdOVkhSRUVFekFSZ2c5RWFYWnBibVZCZFhSb2IzSnBkSGt3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUZPTTB2bWp2d3MwNzczaVZ3elNyMlQ5b3Q1ZnAxbkthNXkvTQpiYzhEdDcwc1dVZnptS3RQYVUzcVN3U1kyZi9tVGNEblNRVFp4MHhsL2dMeG9KL2NMOTlKMXlrc0J6a0tNMCtuCnkxU0xSVmdta2o1QmFWZVdLU2U0Qi9XWXppbVBvRDVJWENWUm5XV21NMHdiejVCcUFpQ1ZRdlNScWpZb0lHQysKendibTJKdjdydThLbUcrTWNOOTZFMGJGekRVYXNBa3VVTllNcW1lenQ2dHVRK01ZdDBaWUdjYkNWNnJGYU9lNQozNmVtcUU2bXRoNE5sUHB0c2tCaGJEbjEzeTFkSEQ1blZNR0FQTGFZYU1iTWIvY1F0M0xVbm92RndscXNtOHY4Cm91em50c2VVOUhQZnMyTWxybXZ2RWY1dnFmV045SUliNVZCUTZNaVdVQm9GWkRpclpnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJREFEQ0NBZWlnQXdJQkFnSVVYQVNNM202d0dyYVptbU5UOS9TTmVvVUtUbUl3RFFZSktvWklodmNOQVFFTApCUUF3R2pFWU1CWUdBMVVFQXd3UFJHbDJhVzVsUVhWMGFHOXlhWFI1TUI0WERURTVNRGd5T0RBMk5UYzBOMW9YCkRURTVNVEV4TnpBMk5UYzBOMW93UlRGRE1FRUdBMVVFQXhNNlZtRjFiSFFnU1c1MFpYSnRaV1JwWVhSbElFTmwKY25ScFptbGpZWFJsSUVGMWRHaHZjbWwwZVNBb1kyaGhjbTB0Y0d0cExXeHZZMkZzS1RDQ0FTSXdEUVlKS29aSQpodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUtudEpCSnlxc0pPYXFjRmtyU3Rjbmg0czVINXA5VHdLN29jCnNDck03ZXpVTHZyWFFKVE1KYlFSclBDK2l5NTRsMEZtYzVSb0tzRE9qb2dSMXRtOURpYTBuQk8zRDFzd1NDYjYKMTl1bnRxRzVsMEdBTnFMdFdMWFpvajd4anJoYkVZOGZqdkRqMjF3RVFnYkMyL3ZFejNBMjBlZDFmVG9oYjdMaQovd1E0UncvRHFoZWZydXRTTytwU0grMkZnZ21MTWJSeVdVeStVRS8zZ0JXRkJoYVo4SDN2K0hya29lL1pjbWd6ClFiMFI4Y2s1N1FNZHFxNDlYQkhoaGREdGc3TWRnR3VWbjBsNU4zRXRGOTRMT1ByK3ZPL25MOE1QMEtydVdSVlkKakcxQ1VLTE1BeU9hU2pRM0NOd05WZWI1Z2tYVWxCMUNFQ1lqVFhVa1lCM3Y2UG9rbEpVQ0F3RUFBYU1UTUJFdwpEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFxbXc4amlrOURyQlkxb013CjV5SXN2U0VQVHNKZVF0YTM4Y1I5WFVQSnZJUmNKNU5ObHJ0Wm5NTWxBR3MwRGN6RHlWaDd0dkxzYVB3QmZIbmYKeVJFZzhueVA3NHdLU3k4cXF3WXUrSlV6NlNZRVlQaCtPQy85MXhDd1dIMmxFV1BIL3RoYmRLbmRuczc0N1dKSwpSZ1ZBOCt0clkrOFE3ZHM4QTF0UWhzUmNKc0t0V3VEVFp4UHFmbytTMldMbW1pczN6TC9ibG1oNVJjb0pKQkp0CkgyOFhTVnFDZ1pmY2NkRTF4WlkxeFNQWjFJU1VsWCtlQ2lUcEtNVGdpUzBlUThRMmVnQUNRcHd5c1dvOGlySjgKYmppZ1AzM0dtcjNJSWxMZ2hMTCtLSnVTRWQzaFlMY1hnYmxRNnYxU1E5UDBobWR3VHRWUStHUHhJUlVJSGpMawpVazdKdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
-----BEGIN CERTIFICATE-----
MIIC8TCCAdmgAwIBAgIUWLc3cQlBwzqDuhsKOurcqb/DP9MwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAwwPRGl2aW5lQXV0aG9yaXR5MB4XDTE5MDgyOTA2NTc0NloX
DTE5MDkyODA2NTc0N1owGjEYMBYGA1UEAwwPRGl2aW5lQXV0aG9yaXR5MIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5HwjW2GQRYcpj9lPo3ND0J9DgYmO
GTZSH+J6otCJ0laPChh9MK1he7yFi9oOP6JC1B1jcZElUPGiFmn+rQ/c/LeRqBkI
3Ke8GAJpYYON6Xe8ry2DtzZ2U73S9cKFvjY//By13RR/y3zpt6tMdxRXGrQyZKUO
OgQeW2Eb1pg1MVzDYLOu3/zDz+OShHQymEvLFPPTccYnY1DrDEL3XdknUBpYgnCw
S4zCKagW3W5SJ2IJvfneFzzjl++T18if3HmerjMTCmKtqKuq2zAsEHvSIXvFXILi
lROm/cgKgDWYecYrMXqlxBw6dMX4gn1RoO0G6hImZKrOVQv29M0eE66dNQIDAQAB
oy8wLTAaBgNVHREEEzARgg9EaXZpbmVBdXRob3JpdHkwDwYDVR0TAQH/BAUwAwEB
/zANBgkqhkiG9w0BAQsFAAOCAQEAFOM0vmjvws0773iVwzSr2T9ot5fp1nKa5y/M
bc8Dt70sWUfzmKtPaU3qSwSY2f/mTcDnSQTZx0xl/gLxoJ/cL99J1yksBzkKM0+n
y1SLRVgmkj5BaVeWKSe4B/WYzimPoD5IXCVRnWWmM0wbz5BqAiCVQvSRqjYoIGC+
zwbm2Jv7ru8KmG+McN96E0bFzDUasAkuUNYMqmezt6tuQ+MYt0ZYGcbCV6rFaOe5
36emqE6mth4NlPptskBhbDn13y1dHD5nVMGAPLaYaMbMb/cQt3LUnovFwlqsm8v8
ouzntseU9HPfs2MlrmvvEf5vqfWN9IIb5VBQ6MiWUBoFZDirZg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDADCCAeigAwIBAgIUXASM3m6wGraZmmNT9/SNeoUKTmIwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAwwPRGl2aW5lQXV0aG9yaXR5MB4XDTE5MDgyODA2NTc0N1oX
DTE5MTExNzA2NTc0N1owRTFDMEEGA1UEAxM6VmF1bHQgSW50ZXJtZWRpYXRlIENl
cnRpZmljYXRlIEF1dGhvcml0eSAoY2hhcm0tcGtpLWxvY2FsKTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAKntJBJyqsJOaqcFkrStcnh4s5H5p9TwK7oc
sCrM7ezULvrXQJTMJbQRrPC+iy54l0Fmc5RoKsDOjogR1tm9Dia0nBO3D1swSCb6
19untqG5l0GANqLtWLXZoj7xjrhbEY8fjvDj21wEQgbC2/vEz3A20ed1fTohb7Li
/wQ4Rw/DqhefrutSO+pSH+2FggmLMbRyWUy+UE/3gBWFBhaZ8H3v+Hrkoe/Zcmgz
Qb0R8ck57QMdqq49XBHhhdDtg7MdgGuVn0l5N3EtF94LOPr+vO/nL8MP0KruWRVY
jG1CUKLMAyOaSjQ3CNwNVeb5gkXUlB1CECYjTXUkYB3v6PoklJUCAwEAAaMTMBEw
DwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAqmw8jik9DrBY1oMw
5yIsvSEPTsJeQta38cR9XUPJvIRcJ5NNlrtZnMMlAGs0DczDyVh7tvLsaPwBfHnf
yREg8nyP74wKSy8qqwYu+JUz6SYEYPh+OC/91xCwWH2lEWPH/thbdKndns747WJK
RgVA8+trY+8Q7ds8A1tQhsRcJsKtWuDTZxPqfo+S2WLmmis3zL/blmh5RcoJJBJt
H28XSVqCgZfccdE1xZY1xSPZ1ISUlX+eCiTpKMTgiS0eQ8Q2egACQpwysWo8irJ8
bjigP33Gmr3IIlLghLL+KJuSEd3hYLcXgblQ6v1SQ9P0hmdwTtVQ+GPxIRUIHjLk
Uk7Jug==
-----END CERTIFICATE-----

Changed in charms.openstack:
status:	New → Triaged
importance:	Undecided → High
Changed in layer-openstack:
status:	New → Triaged
importance:	Undecided → High
Changed in layer-openstack-api:
status:	New → Triaged
importance:	Undecided → High

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.