intermediate CA: Incorrect padding

cloud:bionic-stein

Enabled intermediate CA with the following relations:
  - ["keystone:certificates", "vault:certificates"]
  - ["glance:certificates", "vault:certificates"]
  - ["cinder:certificates", "vault:certificates"]
  - ["nova-cloud-controller:certificates", "vault:certificates"]
  - ["neutron-api:certificates", "vault:certificates"]
  - ["designate:certificates", "vault:certificates"]
  - ["designate-bind:certificates", "vault:certificates"]
  - ["heat:certificates", "vault:certificates"]
  - ["gnocchi:certificates", "vault:certificates"]
  - ["aodh:certificates", "vault:certificates"]
  - ["openstack-dashboard:certificates", "vault:certificates"]
  - ["ceph-radosgw:certificates", "vault:certificates"]
  - ["ceilometer:certificates", "vault:certificates"]
  - ["barbican:certificates", "vault:certificates"]
  - ["barbican-vault:certificates", "vault:certificates"]
  - ["octavia:certificates", "vault:certificates"]
  - ["octavia-dashboard:certificates", "vault:certificates"]
  - ["octavia-diskimage-retrofit:certificates", "vault:certificates"]
  - ["glance-simplestreams-sync:certificates", "vault:certificates"]
  - ["keystone-ldap:certificates", "vault:certificates"]
  - ["rabbitmq-server:certificates", "vault:certificates"]
  - ["etcd:certificates", "vault:certificates"]

Followed instructions from this doc:
https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-certificate-management.html

After running "juju run-action upload-signed-csr" and action status showed completed with no failures. After sometimes (not sure how long, I checked juju status a few hours later) ceilometer-agent went into error state with the following stack trace:

ceilometer-agent/1 hook failed: "ceilometer-service-relation-changed"

2020-04-22 12:13:00 ERROR juju.worker.uniter.operation runhook.go:132 hook "ceilometer-service-relation-changed" failed: exit status 1
2020-04-22 12:18:02 INFO juju-log ceilometer-service:122: Registered config file: /etc/ceilometer/ceilometer.conf
2020-04-22 12:18:02 INFO juju-log ceilometer-service:122: Registered config file: /etc/memcached.conf
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed Traceback (most recent call last):
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/hooks/ceilometer-service-relation-changed", line 181, in <module>
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed hooks.execute(sys.argv)
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/core/hookenv.py", line 934, in execute
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed self._hooks[hook_name]()
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/utils.py", line 1597, in wrapped_f
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed stopstart, restart_functions)
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/core/host.py", line 741, in restart_on_change_helper
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed r = lambda_f()
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/utils.py", line 1596, in <lambda>
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/hooks/ceilometer-service-relation-changed", line 93, in ceilometer_changed
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed CONFIGS.write_all()
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/templating.py", line 334, in write_all
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed [self.write(k) for k in six.iterkeys(self.templates)]
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/templating.py", line 334, in <listcomp>
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed [self.write(k) for k in six.iterkeys(self.templates)]
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/templating.py", line 321, in write
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed _out = self.render(config_file)
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/templating.py", line 281, in render
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed ctxt = ostmpl.context()
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/templating.py", line 112, in context
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed _ctxt = context()
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/hooks/ceilometer_contexts.py", line 82, in __call__
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed fh.write(base64.b64decode(conf['rabbit_ssl_ca']))
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed File "/usr/lib/python3.6/base64.py", line 87, in b64decode
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed return binascii.a2b_base64(s)
2020-04-22 12:18:14 DEBUG ceilometer-service-relation-changed binascii.Error: Incorrect padding
2020-04-22 12:18:14 ERROR juju.worker.uniter.operation runhook.go:132 hook "ceilometer-service-relation-changed" failed: exit status 1

On ceilometer-agent/1 node:
root@node1:/etc/ceilometer# cat ceilometer.conf
# mitaka
###############################################################################
# [ WARNING ]
# ceilometer configuration file maintained by Juju
# local changes may be overwritten.
###############################################################################
[DEFAULT]
debug = False
verbose = False
logdir = /var/log/ceilometer

transport_url = rabbit://ceilometer:7ZVScKPmqxNqcLWpyVpHrcYMLkgj9kRqMgrmf27n29kKcf8nkYfby2twmcF94yhj@10.0.2.129:5672/openstack

[service_credentials]
auth_url = http://keystone-internal.spinda.solutionsqa:35357
interface = internalURL
project_name = services
username = ceilometer
password = nJMYmpfYHHxyqfLXzqxZ9hTxxcPFtY32qbgJcnzZRkCz8JHhfVfmpx4V27fj92hC
project_domain_name = service_domain
user_domain_name = service_domain
auth_type = password
[publisher]
telemetry_secret = a0c0d04e-b5a2-497a-92f6-539e49a27d8e
[keystone_authtoken]
auth_type = password
auth_uri = http://keystone.spinda.solutionsqa:5000/v3
auth_url = http://keystone-internal.spinda.solutionsqa:35357/v3
project_domain_name = service_domain
user_domain_name = service_domain
project_name = services
username = ceilometer
password = nJMYmpfYHHxyqfLXzqxZ9hTxxcPFtY32qbgJcnzZRkCz8JHhfVfmpx4V27fj92hC
signing_dir = /var/cache/ceilometer

memcached_servers = inet6:[::1]:11211

[oslo_messaging_rabbit]
root@node1:/etc/ceilometer# ls -l
total 16
-rw-r----- 1 root ceilometer 1311 Apr 20 23:25 ceilometer.conf
-rw-r--r-- 1 root ceilometer 1046 Feb 27 13:23 polling.yaml
-rw-r--r-- 1 root root 0 Apr 22 15:57 rabbit-client-ca.pem
-rw-r--r-- 1 root root 983 Feb 27 13:23 rootwrap.conf
drwxr-xr-x 2 root root 4096 Apr 20 20:31 rootwrap.d
root@node1:/etc/ceilometer#

Note that the rabbit-client-ca.pem size is 0.

On the rabbitmq-server unit, file /etc/ssl/certs/keystone_juju_ca_cert.pem does not exist. However /etc/ssl/certs/keystone_juju_ca_cert.pem file exists on glance, keystone, heat, etc..