As mentioned in #3 I am able to consistently reproduce this by running our automated OpenStack upgrade tests. [1]

Here is what is happening:

1. We pause the principle unit (nova-compute).
2. We perform an upgrade of all packages on the unit, including ceilometer-agent-compute. When upgrading this package, it wants to restart the service but the service refuses to start because nova-compute.service is masked.
3. We resume the principle unit (nova-compute), which restarts nova-compute.service but ceilometer-agent-compute.service is left behind, stopped.

Working on a solution.

unit-nova-compute-0.log:2021-08-17 11:41:59 DEBUG openstack-upgrade   alembic ceilometer-agent-compute ceilometer-common keystone-common
unit-nova-compute-0.log:2021-08-17 11:41:59 DEBUG openstack-upgrade   python3-ceilometer python3-cinderclient python3-cliff python3-cryptography
unit-nova-compute-0.log:2021-08-17 11:42:01 DEBUG openstack-upgrade Get:53 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 ceilometer-common all 1:15.0.0-0ubuntu2~cloud0 [27.0 kB]
unit-nova-compute-0.log:2021-08-17 11:42:01 DEBUG openstack-upgrade Get:54 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 ceilometer-agent-compute all 1:15.0.0-0ubuntu2~cloud0 [16.9 kB]
unit-nova-compute-0.log:2021-08-17 11:42:01 DEBUG openstack-upgrade Get:55 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 python3-ceilometer all 1:15.0.0-0ubuntu2~cloud0 [212 kB]
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../49-ceilometer-common_1%3a15.0.0-0ubuntu2~cloud0_all.deb ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Unpacking ceilometer-common (1:15.0.0-0ubuntu2~cloud0) over (1:14.0.0-0ubuntu0.20.04.3) ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../50-ceilometer-agent-compute_1%3a15.0.0-0ubuntu2~cloud0_all.deb ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Unpacking ceilometer-agent-compute (1:15.0.0-0ubuntu2~cloud0) over (1:14.0.0-0ubuntu0.20.04.3) ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../51-python3-ceilometer_1%3a15.0.0-0ubuntu2~cloud0_all.deb ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Unpacking python3-ceilometer (1:15.0.0-0ubuntu2~cloud0) over (1:14.0.0-0ubuntu0.20.04.3) ...
unit-nova-compute-0.log:2021-08-17 11:42:34 DEBUG openstack-upgrade Setting up ceilometer-common (1:15.0.0-0ubuntu2~cloud0) ...
unit-nova-compute-0.log:2021-08-17 11:42:34 DEBUG openstack-upgrade Configuration file '/etc/ceilometer/ceilometer.conf'
unit-nova-compute-0.log:2021-08-17 11:42:57 DEBUG openstack-upgrade Setting up python3-ceilometer (1:15.0.0-0ubuntu2~cloud0) ...
unit-nova-compute-0.log:2021-08-17 11:43:00 DEBUG openstack-upgrade Setting up ceilometer-agent-compute (1:15.0.0-0ubuntu2~cloud0) ...
unit-nova-compute-0.log:2021-08-17 11:43:00 DEBUG openstack-upgrade Installing new version of config file /etc/init.d/ceilometer-agent-compute ...
unit-nova-compute-0.log:2021-08-17 11:43:02 DEBUG openstack-upgrade Failed to restart ceilometer-agent-compute.service: Unit nova-compute.service is masked.

[1] https://github.com/openstack-charmers/charmed-openstack-tester/blob/master/tests/openstack-upgrade/tests/tests.yaml