[series-upgrade] post-series-upgrade hook may fail due to race with principle

Seen on openstack-dashboard during series upgrade.

The race hazard with hacluster starting in the post-series upgrade hook was hit again.

unit-openstack-dashboard-2: 15:07:58 INFO unit.openstack-dashboard/2.juju-log Unit is ready
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade Traceback (most recent call last):
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/post-series-upgrade", line 508, in <module>
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade main()
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/post-series-upgrade", line 468, in main
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade hooks.execute(sys.argv)
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/core/hookenv.py", line 956, in execute
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade self._hooks[hook_name]()
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/post-series-upgrade", line 503, in post_series_upgrade
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade series_upgrade_complete(
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/utils.py", line 2212, in series_upgrade_complete
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade resume_unit_helper(configs)
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_utils.py", line 853, in resume_unit_helper
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade _pause_resume_helper(resume_unit, configs)
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_utils.py", line 866, in _pause_resume_helper
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade f(assess_status_func(configs),
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/utils.py", line 1772, in resume_unit
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade raise Exception("Couldn't resume: {}".format("; ".join(messages)))
unit-openstack-dashboard-2: 15:07:58 WARNING unit.openstack-dashboard/2.post-series-upgrade Exception: Couldn't resume: haproxy didn't resume cleanly.

From the log directly:

2021-06-18 15:07:58 DEBUG post-series-upgrade active
2021-06-18 15:07:58 INFO juju-log Unit is ready
2021-06-18 15:07:58 WARNING post-series-upgrade Traceback (most recent call last):
2021-06-18 15:07:58 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/post-series-upgrade", line 508, in <module>
2021-06-18 15:07:58 WARNING post-series-upgrade main()
2021-06-18 15:07:58 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/post-series-upgrade", line 468, in main
2021-06-18 15:07:58 WARNING post-series-upgrade hooks.execute(sys.argv)
2021-06-18 15:07:58 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/core/hookenv.py", line 956, in execute
2021-06-18 15:07:58 WARNING post-series-upgrade self._hooks[hook_name]()
2021-06-18 15:07:58 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/post-series-upgrade", line 503, in post_series_upgrade
2021-06-18 15:07:58 WARNING post-series-upgrade series_upgrade_complete(
2021-06-18 15:07:58 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/utils.py", line 2212, in series_upgrade_complete
2021-06-18 15:07:58 WARNING post-series-upgrade resume_unit_helper(configs)
2021-06-18 15:07:58 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_utils.py", line 853, in resume_unit_helper
2021-06-18 15:07:58 WARNING post-series-upgrade _pause_resume_helper(resume_unit, configs)
2021-06-18 15:07:58 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_utils.py", line 866, in _pause_resume_helper
2021-06-18 15:07:58 WARNING post-series-upgrade f(assess_status_func(configs),
2021-06-18 15:07:58 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/utils.py", line 1772, in resume_unit
2021-06-18 15:07:58 WARNING post-series-upgrade raise Exception("Couldn't resume: {}".format("; ".join(messages)))
2021-06-18 15:07:58 WARNING post-series-upgrade Exception: Couldn't resume: haproxy didn't resume cleanly.
2021-06-18 15:07:58 ERROR juju.worker.uniter.operation runhook.go:139 hook "post-series-upgrade" (via explicit, bespoke hook script) failed: exit status 1

The haproxy log for the period just shows:

Jun 18 13:25:21 juju-182b52-0-lxd-6 haproxy[439531]: [WARNING] 133/180728 (439531) : All workers exited. Exiting... (143)
Jun 18 15:07:58 juju-182b52-0-lxd-6 haproxy[2374]: [NOTICE] 168/150758 (2374) : New worker #1 (2383) forked
Jun 18 15:38:48 juju-182b52-0-lxd-6 haproxy[2383]: [WARNING] 168/153848 (2383) : Server dash_secure/openstack-dashboard-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused",
 check duration: 1ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Which may mean that the detection of the resumption wasn't clean??

At the same time pacemaker was logging:

Jun 18 15:07:42 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crm_attribute/4)
Jun 18 15:07:42 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=juju-182b52-0-lxd-6/crm_attribute/4, version=0.66.51)
Jun 18 15:07:47 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_ping) info: Reporting our current digest to juju-182b52-11-lxd-6: 083201dd09c2ced059e8cbda8bda6b06 for 0.66.51 (0x55c9f6ad6d70 0)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) notice: Result of monitor operation for res_horizon_haproxy on juju-182b52-0-lxd-6: 7 (not running) | call=18 key=res_horizon_haproxy_monitor_5000 confirmed=false cib-update=19
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) notice: juju-182b52-0-lxd-6-res_horizon_haproxy_monitor_5000:18 [ * haproxy.service - HAProxy Load Balancer\n Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled)\n Active: inactive (dead)\n Docs: man:haproxy(1)\n file:/usr/share/doc/haproxy/configuration.txt.gz\n\nJun 11 06:19:34 juju-182b52-0-lxd-6 systemd[1]: haproxy.service: Failed to reset devices.list: Operati
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/19)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.51 2
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.52 (null)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=52
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']/lrm_rsc_op[@id='res_horizon_haproxy_last_failure_0']: @operation_key=res_horizon_haproxy_monitor_5000, @transition-key=23:95:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @transition-magic=0:7;23:95:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @call-id=18, @rc-code=7, @interval=5000, @last-rc-change=1624028877, @exec-time=0,
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/19, version=0.66.52)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-attrd [470] (attrd_peer_update) notice: Setting fail-count-res_horizon_haproxy#monitor_5000[juju-182b52-0-lxd-6]: (unset) -> 1 | from juju-182b52-11-lxd-6
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-attrd [470] (attrd_peer_update) notice: Setting last-failure-res_horizon_haproxy#monitor_5000[juju-182b52-0-lxd-6]: (unset) -> 1624028877 | from juju-182b52-11-lxd-6
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.52 2
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.53 (null)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=53
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: ++ /cib/status/node_state[@id='1000']/transient_attributes[@id='1000']/instance_attributes[@id='status-1000']: <nvpair id="status-1000-fail-count-res_horizon_haproxy.monitor_5000" name="fail-count-res_horizon_haproxy#monitor_5000" value="1"/>
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-11-lxd-6/attrd/18, version=0.66.53)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.53 2
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.54 (null)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=54
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: ++ /cib/status/node_state[@id='1000']/transient_attributes[@id='1000']/instance_attributes[@id='status-1000']: <nvpair id="status-1000-last-failure-res_horizon_haproxy.monitor_5000" name="last-failure-res_horizon_haproxy#monitor_5000" value="1624028877"/>
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-11-lxd-6/attrd/19, version=0.66.54)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-execd [469] (cancel_recurring_action) info: Cancelling lsb operation res_horizon_haproxy_status_5000
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-controld [472] (do_lrm_rsc_op) info: Performing key=8:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627 op=res_horizon_haproxy_stop_0
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/20)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-execd [469] (log_execute) info: executing - rsc:res_horizon_haproxy action:stop call_id:20
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) info: Result of monitor operation for res_horizon_haproxy on juju-182b52-0-lxd-6: Cancelled | call=18 key=res_horizon_haproxy_monitor_5000 confirmed=true
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) notice: juju-182b52-0-lxd-6-res_horizon_haproxy_monitor_5000:18 [ * haproxy.service - HAProxy Load Balancer\n Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled)\n Active: inactive (dead)\n Docs: man:haproxy(1)\n file:/usr/share/doc/haproxy/configuration.txt.gz\n\nJun 11 06:19:34 juju-182b52-0-lxd-6 systemd[1]: haproxy.service: Failed to reset devices.list: Operati
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.54 2
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.55 (null)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=55
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']/lrm_rsc_op[@id='res_horizon_haproxy_last_0']: @operation_key=res_horizon_haproxy_stop_0, @operation=stop, @transition-key=8:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @transition-magic=-1:193;8:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1624028877, @last-run
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/20, version=0.66.55)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-execd [469] (log_finished) info: finished - rsc:res_horizon_haproxy action:stop call_id:20 pid:2338 exit-code:0 exec-time:43ms queue-time:0m
s
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) notice: Result of stop operation for res_horizon_haproxy on juju-182b52-0-lxd-6: 0 (ok) | call=20 key=res
_horizon_haproxy_stop_0 confirmed=true cib-update=21
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/21)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.55 2
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.56 (null)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=56
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']
/lrm_rsc_op[@id='res_horizon_haproxy_last_0']: @transition-magic=0:0;8:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @call-id=20, @rc-code=0, @op-status=0, @exec-time=43
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/21, ve
rsion=0.66.56)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-controld [472] (do_lrm_rsc_op) info: Performing key=23:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627 op=res_horizon_haproxy_start_0
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/22)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-execd [469] (log_execute) info: executing - rsc:res_horizon_haproxy action:start call_id:21
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.56 2
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.57 (null)
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=57
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']
/lrm_rsc_op[@id='res_horizon_haproxy_last_0']: @operation_key=res_horizon_haproxy_start_0, @operation=start, @transition-key=23:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @transition-magic=-1:
193;23:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
Jun 18 15:07:57 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/22, ve
rsion=0.66.57)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-execd [469] (log_finished) info: finished - rsc:res_horizon_haproxy action:start call_id:21 pid:2349 exit-code:0 exec-time:68ms queue-time:0
ms
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) notice: Result of start operation for res_horizon_haproxy on juju-182b52-0-lxd-6: 0 (ok) | call=21 key=re
s_horizon_haproxy_start_0 confirmed=true cib-update=23
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/23)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.57 2
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.58 (null)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=58
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']
/lrm_rsc_op[@id='res_horizon_haproxy_last_0']: @transition-magic=0:0;23:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @call-id=21, @rc-code=0, @op-status=0, @exec-time=68
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/23, ve
rsion=0.66.58)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-controld [472] (do_lrm_rsc_op) info: Performing key=7:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627 op=res_horizon_haproxy_monitor_5000
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/24)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.58 2
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.59 (null)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=59
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']
/lrm_rsc_op[@id='res_horizon_haproxy_monitor_5000']: @transition-key=7:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @transition-magic=-1:193;7:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @call-id
=-1, @rc-code=193, @op-status=-1, @last-rc-change=1624028878, @exec-time=0
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/24, ve
rsion=0.66.59)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) notice: Result of monitor operation for res_horizon_haproxy on juju-182b52-0-lxd-6: 0 (ok) | call=22 key=
res_horizon_haproxy_monitor_5000 confirmed=false cib-update=25
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/25)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.59 2
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.60 (null)
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=60
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']
/lrm_rsc_op[@id='res_horizon_haproxy_monitor_5000']: @transition-magic=0:0;7:98:0:c46237e3-f1ea-4ce1-bc73-c593a4dd6627, @call-id=22, @rc-code=0, @op-status=0, @exec-time=69
Jun 18 15:07:58 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/25, ve
rsion=0.66.60)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-attrd [470] (attrd_peer_update) notice: Setting last-failure-res_horizon_haproxy#monitor_5000[juju-182b52-0-lxd-6]: 1624028877 -> (unset)
 | from juju-182b52-11-lxd-6
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-attrd [470] (attrd_peer_update) notice: Setting fail-count-res_horizon_haproxy#monitor_5000[juju-182b52-0-lxd-6]: 1 -> (unset) | from juj
u-182b52-11-lxd-6
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='juju-182b52-0-lxd-6']//lrm_resourc
e[@id='res_horizon_haproxy'] to all (origin=local/crmd/26)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.60 2
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.61 (null)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: -- /cib/status/node_state[@id='1000']/transient_attributes[@id='1000']/instance_attributes[@id='status-1000
']/nvpair[@id='status-1000-last-failure-res_horizon_haproxy.monitor_5000']
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=61
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-11-lxd-6/attrd/20,
version=0.66.61)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.61 2
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.62 (null)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: -- /cib/status/node_state[@id='1000']/transient_attributes[@id='1000']/instance_attributes[@id='status-1000
']/nvpair[@id='status-1000-fail-count-res_horizon_haproxy.monitor_5000']
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=62
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=juju-182b52-11-lxd-6/attrd/21,
version=0.66.62)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.62 2
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.63 (null)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: -- /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=63
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='juju-182b52-0-lxd-6']//lrm_resource
[@id='res_horizon_haproxy']: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/26, version=0.66.62)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-controld [472] (delete_resource) info: Removing resource res_horizon_haproxy for 5d0932c1-bcac-4729-a1c3-41175a0429cf (root) on juju-182b5
2-11-lxd-6
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-execd [469] (cancel_recurring_action) info: Cancelling lsb operation res_horizon_haproxy_status_5000
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-controld [472] (lrm_remove_deleted_op) info: Removing op res_horizon_haproxy_monitor_5000:22 for deleted resource res_horizon_haproxy
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-controld [472] (notify_deleted) info: Notifying 5d0932c1-bcac-4729-a1c3-41175a0429cf on juju-182b52-11-lxd-6 that res_horizon_haproxy was deleted
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='juju-182b52-0-lxd-6']//lrm_resourc
e[@id='res_horizon_haproxy'] to all (origin=local/crmd/27)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-execd [469] (process_lrmd_get_rsc_info) info: Agent information for 'res_horizon_haproxy' not in cache
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) info: Result of monitor operation for res_horizon_haproxy on juju-182b52-0-lxd-6: Cancelled | call=22 key
=res_horizon_haproxy_monitor_5000 confirmed=true
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-controld [472] (process_lrm_event) info: Deletion of resource 'res_horizon_haproxy' complete after res_horizon_haproxy_monitor_5000
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.62 2
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.66.63 (null)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: -- /cib/status/node_state[@id='1000']/lrm[@id='1000']/lrm_resources/lrm_resource[@id='res_horizon_haproxy']
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @num_updates=63
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='juju-182b52-0-lxd-6']//lrm_resource
[@id='res_horizon_haproxy']: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/27, version=0.66.63)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/29)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='juju-182b52-0-lxd-6']//lrm_resourc
e[@id='res_horizon_haproxy'] to all (origin=local/crmd/30)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: --- 0.66.63 2
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: Diff: +++ 0.67.0 (null)
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib: @epoch=67, @num_updates=0
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstra
p-options-last-lrm-refresh']: @value=1624028879
Jun 18 15:07:59 juju-182b52-0-lxd-6 pacemaker-based [466] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=juju-182b52-0-lxd-6/crmd/29
, version=0.67.0)

etc.

It is resolved with "juju resolved <unit>" and then it completes so there's an issue with it not being resumable.