prometheus-grok-exporter is stuck in blocked state after node reboot

OpenStack Ussuri, 21.04 charms.

As part of the OpenStack deployment, we restart all servers, AZ by AZ at the end of the process. For this particular deployment, 2 out of 93 prometheus-grok-exporter units are stuck in blocked state after the reboot with the following message: "Service is not running. Check that files defined in 'global-paths' exists or manually check the service running on the unit the command sudo snap run prometheus-grok-exporter.grok-exporter"

These units are subordinate to glance charm.

This is the second time the problem happens. Each time on a fresh deployment.

The only workaround for me so far was to remove and re-add "glance - prometheus-grok-exporter relation".

ubuntu@juju-c5556f-21-lxd-8:~$ snap list
Name Version Rev Tracking Publisher Notes
core18 2074 latest/stable canonical✓ broken
lxd 21029 4.0/stable/… canonical✓ broken
prometheus-grok-exporter 2 latest/stable peppepetra86 broken
snapd 12398 latest/stable canonical✓ broken
ubuntu@juju-c5556f-21-lxd-8:~$ snap info prometheus-grok-exporter
name: prometheus-grok-exporter
summary: Export prometheus metrics from parsed log data
publisher: Giuseppe Petralia (peppepetra86)
store-url: https://snapcraft.io/prometheus-grok-exporter
contact: <email address hidden>
license: unset
description: |
  Grok is a tool to parse crappy unstructured log data into something structured and queryable. Grok
  is heavily used in Logstash to provide log data as input for ElasticSearch.
services:
  prometheus-grok-exporter.grok-exporter: simple, enabled, inactive
snap-id: tfalgVP400GhNMgQpyHOc7yv01adoVXl
tracking: latest/stable
refresh-date: today at 07:39 UTC
channels:
  latest/stable: 1.0.0 2020-03-25 (2) 6MB -
  latest/candidate: ↑
  latest/beta: 1.0.0 2020-03-20 (1) 6MB -
  latest/edge: ↑
installed: (2) 0B broken

ubuntu@juju-c5556f-21-lxd-8:~$ systemctl | grep grok

ubuntu@juju-c5556f-21-lxd-8:~$ sudo snap start prometheus-grok-exporter
Started.

ubuntu@juju-c5556f-21-lxd-8:~$ systemctl | grep grok
  snap-prometheus\x2dgrok\x2dexporter-2.mount loaded active mounted Mount unit for prometheus-grok-exporter, revision 2
● snap.prometheus-grok-exporter.grok-exporter.service loaded failed failed Service for snap application prometheus-grok-exporter.grok-exporter

ubuntu@juju-c5556f-21-lxd-8:~$ systemctl status snap.prometheus-grok-exporter.grok-exporter.service
● snap.prometheus-grok-exporter.grok-exporter.service - Service for snap application prometheus-grok-exporter.grok-exporter
     Loaded: loaded (/etc/systemd/system/snap.prometheus-grok-exporter.grok-exporter.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2021-08-05 13:41:34 UTC; 12s ago
    Process: 50739 ExecStart=/usr/bin/snap run prometheus-grok-exporter.grok-exporter (code=exited, status=1/FAILURE)
   Main PID: 50739 (code=exited, status=1/FAILURE)

Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Scheduled restart job, restart counter is at 5.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Start request repeated too quickly.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Failed with result 'exit-code'.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Failed to start Service for snap application prometheus-grok-exporter.grok-exporter.

ubuntu@juju-c5556f-21-lxd-8:~$ journalctl -u snap.prometheus-grok-exporter.grok-exporter.service
-- Logs begin at Thu 2021-08-05 07:25:42 UTC, end at Thu 2021-08-05 13:42:12 UTC. --
Aug 05 07:39:46 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 07:39:47 juju-c5556f-21-lxd-8 systemd[1]: Stopping Service for snap application prometheus-grok-exporter.grok-exporter...
Aug 05 07:39:47 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Succeeded.
Aug 05 07:39:47 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 07:39:48 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 07:39:49 juju-c5556f-21-lxd-8 prometheus-grok-exporter.grok-exporter[44012]: Starting server on http://juju-c5556f-21-lxd-8:9144/metrics
Aug 05 07:39:49 juju-c5556f-21-lxd-8 systemd[1]: Stopping Service for snap application prometheus-grok-exporter.grok-exporter...
Aug 05 07:39:49 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Succeeded.
Aug 05 07:39:49 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 07:39:49 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 07:39:49 juju-c5556f-21-lxd-8 prometheus-grok-exporter.grok-exporter[44174]: Starting server on http://0.0.0.0:9144/metrics
Aug 05 07:43:46 juju-c5556f-21-lxd-8 systemd[1]: Stopping Service for snap application prometheus-grok-exporter.grok-exporter...
Aug 05 07:43:46 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Succeeded.
Aug 05 07:43:46 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 07:43:46 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 07:43:47 juju-c5556f-21-lxd-8 prometheus-grok-exporter.grok-exporter[54155]: Starting server on http://0.0.0.0:9144/metrics
Aug 05 12:27:52 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Succeeded.
Aug 05 12:27:52 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
-- Reboot --
Aug 05 12:32:39 juju-c5556f-21-lxd-8 systemd[1]: Dependency failed for Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 12:32:39 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Job snap.prometheus-grok-exporter.grok-exporter.service/start failed with result 'dependency'.
Aug 05 13:41:33 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:33 juju-c5556f-21-lxd-8 prometheus-grok-exporter.grok-exporter[50681]: cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_t9c7Mt//dev: No such file or directory
Aug 05 13:41:33 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Main process exited, code=exited, status=1/FAILURE
Aug 05 13:41:33 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Failed with result 'exit-code'.
Aug 05 13:41:33 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Scheduled restart job, restart counter is at 1.
Aug 05 13:41:33 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:33 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:33 juju-c5556f-21-lxd-8 prometheus-grok-exporter.grok-exporter[50695]: cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_5if4dy//dev: No such file or directory
Aug 05 13:41:33 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Main process exited, code=exited, status=1/FAILURE
Aug 05 13:41:33 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Failed with result 'exit-code'.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Scheduled restart job, restart counter is at 2.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 prometheus-grok-exporter.grok-exporter[50709]: cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_L3zAnB//dev: No such file or directory
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Main process exited, code=exited, status=1/FAILURE
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Failed with result 'exit-code'.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Scheduled restart job, restart counter is at 3.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 prometheus-grok-exporter.grok-exporter[50723]: cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_75xUdC//dev: No such file or directory
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Main process exited, code=exited, status=1/FAILURE
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Failed with result 'exit-code'.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Scheduled restart job, restart counter is at 4.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Started Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 prometheus-grok-exporter.grok-exporter[50739]: cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_bqq23O//dev: No such file or directory
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Main process exited, code=exited, status=1/FAILURE
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Failed with result 'exit-code'.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Scheduled restart job, restart counter is at 5.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Stopped Service for snap application prometheus-grok-exporter.grok-exporter.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Start request repeated too quickly.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: snap.prometheus-grok-exporter.grok-exporter.service: Failed with result 'exit-code'.
Aug 05 13:41:34 juju-c5556f-21-lxd-8 systemd[1]: Failed to start Service for snap application prometheus-grok-exporter.grok-exporter.

ubuntu@juju-c5556f-21-lxd-8:~$ sudo snap run prometheus-grok-exporter.grok-exporter
cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_VTIXx5//dev: No such file or directory