Comment 26 for bug 1850281

Revision history for this message
In , ferd352+000143 (ferd352+000143-redhat-bugs) wrote :

This morning this was logged for pmlogger_daily:

[root@MAGGIE ~]# systemctl status pmlogger_daily
● pmlogger_daily.service - Process archive logs
   Loaded: loaded (/usr/lib/systemd/system/pmlogger_daily.service; static; vendor preset: disabled)
   Active: failed (Result: timeout) since Mon 2019-11-04 00:12:01 CET; 7h ago
     Docs: man:pmlogger_daily(1)
  Process: 216795 ExecStart=/usr/libexec/pcp/bin/pmlogger_daily $PMLOGGER_DAILY_PARAMS (code=exited, status=0/SUCCESS)
 Main PID: 216795 (code=exited, status=0/SUCCESS)

Nov 04 00:10:01 MAGGIE systemd[1]: Starting Process archive logs...
Nov 04 00:12:01 MAGGIE systemd[1]: pmlogger_daily.service: Start operation timed out. Terminating.
Nov 04 00:12:01 MAGGIE systemd[1]: pmlogger_daily.service: Failed with result 'timeout'.
Nov 04 00:12:01 MAGGIE systemd[1]: Failed to start Process archive logs.

[root@MAGGIE ~]# journalctl -xe -u pmlogger_daily-- Defined-By: systemd-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- The unit pmlogger_daily.service has successfully entered the 'dead' state.
Nov 03 00:11:48 MAGGIE systemd[1]: Started Process archive logs.
-- Subject: A start job for unit pmlogger_daily.service has finished successfully
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit pmlogger_daily.service has finished successfully.
--
-- The job identifier is 366735.
Nov 04 00:10:01 MAGGIE systemd[1]: Starting Process archive logs...
-- Subject: A start job for unit pmlogger_daily.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit pmlogger_daily.service has begun execution.
--
-- The job identifier is 579462.
Nov 04 00:12:01 MAGGIE systemd[1]: pmlogger_daily.service: Start operation timed out. Terminating.
Nov 04 00:12:01 MAGGIE systemd[1]: pmlogger_daily.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit pmlogger_daily.service has entered the 'failed' state with result 'timeout'.
Nov 04 00:12:01 MAGGIE systemd[1]: Failed to start Process archive logs.
-- Subject: A start job for unit pmlogger_daily.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit pmlogger_daily.service has finished with a failure.
--
-- The job identifier is 579462 and the job result is failed.

The service monitor in Cockpit shows pmlogger_daily as failed, and starting it manually causes the same error.

November 4, 2019
8:10 AM
Failed to start Process archive logs.
systemd
8:10 AM
pmlogger_daily.service: Failed with result 'exit-code'.
systemd
8:10 AM
pmlogger_daily.service: Main process exited, code=exited, status=1/FAILURE
systemd
8:10 AM
Starting Process archive logs...
systemd
8:10 AM
Failed to start Process archive logs.
systemd
8:10 AM
pmlogger_daily.service: Failed with result 'exit-code'.
systemd
8:10 AM
pmlogger_daily.service: Main process exited, code=exited, status=1/FAILURE
systemd
8:10 AM
pmlogger_daily failed - see /var/log/pcp/pmlogger/pmlogger_daily.log
pcp
8:10 AM
Starting Process archive logs...
systemd
12:12 AM
Failed to start Process archive logs.
systemd

I then ran the script and this was logged:

[root@MAGGIE ~]# systemctl status pmlogger.service
● pmlogger.service - Performance Metrics Archive Logger
   Loaded: loaded (/usr/lib/systemd/system/pmlogger.service; enabled; vendor preset: enabled)
   Active: activating (start) since Mon 2019-11-04 08:04:03 CET; 1min 44s ago
     Docs: man:pmlogger(1)
  Process: 226303 ExecStart=/usr/share/pcp/lib/pmlogger start (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 393)
   Memory: 1.9M
   CGroup: /system.slice/pmlogger.service

Nov 04 08:04:03 MAGGIE systemd[1]: pmlogger.service: Service RestartSec=100ms expired, scheduling restart.
Nov 04 08:04:03 MAGGIE systemd[1]: pmlogger.service: Scheduled restart job, restart counter is at 2.
Nov 04 08:04:03 MAGGIE systemd[1]: Stopped Performance Metrics Archive Logger.
Nov 04 08:04:03 MAGGIE systemd[1]: Starting Performance Metrics Archive Logger...
Nov 04 08:04:03 MAGGIE pmlogger[226303]: Starting pmlogger ...
Nov 04 08:04:03 MAGGIE systemd[1]: pmlogger.service: Can't open PID file /run/pcp/pmlogger.pid (yet?) after start: No such file or directory
Nov 04 08:04:08 MAGGIE systemd[1]: pmlogger.service: New main PID 232026 does not belong to service, and PID file is not owned by root. Refusing.
Nov 04 08:04:08 MAGGIE systemd[1]: pmlogger.service: New main PID 232026 does not belong to service, and PID file is not owned by root. Refusing.

[root@MAGGIE ~]# journalctl -xe
-- Subject: A stop job for unit pmlogger.service has finished
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A stop job for unit pmlogger.service has finished.
--
-- The job identifier is 657493 and the job result is done.
Nov 04 08:04:03 MAGGIE systemd[1]: Starting Performance Metrics Archive Logger...
-- Subject: A start job for unit pmlogger.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit pmlogger.service has begun execution.
--
-- The job identifier is 657493.
Nov 04 08:04:03 MAGGIE pmlogger[226303]: Starting pmlogger ...
Nov 04 08:04:03 MAGGIE systemd[1]: pmlogger.service: Can't open PID file /run/pcp/pmlogger.pid (yet?) after start: No such file or directory
Nov 04 08:04:08 MAGGIE systemd[1]: pmlogger.service: New main PID 232026 does not belong to service, and PID file is not owned by root. Refusing.
Nov 04 08:04:08 MAGGIE systemd[1]: pmlogger.service: New main PID 232026 does not belong to service, and PID file is not owned by root. Refusing.
Nov 04 08:04:10 MAGGIE systemd[1]: pmlogger_check.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit pmlogger_check.service has successfully entered the 'dead' state.
Nov 04 08:04:10 MAGGIE systemd[1]: Started Check pmlogger instances are running.
-- Subject: A start job for unit pmlogger_check.service has finished successfully
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit pmlogger_check.service has finished successfully.
--
-- The job identifier is 657398.
Nov 04 08:04:10 MAGGIE audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmlogger_check comm="syste>
Nov 04 08:04:10 MAGGIE audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmlogger_check comm="system

The pmlogger service is still running but is shown as "Activating (starting)" rather than "Active (running)".