| 2021-12-13 19:51:25 |
Brian Goff |
bug |
|
|
added bug |
| 2021-12-13 19:52:37 |
Brian Goff |
description |
Package: systemd
Version: 245.4-4ubuntu3.13
When running a systemd unit where type=forking and setting a pidfile (example at the end of the description), if the process referred to by the pid in the pidfile exits before systemd has read the file, systemd complains (visible in journald logs) "New main PID <PID> does not exist or is a zombie."
The problem is systemd never records the actual exit status, and querying them from the unit properties shows that the exit status is 0/success even though.
The "Result" property on the unit is "protocol", signifying that we've run afoul of the forking protocol with systemd.
In this case we haven't really broken protocol just that we've exposed a race between monitoring the forked process.
This can happen with any sort of error in the forked process,
Since systemd should be reaping the process anyway, it seems like we should be able to get a correct exit status here.
If there is a small delay between starting the process and the exit then systemd has enough time to attach to the process and monitor correctly.
The properties one would normally check on this process are all zeroed out:
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
As is the `EXIT_STATUS` environment variable passed along to any "ExecStop" commands.
In some cases I've seen the "stop_time" set in the ExecStart properties of the service, but found this to be unreliable.
I've tried working around this by keeping the control process alive to wait and see if the forked process exits quickly and recording the exit status myself. This is a decent work-around however causes some extra overhead, and seems like it gets into the territory of what I'd expect systemd to do for me.
The example below is just simulating what might happen with a real process that errors out quickly.
Example:
cat << EOF > /tmp/systemd-forking-bug.sh
#!/usr/bin/env bash
(
echo \$BASHPID > \$PIDFILE
exit 1
) &
echo control process exiting
EOF
cat << EOF > /tmp/systemd-forking-bug.service
[Service]
Type=forking
PIDFile=/tmp/systemd-forking-bug.pid
ExecStart=/bin/bash /tmp/systemd-forking-bug.sh
EOF
sudo mv /tmp/systemd-forking-bug.service /run/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start systemd-forking-bug.service
sudo systemctl status systemd-forking-bug.service
sudo journalctl --lines=5 -u systemd-forking-bug.service |
Package: systemd
Version: 245.4-4ubuntu3.13
When running a systemd unit where type=forking and setting a pidfile (example at the end of the description), if the process referred to by the pid in the pidfile exits before systemd has read the file, systemd complains (visible in journald logs) "New main PID <PID> does not exist or is a zombie."
The problem is systemd never records the actual exit status, and querying them from the unit properties shows that the exit status is 0/success even though it exited non-zero.
The "Result" property on the unit is "protocol", signifying that we've run afoul of the forking protocol with systemd.
In this case we haven't really broken protocol just that we've exposed a race between monitoring the forked process.
This can happen with any sort of error in the forked process,
Since systemd should be reaping the process anyway, it seems like we should be able to get a correct exit status here.
If there is a small delay between starting the process and the exit then systemd has enough time to attach to the process and monitor correctly.
The properties one would normally check on this process are all zeroed out:
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
As is the `EXIT_STATUS` environment variable passed along to any "ExecStop" commands.
In some cases I've seen the "stop_time" set in the ExecStart properties of the service, but found this to be unreliable.
I've tried working around this by keeping the control process alive to wait and see if the forked process exits quickly and recording the exit status myself. This is a decent work-around however causes some extra overhead, and seems like it gets into the territory of what I'd expect systemd to do for me.
The example below is just simulating what might happen with a real process that errors out quickly.
Example:
cat << EOF > /tmp/systemd-forking-bug.sh
#!/usr/bin/env bash
(
echo \$BASHPID > \$PIDFILE
exit 1
) &
echo control process exiting
EOF
cat << EOF > /tmp/systemd-forking-bug.service
[Service]
Type=forking
PIDFile=/tmp/systemd-forking-bug.pid
ExecStart=/bin/bash /tmp/systemd-forking-bug.sh
EOF
sudo mv /tmp/systemd-forking-bug.service /run/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start systemd-forking-bug.service
sudo systemctl status systemd-forking-bug.service
sudo journalctl --lines=5 -u systemd-forking-bug.service |
|
| 2021-12-13 19:54:27 |
Brian Goff |
description |
Package: systemd
Version: 245.4-4ubuntu3.13
When running a systemd unit where type=forking and setting a pidfile (example at the end of the description), if the process referred to by the pid in the pidfile exits before systemd has read the file, systemd complains (visible in journald logs) "New main PID <PID> does not exist or is a zombie."
The problem is systemd never records the actual exit status, and querying them from the unit properties shows that the exit status is 0/success even though it exited non-zero.
The "Result" property on the unit is "protocol", signifying that we've run afoul of the forking protocol with systemd.
In this case we haven't really broken protocol just that we've exposed a race between monitoring the forked process.
This can happen with any sort of error in the forked process,
Since systemd should be reaping the process anyway, it seems like we should be able to get a correct exit status here.
If there is a small delay between starting the process and the exit then systemd has enough time to attach to the process and monitor correctly.
The properties one would normally check on this process are all zeroed out:
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
As is the `EXIT_STATUS` environment variable passed along to any "ExecStop" commands.
In some cases I've seen the "stop_time" set in the ExecStart properties of the service, but found this to be unreliable.
I've tried working around this by keeping the control process alive to wait and see if the forked process exits quickly and recording the exit status myself. This is a decent work-around however causes some extra overhead, and seems like it gets into the territory of what I'd expect systemd to do for me.
The example below is just simulating what might happen with a real process that errors out quickly.
Example:
cat << EOF > /tmp/systemd-forking-bug.sh
#!/usr/bin/env bash
(
echo \$BASHPID > \$PIDFILE
exit 1
) &
echo control process exiting
EOF
cat << EOF > /tmp/systemd-forking-bug.service
[Service]
Type=forking
PIDFile=/tmp/systemd-forking-bug.pid
ExecStart=/bin/bash /tmp/systemd-forking-bug.sh
EOF
sudo mv /tmp/systemd-forking-bug.service /run/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start systemd-forking-bug.service
sudo systemctl status systemd-forking-bug.service
sudo journalctl --lines=5 -u systemd-forking-bug.service |
Package: systemd
Version: 245.4-4ubuntu3.13
When running a systemd unit where type=forking and setting a pidfile (example at the end of the description), if the process referred to by the pid in the pidfile exits before systemd has read the file, systemd complains (visible in journald logs) "New main PID <PID> does not exist or is a zombie."
The problem is systemd never records the actual exit status, and querying them from the unit properties shows that the exit status is 0/success even though it exited non-zero.
The "Result" property on the unit is "protocol", indicating that we've run afoul of the forking protocol with systemd.
In this case we haven't really broken protocol just that we've exposed a race between monitoring the forked process.
This can happen with any sort of error in the forked process,
Since systemd should be reaping the process anyway, it seems like we should be able to get a correct exit status here.
If there is a small delay between starting the process and the exit then systemd has enough time to attach to the process and monitor correctly.
The properties one would normally check on this process are all zeroed out:
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
As is the `EXIT_STATUS` environment variable passed along to any "ExecStop" commands.
In some cases I've seen the "stop_time" set in the ExecStart properties of the service, but found this to be unreliable.
I've tried working around this by keeping the control process alive to wait and see if the forked process exits quickly and recording the exit status myself. This is a decent work-around however causes some extra overhead, and seems like it gets into the territory of what I'd expect systemd to do for me.
The example below is just simulating what might happen with a real process that errors out quickly.
Example:
cat << EOF > /tmp/systemd-forking-bug.sh
#!/usr/bin/env bash
(
echo \$BASHPID > \$PIDFILE
exit 1
) &
echo control process exiting
EOF
cat << EOF > /tmp/systemd-forking-bug.service
[Service]
Type=forking
PIDFile=/tmp/systemd-forking-bug.pid
ExecStart=/bin/bash /tmp/systemd-forking-bug.sh
EOF
sudo mv /tmp/systemd-forking-bug.service /run/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start systemd-forking-bug.service
sudo systemctl status systemd-forking-bug.service
sudo journalctl --lines=5 -u systemd-forking-bug.service |
|
| 2021-12-13 19:54:43 |
Brian Goff |
description |
Package: systemd
Version: 245.4-4ubuntu3.13
When running a systemd unit where type=forking and setting a pidfile (example at the end of the description), if the process referred to by the pid in the pidfile exits before systemd has read the file, systemd complains (visible in journald logs) "New main PID <PID> does not exist or is a zombie."
The problem is systemd never records the actual exit status, and querying them from the unit properties shows that the exit status is 0/success even though it exited non-zero.
The "Result" property on the unit is "protocol", indicating that we've run afoul of the forking protocol with systemd.
In this case we haven't really broken protocol just that we've exposed a race between monitoring the forked process.
This can happen with any sort of error in the forked process,
Since systemd should be reaping the process anyway, it seems like we should be able to get a correct exit status here.
If there is a small delay between starting the process and the exit then systemd has enough time to attach to the process and monitor correctly.
The properties one would normally check on this process are all zeroed out:
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
As is the `EXIT_STATUS` environment variable passed along to any "ExecStop" commands.
In some cases I've seen the "stop_time" set in the ExecStart properties of the service, but found this to be unreliable.
I've tried working around this by keeping the control process alive to wait and see if the forked process exits quickly and recording the exit status myself. This is a decent work-around however causes some extra overhead, and seems like it gets into the territory of what I'd expect systemd to do for me.
The example below is just simulating what might happen with a real process that errors out quickly.
Example:
cat << EOF > /tmp/systemd-forking-bug.sh
#!/usr/bin/env bash
(
echo \$BASHPID > \$PIDFILE
exit 1
) &
echo control process exiting
EOF
cat << EOF > /tmp/systemd-forking-bug.service
[Service]
Type=forking
PIDFile=/tmp/systemd-forking-bug.pid
ExecStart=/bin/bash /tmp/systemd-forking-bug.sh
EOF
sudo mv /tmp/systemd-forking-bug.service /run/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start systemd-forking-bug.service
sudo systemctl status systemd-forking-bug.service
sudo journalctl --lines=5 -u systemd-forking-bug.service |
Package: systemd
Version: 245.4-4ubuntu3.13
When running a systemd unit where type=forking and setting a pidfile (example at the end of the description), if the process referred to by the pid in the pidfile exits before systemd has read the file, systemd complains (visible in journald logs) "New main PID <PID> does not exist or is a zombie."
The problem is systemd never records the actual exit status, and querying them from the unit properties shows that the exit status is 0/success even though it exited non-zero.
The "Result" property on the unit is "protocol", indicating that we've run afoul of the forking protocol with systemd.
In this case we haven't really broken protocol just that we've exposed a race with monitoring the forked process.
This can happen with any sort of error in the forked process,
Since systemd should be reaping the process anyway, it seems like we should be able to get a correct exit status here.
If there is a small delay between starting the process and the exit then systemd has enough time to attach to the process and monitor correctly.
The properties one would normally check on this process are all zeroed out:
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
As is the `EXIT_STATUS` environment variable passed along to any "ExecStop" commands.
In some cases I've seen the "stop_time" set in the ExecStart properties of the service, but found this to be unreliable.
I've tried working around this by keeping the control process alive to wait and see if the forked process exits quickly and recording the exit status myself. This is a decent work-around however causes some extra overhead, and seems like it gets into the territory of what I'd expect systemd to do for me.
The example below is just simulating what might happen with a real process that errors out quickly.
Example:
cat << EOF > /tmp/systemd-forking-bug.sh
#!/usr/bin/env bash
(
echo \$BASHPID > \$PIDFILE
exit 1
) &
echo control process exiting
EOF
cat << EOF > /tmp/systemd-forking-bug.service
[Service]
Type=forking
PIDFile=/tmp/systemd-forking-bug.pid
ExecStart=/bin/bash /tmp/systemd-forking-bug.sh
EOF
sudo mv /tmp/systemd-forking-bug.service /run/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start systemd-forking-bug.service
sudo systemctl status systemd-forking-bug.service
sudo journalctl --lines=5 -u systemd-forking-bug.service |
|
| 2021-12-13 20:03:37 |
Tianon Gravi |
bug |
|
|
added subscriber Tianon Gravi |
