switch: refuse to arm at a dead target; watchdog: use a pidfile
Three times now the same sequence has broken the client path: a build guard correctly refuses to start the Rust replacement, and the `switch on` that follows in the same script arms anyway, because it never checked whether anything was listening. The redirect then lands on a closed socket and the working Python service is bypassed for no benefit. `on` now refuses unless the target port is listening. ALLOW_DEAD_TARGET=1 overrides it for arming ahead of a service that is about to start, but that has to be deliberate. Verified both ways: rc=2 and nothing installed against a dead port, rc=0 and two rules with the override. The watchdog now writes a pidfile. Stopping it by command-line match is unsafe -- any shell whose arguments merely mention the script name matches too, which has now killed the wrong process twice here (once via `pkill -f`, once via a /proc/*/cmdline substring loop).
This commit is contained in:
@@ -201,6 +201,24 @@ cmd_on() {
|
||||
is_port "${TPORT:-}" || die "--target-port must be a port"
|
||||
[[ "$IPORT" != "$TPORT" ]] || die "--intercept-port and --target-port must differ"
|
||||
|
||||
# REFUSE to arm at a port nothing is listening on. Arming a switch whose
|
||||
# target is dead silently breaks the client path: the redirect happens, the
|
||||
# connection is refused, and the proven Python service is bypassed for no
|
||||
# benefit. This has happened three times, always the same way — a build guard
|
||||
# correctly refuses to start the replacement, and the `on` that follows in the
|
||||
# same script arms anyway because it never checked.
|
||||
#
|
||||
# Overridable for the rare case of arming ahead of a service that is about to
|
||||
# start, but it must be deliberate rather than the default.
|
||||
if [[ "${ALLOW_DEAD_TARGET:-0}" != "1" ]]; then
|
||||
if ! ss -ltn 2>/dev/null | grep -qE "[:.]${TPORT}[[:space:]]"; then
|
||||
die "REFUSING: nothing is listening on target port $TPORT.
|
||||
Arming would break the client path — the redirect would land on a closed
|
||||
socket and the working service would be bypassed.
|
||||
Start the replacement first, or set ALLOW_DEAD_TARGET=1 if that is intended."
|
||||
fi
|
||||
fi
|
||||
|
||||
local tag; tag="$(tag_for "$NAME")"
|
||||
|
||||
# Never stack: start from a known state for THIS name only.
|
||||
|
||||
@@ -76,6 +76,16 @@ except Exception:
|
||||
fi
|
||||
}
|
||||
|
||||
# A pidfile, because stopping this by command-line match is unsafe: any shell
|
||||
# whose arguments merely mention the script name matches too. That mistake has
|
||||
# killed the wrong process twice in this project.
|
||||
PIDFILE="${OPENFUT_WATCHDOG_PIDFILE:-${TMPDIR:-/tmp}/openfut-watchdog-$NAME.pid}"
|
||||
if [[ -f "$PIDFILE" ]] && kill -0 "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null; then
|
||||
echo "watchdog: already running for '$NAME' (pid $(cat "$PIDFILE"))" >&2; exit 1
|
||||
fi
|
||||
echo $$ > "$PIDFILE"
|
||||
trap 'rm -f "$PIDFILE"' EXIT
|
||||
|
||||
say "started: probing $PROBE every ${INTERVAL}s ($KIND); rolls back after $MAX_FAIL consecutive failures"
|
||||
fails=0
|
||||
while true; do
|
||||
|
||||
Reference in New Issue
Block a user