switch: refuse to arm at a dead target; watchdog: use a pidfile
Three times now the same sequence has broken the client path: a build guard correctly refuses to start the Rust replacement, and the `switch on` that follows in the same script arms anyway, because it never checked whether anything was listening. The redirect then lands on a closed socket and the working Python service is bypassed for no benefit. `on` now refuses unless the target port is listening. ALLOW_DEAD_TARGET=1 overrides it for arming ahead of a service that is about to start, but that has to be deliberate. Verified both ways: rc=2 and nothing installed against a dead port, rc=0 and two rules with the override. The watchdog now writes a pidfile. Stopping it by command-line match is unsafe -- any shell whose arguments merely mention the script name matches too, which has now killed the wrong process twice here (once via `pkill -f`, once via a /proc/*/cmdline substring loop).
This commit is contained in:
@@ -201,6 +201,24 @@ cmd_on() {
|
|||||||
is_port "${TPORT:-}" || die "--target-port must be a port"
|
is_port "${TPORT:-}" || die "--target-port must be a port"
|
||||||
[[ "$IPORT" != "$TPORT" ]] || die "--intercept-port and --target-port must differ"
|
[[ "$IPORT" != "$TPORT" ]] || die "--intercept-port and --target-port must differ"
|
||||||
|
|
||||||
|
# REFUSE to arm at a port nothing is listening on. Arming a switch whose
|
||||||
|
# target is dead silently breaks the client path: the redirect happens, the
|
||||||
|
# connection is refused, and the proven Python service is bypassed for no
|
||||||
|
# benefit. This has happened three times, always the same way — a build guard
|
||||||
|
# correctly refuses to start the replacement, and the `on` that follows in the
|
||||||
|
# same script arms anyway because it never checked.
|
||||||
|
#
|
||||||
|
# Overridable for the rare case of arming ahead of a service that is about to
|
||||||
|
# start, but it must be deliberate rather than the default.
|
||||||
|
if [[ "${ALLOW_DEAD_TARGET:-0}" != "1" ]]; then
|
||||||
|
if ! ss -ltn 2>/dev/null | grep -qE "[:.]${TPORT}[[:space:]]"; then
|
||||||
|
die "REFUSING: nothing is listening on target port $TPORT.
|
||||||
|
Arming would break the client path — the redirect would land on a closed
|
||||||
|
socket and the working service would be bypassed.
|
||||||
|
Start the replacement first, or set ALLOW_DEAD_TARGET=1 if that is intended."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
local tag; tag="$(tag_for "$NAME")"
|
local tag; tag="$(tag_for "$NAME")"
|
||||||
|
|
||||||
# Never stack: start from a known state for THIS name only.
|
# Never stack: start from a known state for THIS name only.
|
||||||
|
|||||||
@@ -76,6 +76,16 @@ except Exception:
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# A pidfile, because stopping this by command-line match is unsafe: any shell
|
||||||
|
# whose arguments merely mention the script name matches too. That mistake has
|
||||||
|
# killed the wrong process twice in this project.
|
||||||
|
PIDFILE="${OPENFUT_WATCHDOG_PIDFILE:-${TMPDIR:-/tmp}/openfut-watchdog-$NAME.pid}"
|
||||||
|
if [[ -f "$PIDFILE" ]] && kill -0 "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null; then
|
||||||
|
echo "watchdog: already running for '$NAME' (pid $(cat "$PIDFILE"))" >&2; exit 1
|
||||||
|
fi
|
||||||
|
echo $$ > "$PIDFILE"
|
||||||
|
trap 'rm -f "$PIDFILE"' EXIT
|
||||||
|
|
||||||
say "started: probing $PROBE every ${INTERVAL}s ($KIND); rolls back after $MAX_FAIL consecutive failures"
|
say "started: probing $PROBE every ${INTERVAL}s ($KIND); rolls back after $MAX_FAIL consecutive failures"
|
||||||
fails=0
|
fails=0
|
||||||
while true; do
|
while true; do
|
||||||
|
|||||||
Reference in New Issue
Block a user