funman300
1e8d46b258
Recreating the Docker anchor left the supervised Core and host stranded in the dead namespace while systemd still reported them active — serving nobody, invisible to any monitoring that trusts unit state. Reproduced on staging: netns 4026539938 -> 4026540033, both pids unchanged in the old one, both units "active", traffic ConnectionResetError. There is no systemd-native edge signal to bind to. Containers do appear as units, but the scope name embeds the container ID (docker-<id>.scope), which changes on every recreate, so BindsTo= has no stable target; NetworkNamespacePath= resolves once at start; a .path unit on /run/netns would watch the file this tooling maintains. So: a level-triggered reconcile on a 10s timer, comparing the namespace the services are ACTUALLY in against the anchor's CURRENT one, acting only on a real difference. That cannot miss an event while the watcher restarts or dockerd is down, and needs no debounce — a burst of three recreations produced exactly one rebind. The trigger stays separable: a docker-events unit could invoke the same script. Anchor absent stops the dependants rather than falling back to host networking; docker unavailable logs once and retries on the next tick. Two defects found while testing and fixed here: - mount --bind STACKS when the old mount is busy, silently leaking nsfs entries; the bind helper now drains stale mounts in a loop. - reconcile must stop -> rebind -> start, not rebind -> restart: a running service holds the old namespace open and makes the umount fail busy. Staging also gained a faithful anchor container so the reproduction is structural rather than mocked. Economy state was byte-identical across every lifecycle test. Production units are templates only and remain uninstalled.
148 lines
6.7 KiB
Bash
Executable File
148 lines
6.7 KiB
Bash
Executable File
#!/bin/sh
|
|
# Keep supervised native services inside the CURRENT network namespace of a
|
|
# Docker anchor container, and stop them when that anchor is gone.
|
|
#
|
|
# ── THE FAILURE THIS EXISTS FOR ────────────────────────────────────────────
|
|
# OpenFUT's Core and host are native binaries that must run inside the anchor
|
|
# container's netns. Recreating (or merely restarting) that container gives it a
|
|
# NEW netns; the already-running services stay in the old one. Measured on
|
|
# staging 2026-08-22:
|
|
#
|
|
# anchor 37288a0fe816 net:[4026539938] -> 55541a03b66e net:[4026540033]
|
|
# core pid 2081773 net:[4026539938] (unchanged, now orphaned)
|
|
# host pid 2081791 net:[4026539938] (unchanged, now orphaned)
|
|
# systemd: both "active" traffic: ConnectionResetError
|
|
#
|
|
# Both units report HEALTHY while serving nobody. That is the whole problem:
|
|
# the failure is invisible to the supervisor.
|
|
#
|
|
# ── WHY A RECONCILE AND NOT AN EVENT WATCHER ───────────────────────────────
|
|
# There is no systemd-native edge signal to bind to. Containers do appear as
|
|
# `docker-<id>.scope` units (cgroup driver is systemd), but the scope NAME
|
|
# embeds the container ID, and the ID changes on recreate — so there is no
|
|
# stable unit for BindsTo=. `NetworkNamespacePath` is resolved at unit start
|
|
# only, and a .path unit on /run/netns/<name> would watch the very file this
|
|
# script maintains (circular).
|
|
#
|
|
# So the trigger is a timer, and the check is LEVEL-triggered: it compares the
|
|
# namespace the services are ACTUALLY in against the anchor's CURRENT one. That
|
|
# is strictly more robust than an edge-triggered watcher, which can miss events
|
|
# while it is itself restarting or while dockerd is down, and it needs no
|
|
# debounce logic — a burst of container events collapses into at most one
|
|
# reconcile per tick, because the only question asked is "does the observed
|
|
# state differ from the desired state right now?".
|
|
#
|
|
# The trigger is deliberately separable from the action. If detection latency
|
|
# ever matters, a `docker events` unit can invoke THIS SAME script; nothing here
|
|
# would change.
|
|
#
|
|
# ── INVARIANTS ─────────────────────────────────────────────────────────────
|
|
# * A container PID is runtime state and is NEVER persisted. It is resolved from
|
|
# Docker on every run.
|
|
# * Identity is the netns inode of /proc/<current-anchor-pid>/ns/net, never the
|
|
# container name (same name != same namespace) and never a cached value.
|
|
# * systemd stays the service authority: this script only requests start/stop/
|
|
# restart, and unit ordering does the sequencing.
|
|
# * Silence when correct. It logs only when it acts or fails, so a 10s timer
|
|
# does not fill the journal.
|
|
#
|
|
# usage: openfut-netns-reconcile.sh <container> <nsname> <netns-unit> <unit>...
|
|
set -eu
|
|
|
|
CONTAINER="${1:?container name}"; shift
|
|
NSNAME="${1:?netns name}"; shift
|
|
NSUNIT="${1:?netns unit}"; shift
|
|
[ "$#" -ge 1 ] || { echo "reconcile: at least one dependent unit required" >&2; exit 2; }
|
|
UNITS="$*"
|
|
HERE="$(dirname "$0")"
|
|
|
|
log() { echo "openfut-netns-reconcile: $*"; }
|
|
|
|
# ---- 1. Is Docker even answering? -----------------------------------------
|
|
# A daemon outage must be a clean, quiet failure that the timer retries, never a
|
|
# spin and never a destructive action taken on incomplete information.
|
|
if ! docker info >/dev/null 2>&1; then
|
|
log "docker daemon unavailable — taking NO action, will retry on the next tick"
|
|
exit 0
|
|
fi
|
|
|
|
# ---- 2. Resolve the anchor, by name, right now ----------------------------
|
|
CPID="$(docker inspect -f '{{.State.Pid}}' "$CONTAINER" 2>/dev/null || true)"
|
|
|
|
if [ -z "$CPID" ] || [ "$CPID" = "0" ] || [ ! -e "/proc/$CPID/ns/net" ]; then
|
|
# Anchor gone. Services must NOT keep pretending to be healthy inside a
|
|
# namespace whose owner has died — and must never fall back to host
|
|
# networking. Stop them; a later tick starts them again once the anchor is
|
|
# back, which is what makes recovery automatic.
|
|
RUNNING=""
|
|
for u in $UNITS; do
|
|
[ "$(systemctl is-active "$u" 2>/dev/null)" = "active" ] && RUNNING="$RUNNING $u"
|
|
done
|
|
if [ -n "$RUNNING" ]; then
|
|
log "anchor '$CONTAINER' is ABSENT — stopping$RUNNING (no host-network fallback)"
|
|
# Reverse order: dependants before the thing they depend on.
|
|
# shellcheck disable=SC2086
|
|
systemctl stop $RUNNING || true
|
|
systemctl stop "$NSUNIT" || true
|
|
fi
|
|
exit 0
|
|
fi
|
|
|
|
WANT="$(readlink "/proc/$CPID/ns/net")"
|
|
|
|
# ---- 3. Compare against where the services ACTUALLY are -------------------
|
|
# Observed state, not a remembered value: this self-heals no matter how the
|
|
# drift happened (container recreate, restart, manual nsenter, anything).
|
|
NEED_ACTION=0
|
|
REASON=""
|
|
for u in $UNITS; do
|
|
state="$(systemctl is-active "$u" 2>/dev/null || true)"
|
|
if [ "$state" != "active" ]; then
|
|
NEED_ACTION=1; REASON="$REASON $u=$state"
|
|
continue
|
|
fi
|
|
mp="$(systemctl show -p MainPID --value "$u" 2>/dev/null || echo 0)"
|
|
if [ -z "$mp" ] || [ "$mp" = "0" ] || [ ! -e "/proc/$mp/ns/net" ]; then
|
|
NEED_ACTION=1; REASON="$REASON $u=nopid"
|
|
continue
|
|
fi
|
|
have="$(readlink "/proc/$mp/ns/net")"
|
|
if [ "$have" != "$WANT" ]; then
|
|
NEED_ACTION=1; REASON="$REASON $u=$have"
|
|
fi
|
|
done
|
|
|
|
if [ "$NEED_ACTION" = "0" ]; then
|
|
exit 0 # correct and silent
|
|
fi
|
|
|
|
# ---- 4. One controlled rebind cycle ---------------------------------------
|
|
log "anchor '$CONTAINER' pid=$CPID ns=$WANT; drift:$REASON"
|
|
log "namespace changed or services adrift — requesting one rebind cycle"
|
|
|
|
# STOP FIRST, then rebind, then start — not restart-around-a-rebind. While a
|
|
# service is still running it holds the OLD namespace open, the umount fails
|
|
# busy, and `mount --bind` silently STACKS a second nsfs entry over it. Measured:
|
|
# the first rebind left two mounts on the path. Stopping the dependants releases
|
|
# the old namespace so the drain actually succeeds.
|
|
#
|
|
# Reverse order on the way down (dependants before their dependency), forward on
|
|
# the way up — and the way up is systemd's job: `start` honours the units' own
|
|
# After=/Requires=, so Core is listening before the host's readiness gate runs.
|
|
REV=""
|
|
for u in $UNITS; do REV="$u $REV"; done
|
|
# shellcheck disable=SC2086
|
|
systemctl stop $REV || true
|
|
|
|
"$HERE/openfut-netns-bind.sh" "$CONTAINER" "$NSNAME"
|
|
systemctl restart "$NSUNIT"
|
|
|
|
# shellcheck disable=SC2086
|
|
systemctl start $UNITS
|
|
|
|
for u in $UNITS; do
|
|
mp="$(systemctl show -p MainPID --value "$u" 2>/dev/null || echo 0)"
|
|
now="$( [ "$mp" != "0" ] && readlink "/proc/$mp/ns/net" || echo '-')"
|
|
log "rebound $u pid=$mp ns=$now"
|
|
done
|