funman300
1e8d46b258
Recreating the Docker anchor left the supervised Core and host stranded in the dead namespace while systemd still reported them active — serving nobody, invisible to any monitoring that trusts unit state. Reproduced on staging: netns 4026539938 -> 4026540033, both pids unchanged in the old one, both units "active", traffic ConnectionResetError. There is no systemd-native edge signal to bind to. Containers do appear as units, but the scope name embeds the container ID (docker-<id>.scope), which changes on every recreate, so BindsTo= has no stable target; NetworkNamespacePath= resolves once at start; a .path unit on /run/netns would watch the file this tooling maintains. So: a level-triggered reconcile on a 10s timer, comparing the namespace the services are ACTUALLY in against the anchor's CURRENT one, acting only on a real difference. That cannot miss an event while the watcher restarts or dockerd is down, and needs no debounce — a burst of three recreations produced exactly one rebind. The trigger stays separable: a docker-events unit could invoke the same script. Anchor absent stops the dependants rather than falling back to host networking; docker unavailable logs once and retries on the next tick. Two defects found while testing and fixed here: - mount --bind STACKS when the old mount is busy, silently leaking nsfs entries; the bind helper now drains stale mounts in a loop. - reconcile must stop -> rebind -> start, not rebind -> restart: a running service holds the old namespace open and makes the umount fail busy. Staging also gained a faithful anchor container so the reproduction is structural rather than mocked. Economy state was byte-identical across every lifecycle test. Production units are templates only and remain uninstalled.
73 lines
3.2 KiB
Bash
Executable File
73 lines
3.2 KiB
Bash
Executable File
#!/bin/sh
|
|
# Publish a Docker container's network namespace into /run/netns so systemd
|
|
# units can enter it declaratively with NetworkNamespacePath=.
|
|
#
|
|
# WHY THIS EXISTS. Production Core and host must run inside the
|
|
# `openfut-fut-backend` container's netns: that is where the published client
|
|
# ports live and where the Python oracle answers on 127.0.0.1:8199. Today they
|
|
# get there with `nsenter --net=/proc/<pid>/ns/net`, where <pid> is typed by
|
|
# hand into a runbook.
|
|
#
|
|
# THE HAZARD THIS FIXES, MEASURED NOT ASSUMED. The container runs with
|
|
# `restart=unless-stopped`. On a container restart its netns inode CHANGES
|
|
# (observed 2026-08-22: net:[4026539938] -> net:[4026540033]). A hardcoded pid
|
|
# is then simply wrong, and — worse — any process already inside the old
|
|
# namespace keeps running in a namespace with no interfaces, silently serving
|
|
# nobody. Resolving by container NAME at every start removes the hardcoded pid;
|
|
# the companion watcher unit handles the already-running case by restarting the
|
|
# stack when the container restarts.
|
|
#
|
|
# Idempotent: a stale bind mount is unmounted and re-made, so re-running after a
|
|
# container restart is the fix, not a second problem.
|
|
#
|
|
# usage: openfut-netns-bind.sh <container-name> <netns-name>
|
|
set -eu
|
|
|
|
CONTAINER="${1:?container name}"
|
|
NSNAME="${2:?netns name}"
|
|
TARGET="/run/netns/${NSNAME}"
|
|
|
|
CPID="$(docker inspect -f '{{.State.Pid}}' "$CONTAINER" 2>/dev/null || true)"
|
|
if [ -z "$CPID" ] || [ "$CPID" = "0" ]; then
|
|
echo "openfut-netns-bind: container '$CONTAINER' is not running (pid='$CPID')" >&2
|
|
exit 1
|
|
fi
|
|
if [ ! -e "/proc/$CPID/ns/net" ]; then
|
|
echo "openfut-netns-bind: /proc/$CPID/ns/net does not exist" >&2
|
|
exit 1
|
|
fi
|
|
|
|
WANT="$(readlink "/proc/$CPID/ns/net")"
|
|
mkdir -p /run/netns
|
|
|
|
# Already published and already CURRENT? Then do nothing — re-mounting under a
|
|
# live service would be gratuitous churn.
|
|
if mountpoint -q "$TARGET" 2>/dev/null; then
|
|
HAVE="ns:[$(stat -c %i "$TARGET" 2>/dev/null || echo 0)]"
|
|
if [ "net:[$(stat -c %i "$TARGET" 2>/dev/null)]" = "$WANT" ]; then
|
|
echo "openfut-netns-bind: $TARGET already current ($WANT)"
|
|
exit 0
|
|
fi
|
|
echo "openfut-netns-bind: $TARGET is STALE ($HAVE, want $WANT) — refreshing"
|
|
# DRAIN, do not just pop. `mount --bind` STACKS: binding over a busy mount
|
|
# silently leaves the old one underneath, and staging grew two nsfs entries
|
|
# on the first rebind before this loop existed. Left alone that is one
|
|
# leaked mount per container recreation, and the buried namespaces are
|
|
# exactly the dead ones we are trying to get rid of.
|
|
#
|
|
# A umount can legitimately fail while a service still holds the old
|
|
# namespace open; the caller's job is to stop dependants FIRST. If it is
|
|
# still busy we stack rather than fail — a current top-of-stack mount is
|
|
# correct, just untidy — and say so.
|
|
while mountpoint -q "$TARGET" 2>/dev/null; do
|
|
umount "$TARGET" 2>/dev/null || {
|
|
echo "openfut-netns-bind: $TARGET still busy; stacking a current mount over it" >&2
|
|
break
|
|
}
|
|
done
|
|
fi
|
|
|
|
[ -e "$TARGET" ] || touch "$TARGET"
|
|
mount --bind "/proc/$CPID/ns/net" "$TARGET"
|
|
echo "openfut-netns-bind: $TARGET -> $CONTAINER pid $CPID $WANT"
|