Two things surfaced by the production promotion.
`status` counted namespace mounts with an unanchored grep, so on production
"run/netns/openfut" also matched "openfut-staging" and reported a phantom
"2 = leaked stack" against a perfectly healthy host. A status command that
invents a fault is the same class of bug as a unit that reports active while
serving nobody, so it is fixed with an exact mount-point match. The bind and
reconcile logic is untouched; it always umounted an exact path.
openfut-rollback-detached.sh makes the documented rollback executable rather
than a paragraph in a runbook: it removes supervision, resolves the anchor's
CURRENT pid from Docker, and relaunches the incumbent detached pair with the
environment replayed from the captured env.json. --dry-run prints the exact
commands and touches nothing, which is how it was validated while production
was still being served by the processes it would restore.
Recreating the Docker anchor left the supervised Core and host stranded in
the dead namespace while systemd still reported them active — serving
nobody, invisible to any monitoring that trusts unit state. Reproduced on
staging: netns 4026539938 -> 4026540033, both pids unchanged in the old
one, both units "active", traffic ConnectionResetError.
There is no systemd-native edge signal to bind to. Containers do appear as
units, but the scope name embeds the container ID (docker-<id>.scope), which
changes on every recreate, so BindsTo= has no stable target;
NetworkNamespacePath= resolves once at start; a .path unit on /run/netns
would watch the file this tooling maintains. So: a level-triggered reconcile
on a 10s timer, comparing the namespace the services are ACTUALLY in against
the anchor's CURRENT one, acting only on a real difference. That cannot miss
an event while the watcher restarts or dockerd is down, and needs no
debounce — a burst of three recreations produced exactly one rebind. The
trigger stays separable: a docker-events unit could invoke the same script.
Anchor absent stops the dependants rather than falling back to host
networking; docker unavailable logs once and retries on the next tick.
Two defects found while testing and fixed here:
- mount --bind STACKS when the old mount is busy, silently leaking nsfs
entries; the bind helper now drains stale mounts in a loop.
- reconcile must stop -> rebind -> start, not rebind -> restart: a running
service holds the old namespace open and makes the umount fail busy.
Staging also gained a faithful anchor container so the reproduction is
structural rather than mocked. Economy state was byte-identical across every
lifecycle test. Production units are templates only and remain uninstalled.
funman300