Recreating the Docker anchor left the supervised Core and host stranded in
the dead namespace while systemd still reported them active — serving
nobody, invisible to any monitoring that trusts unit state. Reproduced on
staging: netns 4026539938 -> 4026540033, both pids unchanged in the old
one, both units "active", traffic ConnectionResetError.
There is no systemd-native edge signal to bind to. Containers do appear as
units, but the scope name embeds the container ID (docker-<id>.scope), which
changes on every recreate, so BindsTo= has no stable target;
NetworkNamespacePath= resolves once at start; a .path unit on /run/netns
would watch the file this tooling maintains. So: a level-triggered reconcile
on a 10s timer, comparing the namespace the services are ACTUALLY in against
the anchor's CURRENT one, acting only on a real difference. That cannot miss
an event while the watcher restarts or dockerd is down, and needs no
debounce — a burst of three recreations produced exactly one rebind. The
trigger stays separable: a docker-events unit could invoke the same script.
Anchor absent stops the dependants rather than falling back to host
networking; docker unavailable logs once and retries on the next tick.
Two defects found while testing and fixed here:
- mount --bind STACKS when the old mount is busy, silently leaking nsfs
entries; the bind helper now drains stale mounts in a loop.
- reconcile must stop -> rebind -> start, not rebind -> restart: a running
service holds the old namespace open and makes the umount fail busy.
Staging also gained a faithful anchor container so the reproduction is
structural rather than mocked. Economy state was byte-identical across every
lifecycle test. Production units are templates only and remain uninstalled.
Replaces the detached `setsid nohup … nsenter …` launch, which had no restart
policy, no boot persistence and no supervisor-visible logs. Staging units are
installed and proven; production units are TEMPLATES and are not installed.
Three decisions, each measured rather than assumed:
* `Wants=`, not `Requires=`, from host to Core. With `Requires`, stopping Core
stopped the host AND a later Core start did not bring it back -- a routine
Core restart would leave the client with no server. With `Wants` the host
survives a Core outage, answers 503 core_unavailable, never falls back to
Python, and resumes the moment Core returns with no intervention. Both halves
tested.
* Readiness is a bounded ExecStartPre TCP gate, because ordering proves nothing
about readiness and Type=exec only proves the binary exec'd. Core binds its
listener after migrations and content load, so "port open" is a real signal.
The gate FAILS rather than blocking: a host that waits forever looks healthy
while serving nobody.
* The netns is resolved by container NAME every start. The container is
restart=unless-stopped and its netns inode CHANGES on restart (measured:
4026539938 -> 4026540033), so a hardcoded pid is wrong by construction and
anything left in the old namespace serves nobody. Proven equivalent to today's
nsenter against a scratch container, never production's namespace.
`systemd-analyze verify` caught two real defects before deployment:
StartLimitIntervalSec/StartLimitBurst sat in [Service], where systemd 252
silently ignores them, so the crash-loop ceiling was not taking effect; and a
Documentation URL containing %20 parsed as a specifier. Both fixed and the
effective properties re-confirmed from the running units.
Staging evidence: Core-first ordering, host refused when Core is absent or
merely not listening, outage survival, automatic recovery, restart, graceful
stop with no strays, boot simulated via multi-user.target, 3x SIGKILL contained
at ~5s spacing, journald logs, and economy state byte-identical throughout
(integrity ok, fk 0).
funman300