funman300
1e8d46b258
Recreating the Docker anchor left the supervised Core and host stranded in the dead namespace while systemd still reported them active — serving nobody, invisible to any monitoring that trusts unit state. Reproduced on staging: netns 4026539938 -> 4026540033, both pids unchanged in the old one, both units "active", traffic ConnectionResetError. There is no systemd-native edge signal to bind to. Containers do appear as units, but the scope name embeds the container ID (docker-<id>.scope), which changes on every recreate, so BindsTo= has no stable target; NetworkNamespacePath= resolves once at start; a .path unit on /run/netns would watch the file this tooling maintains. So: a level-triggered reconcile on a 10s timer, comparing the namespace the services are ACTUALLY in against the anchor's CURRENT one, acting only on a real difference. That cannot miss an event while the watcher restarts or dockerd is down, and needs no debounce — a burst of three recreations produced exactly one rebind. The trigger stays separable: a docker-events unit could invoke the same script. Anchor absent stops the dependants rather than falling back to host networking; docker unavailable logs once and retries on the next tick. Two defects found while testing and fixed here: - mount --bind STACKS when the old mount is busy, silently leaking nsfs entries; the bind helper now drains stale mounts in a loop. - reconcile must stop -> rebind -> start, not rebind -> restart: a running service holds the old namespace open and makes the umount fail busy. Staging also gained a faithful anchor container so the reproduction is structural rather than mocked. Economy state was byte-identical across every lifecycle test. Production units are templates only and remain uninstalled.
OpenFUT systemd units
Supervision for OpenFUT Core and the FIFA17 UTAS host. Replaces the previous
setsid nohup … nsenter … launch, which had no restart policy, no boot
persistence and no supervisor-visible logs.
| file | scope | installed? |
|---|---|---|
openfut-staging-netns.service |
binds the staging anchor netns | yes — proving ground |
openfut-staging-core.service |
staging Core, port 18081 | yes |
openfut-staging-host.service |
staging host, port 8299 | yes |
openfut-staging-netns-reconcile.{service,timer} |
staging netns lifecycle | yes |
openfut-netns.service |
binds the production anchor netns to /run/netns/openfut |
template only |
openfut-core.service |
production Core, port 18080 | template only |
openfut-host.service |
production host, port 8099 | template only |
openfut-netns-reconcile.{service,timer} |
production netns lifecycle | template only |
openfut-netns-bind.sh |
resolves the anchor netns by NAME, idempotently, drains stale mounts | helper |
openfut-netns-reconcile.sh |
keeps services in the anchor's CURRENT netns | helper |
openfut-wait-tcp.sh |
bounded readiness gate | helper |
openfut-supervision-install.sh |
install / start / status / uninstall per environment | helper |
Production templates are not installed. Deploy only via the plan in
OpenFUT-Vault/06 Operations/OpenFUT Service Supervision (staging-proven).md.
The three decisions worth knowing
1. Wants=, not Requires=, from host → Core. Measured on staging:
Requires propagates a Core stop into a host stop, and a later Core start does
not bring the host back — a routine Core restart would leave the client with
no server. With Wants, the host survives a Core outage, answers
503 core_unavailable (never a Python fallback), and resumes the moment Core
returns, with no supervisor intervention.
2. Readiness is an ExecStartPre TCP gate, not ordering. After=/Wants=
order units; Type=exec only proves the binary exec'd. Neither means Core can
serve. Core binds its listener after opening the DB, running migrations and
loading the content pack, so "port open" is a genuine readiness signal. The gate
is bounded and fails rather than blocking: a host that waits forever looks
healthy to the supervisor while serving nobody.
3. The netns is resolved by container NAME at every start. Production must
run inside openfut-fut-backend's network namespace. The container is
restart=unless-stopped, and its netns inode changes on restart — observed
net:[4026539938] → net:[4026540033]. A hardcoded pid is therefore wrong by
construction, and any process left in the old namespace keeps running with no
interfaces, silently serving nobody. openfut-netns-bind.sh re-resolves by name
and refreshes a stale bind mount; NetworkNamespacePath= then enters it
declaratively.
4. Stale namespaces are repaired automatically (this closes the old gap).
Recreating the anchor strands already-running services in the dead namespace,
and — the dangerous part — systemd still reports them active. Measured before
the fix: anchor net:[4026539938] → net:[4026540033], Core and host unchanged in
the old one, both units active, traffic ConnectionResetError.
openfut-netns-reconcile.sh on a 10s timer compares the namespace the services
are actually in against the anchor's current one and, only on a real
difference, performs one stop → rebind → start cycle. It is level-triggered, so
it cannot miss an event and needs no debounce: a burst of three back-to-back
recreations produced exactly one rebind.
Operating
sudo ./openfut-supervision-install.sh status staging # units, anchor, netns agreement, mounts
journalctl -u openfut-staging-netns-reconcile -f # silent unless it acts
sudo systemctl restart openfut-staging-core # host survives and recovers
sudo ./openfut-supervision-install.sh install staging # copy + enable (does not start)
sudo ./openfut-supervision-install.sh uninstall staging # disable, stop, remove units
status is the one command worth knowing: it prints each unit's state, the
anchor's current netns, the namespace each service is actually in with an
ok/MISMATCH verdict, and the mount count (1 healthy, >1 a leaked stack).
Config lives in EnvironmentFiles (…/systemd/core.env, host.env), generated
from the live process environment so supervision changed how the processes
start and nothing about what they do. Binaries are immutable copies, so a
later cargo build cannot change what is running.
Interaction with the staging lifecycle script
scripts/sold-staging-up.py still starts its own unsupervised processes and
refuses to run while a staging stack is up. Stop the units first:
sudo systemctl stop openfut-staging-host openfut-staging-core
python3 scripts/sold-staging-up.py --club real --variant highest …
Reconciling the two (having the script drive the units) is deliberately out of scope for the supervision milestone.