funman300
1e8d46b258
Recreating the Docker anchor left the supervised Core and host stranded in the dead namespace while systemd still reported them active — serving nobody, invisible to any monitoring that trusts unit state. Reproduced on staging: netns 4026539938 -> 4026540033, both pids unchanged in the old one, both units "active", traffic ConnectionResetError. There is no systemd-native edge signal to bind to. Containers do appear as units, but the scope name embeds the container ID (docker-<id>.scope), which changes on every recreate, so BindsTo= has no stable target; NetworkNamespacePath= resolves once at start; a .path unit on /run/netns would watch the file this tooling maintains. So: a level-triggered reconcile on a 10s timer, comparing the namespace the services are ACTUALLY in against the anchor's CURRENT one, acting only on a real difference. That cannot miss an event while the watcher restarts or dockerd is down, and needs no debounce — a burst of three recreations produced exactly one rebind. The trigger stays separable: a docker-events unit could invoke the same script. Anchor absent stops the dependants rather than falling back to host networking; docker unavailable logs once and retries on the next tick. Two defects found while testing and fixed here: - mount --bind STACKS when the old mount is busy, silently leaking nsfs entries; the bind helper now drains stale mounts in a loop. - reconcile must stop -> rebind -> start, not rebind -> restart: a running service holds the old namespace open and makes the umount fail busy. Staging also gained a faithful anchor container so the reproduction is structural rather than mocked. Economy state was byte-identical across every lifecycle test. Production units are templates only and remain uninstalled.
5.0 KiB
OpenFUT systemd units
Supervision for OpenFUT Core and the FIFA17 UTAS host. Replaces the previous
setsid nohup … nsenter … launch, which had no restart policy, no boot
persistence and no supervisor-visible logs.
| file | scope | installed? |
|---|---|---|
openfut-staging-netns.service |
binds the staging anchor netns | yes — proving ground |
openfut-staging-core.service |
staging Core, port 18081 | yes |
openfut-staging-host.service |
staging host, port 8299 | yes |
openfut-staging-netns-reconcile.{service,timer} |
staging netns lifecycle | yes |
openfut-netns.service |
binds the production anchor netns to /run/netns/openfut |
template only |
openfut-core.service |
production Core, port 18080 | template only |
openfut-host.service |
production host, port 8099 | template only |
openfut-netns-reconcile.{service,timer} |
production netns lifecycle | template only |
openfut-netns-bind.sh |
resolves the anchor netns by NAME, idempotently, drains stale mounts | helper |
openfut-netns-reconcile.sh |
keeps services in the anchor's CURRENT netns | helper |
openfut-wait-tcp.sh |
bounded readiness gate | helper |
openfut-supervision-install.sh |
install / start / status / uninstall per environment | helper |
Production templates are not installed. Deploy only via the plan in
OpenFUT-Vault/06 Operations/OpenFUT Service Supervision (staging-proven).md.
The three decisions worth knowing
1. Wants=, not Requires=, from host → Core. Measured on staging:
Requires propagates a Core stop into a host stop, and a later Core start does
not bring the host back — a routine Core restart would leave the client with
no server. With Wants, the host survives a Core outage, answers
503 core_unavailable (never a Python fallback), and resumes the moment Core
returns, with no supervisor intervention.
2. Readiness is an ExecStartPre TCP gate, not ordering. After=/Wants=
order units; Type=exec only proves the binary exec'd. Neither means Core can
serve. Core binds its listener after opening the DB, running migrations and
loading the content pack, so "port open" is a genuine readiness signal. The gate
is bounded and fails rather than blocking: a host that waits forever looks
healthy to the supervisor while serving nobody.
3. The netns is resolved by container NAME at every start. Production must
run inside openfut-fut-backend's network namespace. The container is
restart=unless-stopped, and its netns inode changes on restart — observed
net:[4026539938] → net:[4026540033]. A hardcoded pid is therefore wrong by
construction, and any process left in the old namespace keeps running with no
interfaces, silently serving nobody. openfut-netns-bind.sh re-resolves by name
and refreshes a stale bind mount; NetworkNamespacePath= then enters it
declaratively.
4. Stale namespaces are repaired automatically (this closes the old gap).
Recreating the anchor strands already-running services in the dead namespace,
and — the dangerous part — systemd still reports them active. Measured before
the fix: anchor net:[4026539938] → net:[4026540033], Core and host unchanged in
the old one, both units active, traffic ConnectionResetError.
openfut-netns-reconcile.sh on a 10s timer compares the namespace the services
are actually in against the anchor's current one and, only on a real
difference, performs one stop → rebind → start cycle. It is level-triggered, so
it cannot miss an event and needs no debounce: a burst of three back-to-back
recreations produced exactly one rebind.
Operating
sudo ./openfut-supervision-install.sh status staging # units, anchor, netns agreement, mounts
journalctl -u openfut-staging-netns-reconcile -f # silent unless it acts
sudo systemctl restart openfut-staging-core # host survives and recovers
sudo ./openfut-supervision-install.sh install staging # copy + enable (does not start)
sudo ./openfut-supervision-install.sh uninstall staging # disable, stop, remove units
status is the one command worth knowing: it prints each unit's state, the
anchor's current netns, the namespace each service is actually in with an
ok/MISMATCH verdict, and the mount count (1 healthy, >1 a leaked stack).
Config lives in EnvironmentFiles (…/systemd/core.env, host.env), generated
from the live process environment so supervision changed how the processes
start and nothing about what they do. Binaries are immutable copies, so a
later cargo build cannot change what is running.
Interaction with the staging lifecycle script
scripts/sold-staging-up.py still starts its own unsupervised processes and
refuses to run while a staging stack is up. Stop the units first:
sudo systemctl stop openfut-staging-host openfut-staging-core
python3 scripts/sold-staging-up.py --club real --variant highest …
Reconciling the two (having the script drive the units) is deliberately out of scope for the supervision milestone.