funman300
fc1fdcc5ab
Two things surfaced by the production promotion. `status` counted namespace mounts with an unanchored grep, so on production "run/netns/openfut" also matched "openfut-staging" and reported a phantom "2 = leaked stack" against a perfectly healthy host. A status command that invents a fault is the same class of bug as a unit that reports active while serving nobody, so it is fixed with an exact mount-point match. The bind and reconcile logic is untouched; it always umounted an exact path. openfut-rollback-detached.sh makes the documented rollback executable rather than a paragraph in a runbook: it removes supervision, resolves the anchor's CURRENT pid from Docker, and relaunches the incumbent detached pair with the environment replayed from the captured env.json. --dry-run prints the exact commands and touches nothing, which is how it was validated while production was still being served by the processes it would restore.
OpenFUT systemd units
Supervision for OpenFUT Core and the FIFA17 UTAS host. Replaces the previous
setsid nohup … nsenter … launch, which had no restart policy, no boot
persistence and no supervisor-visible logs.
| file | scope | installed? |
|---|---|---|
openfut-staging-netns.service |
binds the staging anchor netns | yes — proving ground |
openfut-staging-core.service |
staging Core, port 18081 | yes |
openfut-staging-host.service |
staging host, port 8299 | yes |
openfut-staging-netns-reconcile.{service,timer} |
staging netns lifecycle | yes |
openfut-netns.service |
binds the production anchor netns to /run/netns/openfut |
template only |
openfut-core.service |
production Core, port 18080 | template only |
openfut-host.service |
production host, port 8099 | template only |
openfut-netns-reconcile.{service,timer} |
production netns lifecycle | template only |
openfut-netns-bind.sh |
resolves the anchor netns by NAME, idempotently, drains stale mounts | helper |
openfut-netns-reconcile.sh |
keeps services in the anchor's CURRENT netns | helper |
openfut-wait-tcp.sh |
bounded readiness gate | helper |
openfut-supervision-install.sh |
install / start / status / uninstall per environment | helper |
Production templates are not installed. Deploy only via the plan in
OpenFUT-Vault/06 Operations/OpenFUT Service Supervision (staging-proven).md.
The three decisions worth knowing
1. Wants=, not Requires=, from host → Core. Measured on staging:
Requires propagates a Core stop into a host stop, and a later Core start does
not bring the host back — a routine Core restart would leave the client with
no server. With Wants, the host survives a Core outage, answers
503 core_unavailable (never a Python fallback), and resumes the moment Core
returns, with no supervisor intervention.
2. Readiness is an ExecStartPre TCP gate, not ordering. After=/Wants=
order units; Type=exec only proves the binary exec'd. Neither means Core can
serve. Core binds its listener after opening the DB, running migrations and
loading the content pack, so "port open" is a genuine readiness signal. The gate
is bounded and fails rather than blocking: a host that waits forever looks
healthy to the supervisor while serving nobody.
3. The netns is resolved by container NAME at every start. Production must
run inside openfut-fut-backend's network namespace. The container is
restart=unless-stopped, and its netns inode changes on restart — observed
net:[4026539938] → net:[4026540033]. A hardcoded pid is therefore wrong by
construction, and any process left in the old namespace keeps running with no
interfaces, silently serving nobody. openfut-netns-bind.sh re-resolves by name
and refreshes a stale bind mount; NetworkNamespacePath= then enters it
declaratively.
4. Stale namespaces are repaired automatically (this closes the old gap).
Recreating the anchor strands already-running services in the dead namespace,
and — the dangerous part — systemd still reports them active. Measured before
the fix: anchor net:[4026539938] → net:[4026540033], Core and host unchanged in
the old one, both units active, traffic ConnectionResetError.
openfut-netns-reconcile.sh on a 10s timer compares the namespace the services
are actually in against the anchor's current one and, only on a real
difference, performs one stop → rebind → start cycle. It is level-triggered, so
it cannot miss an event and needs no debounce: a burst of three back-to-back
recreations produced exactly one rebind.
Operating
sudo ./openfut-supervision-install.sh status staging # units, anchor, netns agreement, mounts
journalctl -u openfut-staging-netns-reconcile -f # silent unless it acts
sudo systemctl restart openfut-staging-core # host survives and recovers
sudo ./openfut-supervision-install.sh install staging # copy + enable (does not start)
sudo ./openfut-supervision-install.sh uninstall staging # disable, stop, remove units
status is the one command worth knowing: it prints each unit's state, the
anchor's current netns, the namespace each service is actually in with an
ok/MISMATCH verdict, and the mount count (1 healthy, >1 a leaked stack).
Config lives in EnvironmentFiles (…/systemd/core.env, host.env), generated
from the live process environment so supervision changed how the processes
start and nothing about what they do. Binaries are immutable copies, so a
later cargo build cannot change what is running.
Interaction with the staging lifecycle script
scripts/sold-staging-up.py still starts its own unsupervised processes and
refuses to run while a staging stack is up. Stop the units first:
sudo systemctl stop openfut-staging-host openfut-staging-core
python3 scripts/sold-staging-up.py --club real --variant highest …
Reconciling the two (having the script drive the units) is deliberately out of scope for the supervision milestone.