ops: systemd supervision for Core and the FIFA17 host (staging-proven)

Replaces the detached `setsid nohup … nsenter …` launch, which had no restart policy, no boot persistence and no supervisor-visible logs. Staging units are installed and proven; production units are TEMPLATES and are not installed. Three decisions, each measured rather than assumed: * `Wants=`, not `Requires=`, from host to Core. With `Requires`, stopping Core stopped the host AND a later Core start did not bring it back -- a routine Core restart would leave the client with no server. With `Wants` the host survives a Core outage, answers 503 core_unavailable, never falls back to Python, and resumes the moment Core returns with no intervention. Both halves tested. * Readiness is a bounded ExecStartPre TCP gate, because ordering proves nothing about readiness and Type=exec only proves the binary exec'd. Core binds its listener after migrations and content load, so "port open" is a real signal. The gate FAILS rather than blocking: a host that waits forever looks healthy while serving nobody. * The netns is resolved by container NAME every start. The container is restart=unless-stopped and its netns inode CHANGES on restart (measured: 4026539938 -> 4026540033), so a hardcoded pid is wrong by construction and anything left in the old namespace serves nobody. Proven equivalent to today's nsenter against a scratch container, never production's namespace. `systemd-analyze verify` caught two real defects before deployment: StartLimitIntervalSec/StartLimitBurst sat in [Service], where systemd 252 silently ignores them, so the crash-loop ceiling was not taking effect; and a Documentation URL containing %20 parsed as a specifier. Both fixed and the effective properties re-confirmed from the running units. Staging evidence: Core-first ordering, host refused when Core is absent or merely not listening, outage survival, automatic recovery, restart, graceful stop with no strays, boot simulated via multi-user.target, 3x SIGKILL contained at ~5s spacing, journald logs, and economy state byte-identical throughout (integrity ok, fk 0).
2026-08-22 20:46:15 +00:00
parent 9026220533
commit 8ae432223a
8 changed files with 414 additions and 0 deletions
@@ -0,0 +1,75 @@
+# OpenFUT systemd units
+
+Supervision for OpenFUT Core and the FIFA17 UTAS host. Replaces the previous
+`setsid nohup … nsenter …` launch, which had no restart policy, no boot
+persistence and no supervisor-visible logs.
+
+| file | scope | installed? |
+|---|---|---|
+| `openfut-staging-core.service` | staging Core, port 18081 | **yes** — proving ground |
+| `openfut-staging-host.service` | staging host, port 8299 | **yes** |
+| `openfut-netns.service` | publishes the container netns to `/run/netns/openfut` | template only |
+| `openfut-core.service` | production Core, port 18080 | template only |
+| `openfut-host.service` | production host, port 8099 | template only |
+| `openfut-netns-bind.sh` | resolves the container netns by NAME, idempotently | helper |
+| `openfut-wait-tcp.sh` | bounded readiness gate | helper |
+
+Production templates are **not installed**. Deploy only via the plan in
+`OpenFUT-Vault/06 Operations/OpenFUT Service Supervision (staging-proven).md`.
+
+## The three decisions worth knowing
+
+**1. `Wants=`, not `Requires=`, from host → Core.** Measured on staging:
+`Requires` propagates a Core stop into a host stop, and a later Core start does
+*not* bring the host back — a routine Core restart would leave the client with
+no server. With `Wants`, the host survives a Core outage, answers
+`503 core_unavailable` (never a Python fallback), and resumes the moment Core
+returns, with no supervisor intervention.
+
+**2. Readiness is an `ExecStartPre` TCP gate, not ordering.** `After=`/`Wants=`
+order units; `Type=exec` only proves the binary exec'd. Neither means Core can
+serve. Core binds its listener *after* opening the DB, running migrations and
+loading the content pack, so "port open" is a genuine readiness signal. The gate
+is bounded and *fails* rather than blocking: a host that waits forever looks
+healthy to the supervisor while serving nobody.
+
+**3. The netns is resolved by container NAME at every start.** Production must
+run inside `openfut-fut-backend`'s network namespace. The container is
+`restart=unless-stopped`, and its netns inode *changes* on restart — observed
+`net:[4026539938] → net:[4026540033]`. A hardcoded pid is therefore wrong by
+construction, and any process left in the old namespace keeps running with no
+interfaces, silently serving nobody. `openfut-netns-bind.sh` re-resolves by name
+and refreshes a stale bind mount; `NetworkNamespacePath=` then enters it
+declaratively.
+
+> **Still open:** a container restart strands *already-running* services in the
+> dead namespace. Re-running `openfut-netns.service` plus restarting Core/host
+> repairs it, but nothing triggers that automatically yet. See the promotion
+> plan's "Residual gap".
+
+## Operating
+
+```bash
+systemctl status openfut-staging-core openfut-staging-host
+journalctl -u openfut-staging-core -f
+sudo systemctl restart openfut-staging-core          # host survives and recovers
+sudo systemctl stop openfut-staging-host openfut-staging-core
+```
+
+Config lives in `EnvironmentFile`s (`…/systemd/core.env`, `host.env`), generated
+from the live process environment so supervision changed *how* the processes
+start and nothing about *what* they do. Binaries are immutable copies, so a
+later `cargo build` cannot change what is running.
+
+## Interaction with the staging lifecycle script
+
+`scripts/sold-staging-up.py` still starts its own unsupervised processes and
+refuses to run while a staging stack is up. Stop the units first:
+
+```bash
+sudo systemctl stop openfut-staging-host openfut-staging-core
+python3 scripts/sold-staging-up.py --club real --variant highest …
+```
+
+Reconciling the two (having the script drive the units) is deliberately out of
+scope for the supervision milestone.
@@ -0,0 +1,51 @@
+[Unit]
+Description=OpenFUT Core (PRODUCTION) — authoritative economy state
+Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md
+# PRODUCTION TEMPLATE — NOT INSTALLED. Deploy only via the promotion plan in
+# `06 Operations/OpenFUT Service Supervision (staging-proven).md`.
+#
+# Core is the SINGLE WRITER of prod-core.db (verified by lsof: exactly one
+# process holds it open). Nothing here may be templated into a second instance.
+After=network-online.target docker.service openfut-netns.service
+Wants=network-online.target
+Requires=openfut-netns.service
+
+# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in
+# [Service] (`systemd-analyze verify` flags it), which would have left the
+# crash-loop ceiling at the 10s/5 default instead of the intended 60s window.
+StartLimitIntervalSec=60
+StartLimitBurst=5
+
+[Service]
+Type=exec
+# root, matching the current production processes exactly. Supervision changes
+# HOW the process is started, never what it is or what it can reach.
+User=root
+
+# The container's netns, published by openfut-netns.service. This replaces the
+# hand-typed `nsenter --net=/proc/<pid>/ns/net` in the runbook: same namespace,
+# no hardcoded pid, and re-resolved on every start.
+NetworkNamespacePath=/run/netns/openfut
+
+EnvironmentFile=/etc/openfut/core.env
+
+# An IMMUTABLE promotion artifact, not target/release. A later `cargo build`
+# must not be able to change what production is running — the same invariant
+# the promotion process already relies on.
+ExecStart=/home/alex/openfut-migration/promote-contract-20260822-184409/artifacts/openfut-core
+
+KillSignal=SIGTERM
+KillMode=mixed
+# Generous, so a WAL checkpoint is never SIGKILLed mid-write. Observed shutdown
+# is sub-second.
+TimeoutStopSec=30
+
+Restart=on-failure
+RestartSec=5s
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=openfut-core
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,55 @@
+[Unit]
+Description=OpenFUT FIFA17 UTAS host (PRODUCTION) — client-facing, Core-dependent
+Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md
+# PRODUCTION TEMPLATE — NOT INSTALLED.
+
+Requires=openfut-netns.service
+After=openfut-netns.service
+Wants=openfut-core.service
+After=openfut-core.service
+
+# `Wants` on Core, deliberately NOT `Requires`/`BindsTo`/`PartOf` — measured on
+# staging: those propagate a Core stop into a host stop, and a later Core start
+# does NOT bring the host back, so a routine Core restart would leave the client
+# with no server at all. With `Wants` the host survives a Core outage, answers
+# 503 `core_unavailable` (never a Python fallback), and resumes serving the
+# moment Core returns with no supervisor intervention. Both halves were proven
+# on the staging units.
+#
+# `Requires` on the netns unit IS correct: without the namespace the host would
+# bind the wrong network entirely.
+
+# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in
+# [Service] (`systemd-analyze verify` flags it), which would have left the
+# crash-loop ceiling at the 10s/5 default instead of the intended 60s window.
+StartLimitIntervalSec=60
+StartLimitBurst=5
+
+[Service]
+Type=exec
+User=root
+NetworkNamespacePath=/run/netns/openfut
+EnvironmentFile=/etc/openfut/host.env
+
+# Admission gate. Ordering alone proves nothing about readiness (Type=exec only
+# proves the binary exec'd). Core binds its port only after migrations and the
+# content pack have loaded, so "port open" is a real readiness signal here.
+# Bounded and FAILING rather than blocking: a host that waits forever looks
+# healthy to the supervisor while serving nobody.
+ExecStartPre=/home/alex/OpenFUT/scripts/systemd/openfut-wait-tcp.sh 127.0.0.1 18080 30
+
+ExecStart=/home/alex/openfut-migration/promote-contract-20260822-184409/artifacts/openfut-utas-host
+
+KillSignal=SIGTERM
+KillMode=mixed
+TimeoutStopSec=30
+
+Restart=on-failure
+RestartSec=5s
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=openfut-host
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,57 @@
+#!/bin/sh
+# Publish a Docker container's network namespace into /run/netns so systemd
+# units can enter it declaratively with NetworkNamespacePath=.
+#
+# WHY THIS EXISTS. Production Core and host must run inside the
+# `openfut-fut-backend` container's netns: that is where the published client
+# ports live and where the Python oracle answers on 127.0.0.1:8199. Today they
+# get there with `nsenter --net=/proc/<pid>/ns/net`, where <pid> is typed by
+# hand into a runbook.
+#
+# THE HAZARD THIS FIXES, MEASURED NOT ASSUMED. The container runs with
+# `restart=unless-stopped`. On a container restart its netns inode CHANGES
+# (observed 2026-08-22: net:[4026539938] -> net:[4026540033]). A hardcoded pid
+# is then simply wrong, and — worse — any process already inside the old
+# namespace keeps running in a namespace with no interfaces, silently serving
+# nobody. Resolving by container NAME at every start removes the hardcoded pid;
+# the companion watcher unit handles the already-running case by restarting the
+# stack when the container restarts.
+#
+# Idempotent: a stale bind mount is unmounted and re-made, so re-running after a
+# container restart is the fix, not a second problem.
+#
+# usage: openfut-netns-bind.sh <container-name> <netns-name>
+set -eu
+
+CONTAINER="${1:?container name}"
+NSNAME="${2:?netns name}"
+TARGET="/run/netns/${NSNAME}"
+
+CPID="$(docker inspect -f '{{.State.Pid}}' "$CONTAINER" 2>/dev/null || true)"
+if [ -z "$CPID" ] || [ "$CPID" = "0" ]; then
+    echo "openfut-netns-bind: container '$CONTAINER' is not running (pid='$CPID')" >&2
+    exit 1
+fi
+if [ ! -e "/proc/$CPID/ns/net" ]; then
+    echo "openfut-netns-bind: /proc/$CPID/ns/net does not exist" >&2
+    exit 1
+fi
+
+WANT="$(readlink "/proc/$CPID/ns/net")"
+mkdir -p /run/netns
+
+# Already published and already CURRENT? Then do nothing — re-mounting under a
+# live service would be gratuitous churn.
+if mountpoint -q "$TARGET" 2>/dev/null; then
+    HAVE="ns:[$(stat -c %i "$TARGET" 2>/dev/null || echo 0)]"
+    if [ "net:[$(stat -c %i "$TARGET" 2>/dev/null)]" = "$WANT" ]; then
+        echo "openfut-netns-bind: $TARGET already current ($WANT)"
+        exit 0
+    fi
+    echo "openfut-netns-bind: $TARGET is STALE ($HAVE, want $WANT) — refreshing"
+    umount "$TARGET" || true
+fi
+
+[ -e "$TARGET" ] || touch "$TARGET"
+mount --bind "/proc/$CPID/ns/net" "$TARGET"
+echo "openfut-netns-bind: $TARGET -> $CONTAINER pid $CPID $WANT"
@@ -0,0 +1,29 @@
+[Unit]
+Description=OpenFUT: publish the FIFA17 container network namespace to /run/netns
+Documentation=file:///home/alex/OpenFUT/scripts/systemd/openfut-netns-bind.sh
+# PRODUCTION TEMPLATE — NOT INSTALLED. Staging runs in the host netns and needs
+# none of this; only production enters the `openfut-fut-backend` container's
+# namespace.
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+
+# Re-resolves the container by NAME on every start, so no pid is ever hardcoded
+# and a container restart is repaired by restarting this unit. It exits non-zero
+# when the container is not running, which is what makes Core's Requires= on it
+# a real admission gate rather than decoration.
+ExecStart=/home/alex/OpenFUT/scripts/systemd/openfut-netns-bind.sh openfut-fut-backend openfut
+
+# Deliberately NO ExecStop unmount. Tearing the bind mount down while Core and
+# the host are still inside that namespace would strand them; the namespace is
+# owned by the container's lifetime, not by this unit's.
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=openfut-netns
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,50 @@
+[Unit]
+Description=OpenFUT Core (STAGING) — authoritative economy state
+Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md
+# Staging Core is the write authority for the staging SQLite DB. Nothing else
+# may hold it open, which is why there is no second instance and no oneshot
+# migration unit: Core runs its own migrations at startup, before it binds.
+After=network-online.target
+Wants=network-online.target
+
+# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in
+# [Service] (`systemd-analyze verify` flags it), which would have left the
+# crash-loop ceiling at the 10s/5 default instead of the intended 60s window.
+StartLimitIntervalSec=60
+StartLimitBurst=5
+
+[Service]
+Type=exec
+User=alex
+Group=alex
+WorkingDirectory=/home/alex/openfut-sold-staging
+
+# Config is DATA, not baked into the unit, so the same unit file promotes to
+# production with a different EnvironmentFile.
+EnvironmentFile=/home/alex/openfut-sold-staging/systemd/core.env
+
+# The binary is an IMMUTABLE COPY taken at staging-up time. A later `cargo
+# build` cannot silently change what staging is running — the same property the
+# promotion artifacts rely on.
+ExecStart=/home/alex/openfut-sold-staging/bin/openfut-core
+
+# Graceful stop. Both binaries already exit cleanly on SIGTERM (the
+# restart-persistence harness has always stopped them that way), and 20s is far
+# more than the observed sub-second shutdown; it exists so a slow WAL
+# checkpoint is never SIGKILLed mid-write.
+KillSignal=SIGTERM
+KillMode=mixed
+TimeoutStopSec=20
+
+# Restart WITHOUT a tight loop. `on-failure` deliberately excludes a clean
+# operator stop. 5s spacing with a 5-in-60s ceiling means a genuinely broken
+# build lands in `failed` where it is visible, instead of thrashing the DB.
+Restart=on-failure
+RestartSec=5s
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=openfut-staging-core
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,55 @@
+[Unit]
+Description=OpenFUT FIFA17 UTAS host (STAGING) — client-facing, Core-dependent
+Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md
+
+# ORDERING AND ADMISSION. `Wants` + `After` order the host after Core and pull
+# Core in when the host is started; neither gives READINESS, because Type=exec
+# only proves the binary exec'd. The ExecStartPre below is what actually admits
+# traffic: the host cannot reach "active" while Core is not listening.
+Wants=openfut-staging-core.service
+After=openfut-staging-core.service
+
+# `Wants`, deliberately NOT `Requires`/`BindsTo`/`PartOf`. Those propagate a
+# Core stop into a host stop, and — measured, not assumed — a later Core start
+# does NOT bring the host back, so a routine Core restart would silently leave
+# the client with no server at all.
+#
+# With `Wants` the host survives a Core outage and answers 503
+# `core_unavailable`, never falling back to Python. That is the behaviour
+# production already exhibited on 2026-08-22 when Core was SIGHUP'd out from
+# under a live host. Because the host holds no Core state between requests, it
+# resumes serving the moment Core returns, with no supervisor intervention.
+
+# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in
+# [Service] (`systemd-analyze verify` flags it), which would have left the
+# crash-loop ceiling at the 10s/5 default instead of the intended 60s window.
+StartLimitIntervalSec=60
+StartLimitBurst=5
+
+[Service]
+Type=exec
+User=alex
+Group=alex
+WorkingDirectory=/home/alex/openfut-sold-staging
+
+EnvironmentFile=/home/alex/openfut-sold-staging/systemd/host.env
+
+# Readiness gate. Bounded, and FAILS rather than blocking forever: a host that
+# waits indefinitely looks healthy to the supervisor while serving nothing.
+ExecStartPre=/home/alex/OpenFUT/scripts/systemd/openfut-wait-tcp.sh 127.0.0.1 18081 30
+
+ExecStart=/home/alex/openfut-sold-staging/bin/openfut-utas-host
+
+KillSignal=SIGTERM
+KillMode=mixed
+TimeoutStopSec=20
+
+Restart=on-failure
+RestartSec=5s
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=openfut-staging-host
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,42 @@
+#!/bin/sh
+# Bounded wait for a TCP listener. Exits 0 as soon as it connects, 1 on timeout.
+#
+# Used as the FIFA17 host's ExecStartPre so the host cannot start "successfully"
+# while Core is absent. A TCP connect is a MEANINGFUL readiness signal here, not
+# a guess: openfut-core binds its listener only after it has opened the
+# database, run migrations and loaded the content pack — the startup log order
+# is `Running database migrations` -> `Loaded N cards` -> `Listening`, and the
+# 2026-08-22 promotion demonstrated it directly when a bind collision aborted a
+# Core that had already applied migration 0028. So "port open" implies
+# "migrations done, content loaded".
+#
+# Deliberately no retry-forever: a host that blocks indefinitely looks healthy
+# to a supervisor while serving nothing. It fails, and the restart policy
+# decides what happens next.
+#
+# usage: openfut-wait-tcp.sh <host> <port> <timeout-seconds>
+set -eu
+
+HOST="${1:?host}"
+PORT="${2:?port}"
+TIMEOUT="${3:-30}"
+
+i=0
+while [ "$i" -lt "$TIMEOUT" ]; do
+    # /dev/tcp is a bash-ism; use a portable connect instead.
+    if command -v nc >/dev/null 2>&1; then
+        nc -z "$HOST" "$PORT" 2>/dev/null && exit 0
+    else
+        python3 - "$HOST" "$PORT" <<'PY' && exit 0
+import socket, sys
+s = socket.socket()
+s.settimeout(1)
+sys.exit(0 if s.connect_ex((sys.argv[1], int(sys.argv[2]))) == 0 else 1)
+PY
+    fi
+    i=$((i + 1))
+    sleep 1
+done
+
+echo "openfut-wait-tcp: $HOST:$PORT did not accept connections within ${TIMEOUT}s" >&2
+exit 1