From 8ae432223a4e9f6a32048ca62d29a8070fb879f9 Mon Sep 17 00:00:00 2001 From: funman300 Date: Sat, 22 Aug 2026 20:46:15 +0000 Subject: [PATCH] ops: systemd supervision for Core and the FIFA17 host (staging-proven) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the detached `setsid nohup … nsenter …` launch, which had no restart policy, no boot persistence and no supervisor-visible logs. Staging units are installed and proven; production units are TEMPLATES and are not installed. Three decisions, each measured rather than assumed: * `Wants=`, not `Requires=`, from host to Core. With `Requires`, stopping Core stopped the host AND a later Core start did not bring it back -- a routine Core restart would leave the client with no server. With `Wants` the host survives a Core outage, answers 503 core_unavailable, never falls back to Python, and resumes the moment Core returns with no intervention. Both halves tested. * Readiness is a bounded ExecStartPre TCP gate, because ordering proves nothing about readiness and Type=exec only proves the binary exec'd. Core binds its listener after migrations and content load, so "port open" is a real signal. The gate FAILS rather than blocking: a host that waits forever looks healthy while serving nobody. * The netns is resolved by container NAME every start. The container is restart=unless-stopped and its netns inode CHANGES on restart (measured: 4026539938 -> 4026540033), so a hardcoded pid is wrong by construction and anything left in the old namespace serves nobody. Proven equivalent to today's nsenter against a scratch container, never production's namespace. `systemd-analyze verify` caught two real defects before deployment: StartLimitIntervalSec/StartLimitBurst sat in [Service], where systemd 252 silently ignores them, so the crash-loop ceiling was not taking effect; and a Documentation URL containing %20 parsed as a specifier. Both fixed and the effective properties re-confirmed from the running units. Staging evidence: Core-first ordering, host refused when Core is absent or merely not listening, outage survival, automatic recovery, restart, graceful stop with no strays, boot simulated via multi-user.target, 3x SIGKILL contained at ~5s spacing, journald logs, and economy state byte-identical throughout (integrity ok, fk 0). --- scripts/systemd/README.md | 75 ++++++++++++++++++++ scripts/systemd/openfut-core.service | 51 +++++++++++++ scripts/systemd/openfut-host.service | 55 ++++++++++++++ scripts/systemd/openfut-netns-bind.sh | 57 +++++++++++++++ scripts/systemd/openfut-netns.service | 29 ++++++++ scripts/systemd/openfut-staging-core.service | 50 +++++++++++++ scripts/systemd/openfut-staging-host.service | 55 ++++++++++++++ scripts/systemd/openfut-wait-tcp.sh | 42 +++++++++++ 8 files changed, 414 insertions(+) create mode 100644 scripts/systemd/README.md create mode 100644 scripts/systemd/openfut-core.service create mode 100644 scripts/systemd/openfut-host.service create mode 100755 scripts/systemd/openfut-netns-bind.sh create mode 100644 scripts/systemd/openfut-netns.service create mode 100644 scripts/systemd/openfut-staging-core.service create mode 100644 scripts/systemd/openfut-staging-host.service create mode 100755 scripts/systemd/openfut-wait-tcp.sh diff --git a/scripts/systemd/README.md b/scripts/systemd/README.md new file mode 100644 index 0000000..3339f1b --- /dev/null +++ b/scripts/systemd/README.md @@ -0,0 +1,75 @@ +# OpenFUT systemd units + +Supervision for OpenFUT Core and the FIFA17 UTAS host. Replaces the previous +`setsid nohup … nsenter …` launch, which had no restart policy, no boot +persistence and no supervisor-visible logs. + +| file | scope | installed? | +|---|---|---| +| `openfut-staging-core.service` | staging Core, port 18081 | **yes** — proving ground | +| `openfut-staging-host.service` | staging host, port 8299 | **yes** | +| `openfut-netns.service` | publishes the container netns to `/run/netns/openfut` | template only | +| `openfut-core.service` | production Core, port 18080 | template only | +| `openfut-host.service` | production host, port 8099 | template only | +| `openfut-netns-bind.sh` | resolves the container netns by NAME, idempotently | helper | +| `openfut-wait-tcp.sh` | bounded readiness gate | helper | + +Production templates are **not installed**. Deploy only via the plan in +`OpenFUT-Vault/06 Operations/OpenFUT Service Supervision (staging-proven).md`. + +## The three decisions worth knowing + +**1. `Wants=`, not `Requires=`, from host → Core.** Measured on staging: +`Requires` propagates a Core stop into a host stop, and a later Core start does +*not* bring the host back — a routine Core restart would leave the client with +no server. With `Wants`, the host survives a Core outage, answers +`503 core_unavailable` (never a Python fallback), and resumes the moment Core +returns, with no supervisor intervention. + +**2. Readiness is an `ExecStartPre` TCP gate, not ordering.** `After=`/`Wants=` +order units; `Type=exec` only proves the binary exec'd. Neither means Core can +serve. Core binds its listener *after* opening the DB, running migrations and +loading the content pack, so "port open" is a genuine readiness signal. The gate +is bounded and *fails* rather than blocking: a host that waits forever looks +healthy to the supervisor while serving nobody. + +**3. The netns is resolved by container NAME at every start.** Production must +run inside `openfut-fut-backend`'s network namespace. The container is +`restart=unless-stopped`, and its netns inode *changes* on restart — observed +`net:[4026539938] → net:[4026540033]`. A hardcoded pid is therefore wrong by +construction, and any process left in the old namespace keeps running with no +interfaces, silently serving nobody. `openfut-netns-bind.sh` re-resolves by name +and refreshes a stale bind mount; `NetworkNamespacePath=` then enters it +declaratively. + +> **Still open:** a container restart strands *already-running* services in the +> dead namespace. Re-running `openfut-netns.service` plus restarting Core/host +> repairs it, but nothing triggers that automatically yet. See the promotion +> plan's "Residual gap". + +## Operating + +```bash +systemctl status openfut-staging-core openfut-staging-host +journalctl -u openfut-staging-core -f +sudo systemctl restart openfut-staging-core # host survives and recovers +sudo systemctl stop openfut-staging-host openfut-staging-core +``` + +Config lives in `EnvironmentFile`s (`…/systemd/core.env`, `host.env`), generated +from the live process environment so supervision changed *how* the processes +start and nothing about *what* they do. Binaries are immutable copies, so a +later `cargo build` cannot change what is running. + +## Interaction with the staging lifecycle script + +`scripts/sold-staging-up.py` still starts its own unsupervised processes and +refuses to run while a staging stack is up. Stop the units first: + +```bash +sudo systemctl stop openfut-staging-host openfut-staging-core +python3 scripts/sold-staging-up.py --club real --variant highest … +``` + +Reconciling the two (having the script drive the units) is deliberately out of +scope for the supervision milestone. diff --git a/scripts/systemd/openfut-core.service b/scripts/systemd/openfut-core.service new file mode 100644 index 0000000..873b7c5 --- /dev/null +++ b/scripts/systemd/openfut-core.service @@ -0,0 +1,51 @@ +[Unit] +Description=OpenFUT Core (PRODUCTION) — authoritative economy state +Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md +# PRODUCTION TEMPLATE — NOT INSTALLED. Deploy only via the promotion plan in +# `06 Operations/OpenFUT Service Supervision (staging-proven).md`. +# +# Core is the SINGLE WRITER of prod-core.db (verified by lsof: exactly one +# process holds it open). Nothing here may be templated into a second instance. +After=network-online.target docker.service openfut-netns.service +Wants=network-online.target +Requires=openfut-netns.service + +# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in +# [Service] (`systemd-analyze verify` flags it), which would have left the +# crash-loop ceiling at the 10s/5 default instead of the intended 60s window. +StartLimitIntervalSec=60 +StartLimitBurst=5 + +[Service] +Type=exec +# root, matching the current production processes exactly. Supervision changes +# HOW the process is started, never what it is or what it can reach. +User=root + +# The container's netns, published by openfut-netns.service. This replaces the +# hand-typed `nsenter --net=/proc//ns/net` in the runbook: same namespace, +# no hardcoded pid, and re-resolved on every start. +NetworkNamespacePath=/run/netns/openfut + +EnvironmentFile=/etc/openfut/core.env + +# An IMMUTABLE promotion artifact, not target/release. A later `cargo build` +# must not be able to change what production is running — the same invariant +# the promotion process already relies on. +ExecStart=/home/alex/openfut-migration/promote-contract-20260822-184409/artifacts/openfut-core + +KillSignal=SIGTERM +KillMode=mixed +# Generous, so a WAL checkpoint is never SIGKILLed mid-write. Observed shutdown +# is sub-second. +TimeoutStopSec=30 + +Restart=on-failure +RestartSec=5s + +StandardOutput=journal +StandardError=journal +SyslogIdentifier=openfut-core + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/openfut-host.service b/scripts/systemd/openfut-host.service new file mode 100644 index 0000000..6b28fd2 --- /dev/null +++ b/scripts/systemd/openfut-host.service @@ -0,0 +1,55 @@ +[Unit] +Description=OpenFUT FIFA17 UTAS host (PRODUCTION) — client-facing, Core-dependent +Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md +# PRODUCTION TEMPLATE — NOT INSTALLED. + +Requires=openfut-netns.service +After=openfut-netns.service +Wants=openfut-core.service +After=openfut-core.service + +# `Wants` on Core, deliberately NOT `Requires`/`BindsTo`/`PartOf` — measured on +# staging: those propagate a Core stop into a host stop, and a later Core start +# does NOT bring the host back, so a routine Core restart would leave the client +# with no server at all. With `Wants` the host survives a Core outage, answers +# 503 `core_unavailable` (never a Python fallback), and resumes serving the +# moment Core returns with no supervisor intervention. Both halves were proven +# on the staging units. +# +# `Requires` on the netns unit IS correct: without the namespace the host would +# bind the wrong network entirely. + +# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in +# [Service] (`systemd-analyze verify` flags it), which would have left the +# crash-loop ceiling at the 10s/5 default instead of the intended 60s window. +StartLimitIntervalSec=60 +StartLimitBurst=5 + +[Service] +Type=exec +User=root +NetworkNamespacePath=/run/netns/openfut +EnvironmentFile=/etc/openfut/host.env + +# Admission gate. Ordering alone proves nothing about readiness (Type=exec only +# proves the binary exec'd). Core binds its port only after migrations and the +# content pack have loaded, so "port open" is a real readiness signal here. +# Bounded and FAILING rather than blocking: a host that waits forever looks +# healthy to the supervisor while serving nobody. +ExecStartPre=/home/alex/OpenFUT/scripts/systemd/openfut-wait-tcp.sh 127.0.0.1 18080 30 + +ExecStart=/home/alex/openfut-migration/promote-contract-20260822-184409/artifacts/openfut-utas-host + +KillSignal=SIGTERM +KillMode=mixed +TimeoutStopSec=30 + +Restart=on-failure +RestartSec=5s + +StandardOutput=journal +StandardError=journal +SyslogIdentifier=openfut-host + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/openfut-netns-bind.sh b/scripts/systemd/openfut-netns-bind.sh new file mode 100755 index 0000000..4b93cb3 --- /dev/null +++ b/scripts/systemd/openfut-netns-bind.sh @@ -0,0 +1,57 @@ +#!/bin/sh +# Publish a Docker container's network namespace into /run/netns so systemd +# units can enter it declaratively with NetworkNamespacePath=. +# +# WHY THIS EXISTS. Production Core and host must run inside the +# `openfut-fut-backend` container's netns: that is where the published client +# ports live and where the Python oracle answers on 127.0.0.1:8199. Today they +# get there with `nsenter --net=/proc//ns/net`, where is typed by +# hand into a runbook. +# +# THE HAZARD THIS FIXES, MEASURED NOT ASSUMED. The container runs with +# `restart=unless-stopped`. On a container restart its netns inode CHANGES +# (observed 2026-08-22: net:[4026539938] -> net:[4026540033]). A hardcoded pid +# is then simply wrong, and — worse — any process already inside the old +# namespace keeps running in a namespace with no interfaces, silently serving +# nobody. Resolving by container NAME at every start removes the hardcoded pid; +# the companion watcher unit handles the already-running case by restarting the +# stack when the container restarts. +# +# Idempotent: a stale bind mount is unmounted and re-made, so re-running after a +# container restart is the fix, not a second problem. +# +# usage: openfut-netns-bind.sh +set -eu + +CONTAINER="${1:?container name}" +NSNAME="${2:?netns name}" +TARGET="/run/netns/${NSNAME}" + +CPID="$(docker inspect -f '{{.State.Pid}}' "$CONTAINER" 2>/dev/null || true)" +if [ -z "$CPID" ] || [ "$CPID" = "0" ]; then + echo "openfut-netns-bind: container '$CONTAINER' is not running (pid='$CPID')" >&2 + exit 1 +fi +if [ ! -e "/proc/$CPID/ns/net" ]; then + echo "openfut-netns-bind: /proc/$CPID/ns/net does not exist" >&2 + exit 1 +fi + +WANT="$(readlink "/proc/$CPID/ns/net")" +mkdir -p /run/netns + +# Already published and already CURRENT? Then do nothing — re-mounting under a +# live service would be gratuitous churn. +if mountpoint -q "$TARGET" 2>/dev/null; then + HAVE="ns:[$(stat -c %i "$TARGET" 2>/dev/null || echo 0)]" + if [ "net:[$(stat -c %i "$TARGET" 2>/dev/null)]" = "$WANT" ]; then + echo "openfut-netns-bind: $TARGET already current ($WANT)" + exit 0 + fi + echo "openfut-netns-bind: $TARGET is STALE ($HAVE, want $WANT) — refreshing" + umount "$TARGET" || true +fi + +[ -e "$TARGET" ] || touch "$TARGET" +mount --bind "/proc/$CPID/ns/net" "$TARGET" +echo "openfut-netns-bind: $TARGET -> $CONTAINER pid $CPID $WANT" diff --git a/scripts/systemd/openfut-netns.service b/scripts/systemd/openfut-netns.service new file mode 100644 index 0000000..9986c22 --- /dev/null +++ b/scripts/systemd/openfut-netns.service @@ -0,0 +1,29 @@ +[Unit] +Description=OpenFUT: publish the FIFA17 container network namespace to /run/netns +Documentation=file:///home/alex/OpenFUT/scripts/systemd/openfut-netns-bind.sh +# PRODUCTION TEMPLATE — NOT INSTALLED. Staging runs in the host netns and needs +# none of this; only production enters the `openfut-fut-backend` container's +# namespace. +After=docker.service +Requires=docker.service + +[Service] +Type=oneshot +RemainAfterExit=yes + +# Re-resolves the container by NAME on every start, so no pid is ever hardcoded +# and a container restart is repaired by restarting this unit. It exits non-zero +# when the container is not running, which is what makes Core's Requires= on it +# a real admission gate rather than decoration. +ExecStart=/home/alex/OpenFUT/scripts/systemd/openfut-netns-bind.sh openfut-fut-backend openfut + +# Deliberately NO ExecStop unmount. Tearing the bind mount down while Core and +# the host are still inside that namespace would strand them; the namespace is +# owned by the container's lifetime, not by this unit's. + +StandardOutput=journal +StandardError=journal +SyslogIdentifier=openfut-netns + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/openfut-staging-core.service b/scripts/systemd/openfut-staging-core.service new file mode 100644 index 0000000..0bd6713 --- /dev/null +++ b/scripts/systemd/openfut-staging-core.service @@ -0,0 +1,50 @@ +[Unit] +Description=OpenFUT Core (STAGING) — authoritative economy state +Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md +# Staging Core is the write authority for the staging SQLite DB. Nothing else +# may hold it open, which is why there is no second instance and no oneshot +# migration unit: Core runs its own migrations at startup, before it binds. +After=network-online.target +Wants=network-online.target + +# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in +# [Service] (`systemd-analyze verify` flags it), which would have left the +# crash-loop ceiling at the 10s/5 default instead of the intended 60s window. +StartLimitIntervalSec=60 +StartLimitBurst=5 + +[Service] +Type=exec +User=alex +Group=alex +WorkingDirectory=/home/alex/openfut-sold-staging + +# Config is DATA, not baked into the unit, so the same unit file promotes to +# production with a different EnvironmentFile. +EnvironmentFile=/home/alex/openfut-sold-staging/systemd/core.env + +# The binary is an IMMUTABLE COPY taken at staging-up time. A later `cargo +# build` cannot silently change what staging is running — the same property the +# promotion artifacts rely on. +ExecStart=/home/alex/openfut-sold-staging/bin/openfut-core + +# Graceful stop. Both binaries already exit cleanly on SIGTERM (the +# restart-persistence harness has always stopped them that way), and 20s is far +# more than the observed sub-second shutdown; it exists so a slow WAL +# checkpoint is never SIGKILLed mid-write. +KillSignal=SIGTERM +KillMode=mixed +TimeoutStopSec=20 + +# Restart WITHOUT a tight loop. `on-failure` deliberately excludes a clean +# operator stop. 5s spacing with a 5-in-60s ceiling means a genuinely broken +# build lands in `failed` where it is visible, instead of thrashing the DB. +Restart=on-failure +RestartSec=5s + +StandardOutput=journal +StandardError=journal +SyslogIdentifier=openfut-staging-core + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/openfut-staging-host.service b/scripts/systemd/openfut-staging-host.service new file mode 100644 index 0000000..373d000 --- /dev/null +++ b/scripts/systemd/openfut-staging-host.service @@ -0,0 +1,55 @@ +[Unit] +Description=OpenFUT FIFA17 UTAS host (STAGING) — client-facing, Core-dependent +Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md + +# ORDERING AND ADMISSION. `Wants` + `After` order the host after Core and pull +# Core in when the host is started; neither gives READINESS, because Type=exec +# only proves the binary exec'd. The ExecStartPre below is what actually admits +# traffic: the host cannot reach "active" while Core is not listening. +Wants=openfut-staging-core.service +After=openfut-staging-core.service + +# `Wants`, deliberately NOT `Requires`/`BindsTo`/`PartOf`. Those propagate a +# Core stop into a host stop, and — measured, not assumed — a later Core start +# does NOT bring the host back, so a routine Core restart would silently leave +# the client with no server at all. +# +# With `Wants` the host survives a Core outage and answers 503 +# `core_unavailable`, never falling back to Python. That is the behaviour +# production already exhibited on 2026-08-22 when Core was SIGHUP'd out from +# under a live host. Because the host holds no Core state between requests, it +# resumes serving the moment Core returns, with no supervisor intervention. + +# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in +# [Service] (`systemd-analyze verify` flags it), which would have left the +# crash-loop ceiling at the 10s/5 default instead of the intended 60s window. +StartLimitIntervalSec=60 +StartLimitBurst=5 + +[Service] +Type=exec +User=alex +Group=alex +WorkingDirectory=/home/alex/openfut-sold-staging + +EnvironmentFile=/home/alex/openfut-sold-staging/systemd/host.env + +# Readiness gate. Bounded, and FAILS rather than blocking forever: a host that +# waits indefinitely looks healthy to the supervisor while serving nothing. +ExecStartPre=/home/alex/OpenFUT/scripts/systemd/openfut-wait-tcp.sh 127.0.0.1 18081 30 + +ExecStart=/home/alex/openfut-sold-staging/bin/openfut-utas-host + +KillSignal=SIGTERM +KillMode=mixed +TimeoutStopSec=20 + +Restart=on-failure +RestartSec=5s + +StandardOutput=journal +StandardError=journal +SyslogIdentifier=openfut-staging-host + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/openfut-wait-tcp.sh b/scripts/systemd/openfut-wait-tcp.sh new file mode 100755 index 0000000..e4480b5 --- /dev/null +++ b/scripts/systemd/openfut-wait-tcp.sh @@ -0,0 +1,42 @@ +#!/bin/sh +# Bounded wait for a TCP listener. Exits 0 as soon as it connects, 1 on timeout. +# +# Used as the FIFA17 host's ExecStartPre so the host cannot start "successfully" +# while Core is absent. A TCP connect is a MEANINGFUL readiness signal here, not +# a guess: openfut-core binds its listener only after it has opened the +# database, run migrations and loaded the content pack — the startup log order +# is `Running database migrations` -> `Loaded N cards` -> `Listening`, and the +# 2026-08-22 promotion demonstrated it directly when a bind collision aborted a +# Core that had already applied migration 0028. So "port open" implies +# "migrations done, content loaded". +# +# Deliberately no retry-forever: a host that blocks indefinitely looks healthy +# to a supervisor while serving nothing. It fails, and the restart policy +# decides what happens next. +# +# usage: openfut-wait-tcp.sh +set -eu + +HOST="${1:?host}" +PORT="${2:?port}" +TIMEOUT="${3:-30}" + +i=0 +while [ "$i" -lt "$TIMEOUT" ]; do + # /dev/tcp is a bash-ism; use a portable connect instead. + if command -v nc >/dev/null 2>&1; then + nc -z "$HOST" "$PORT" 2>/dev/null && exit 0 + else + python3 - "$HOST" "$PORT" <<'PY' && exit 0 +import socket, sys +s = socket.socket() +s.settimeout(1) +sys.exit(0 if s.connect_ex((sys.argv[1], int(sys.argv[2]))) == 0 else 1) +PY + fi + i=$((i + 1)) + sleep 1 +done + +echo "openfut-wait-tcp: $HOST:$PORT did not accept connections within ${TIMEOUT}s" >&2 +exit 1