funman300
8ae432223a
Replaces the detached `setsid nohup … nsenter …` launch, which had no restart policy, no boot persistence and no supervisor-visible logs. Staging units are installed and proven; production units are TEMPLATES and are not installed. Three decisions, each measured rather than assumed: * `Wants=`, not `Requires=`, from host to Core. With `Requires`, stopping Core stopped the host AND a later Core start did not bring it back -- a routine Core restart would leave the client with no server. With `Wants` the host survives a Core outage, answers 503 core_unavailable, never falls back to Python, and resumes the moment Core returns with no intervention. Both halves tested. * Readiness is a bounded ExecStartPre TCP gate, because ordering proves nothing about readiness and Type=exec only proves the binary exec'd. Core binds its listener after migrations and content load, so "port open" is a real signal. The gate FAILS rather than blocking: a host that waits forever looks healthy while serving nobody. * The netns is resolved by container NAME every start. The container is restart=unless-stopped and its netns inode CHANGES on restart (measured: 4026539938 -> 4026540033), so a hardcoded pid is wrong by construction and anything left in the old namespace serves nobody. Proven equivalent to today's nsenter against a scratch container, never production's namespace. `systemd-analyze verify` caught two real defects before deployment: StartLimitIntervalSec/StartLimitBurst sat in [Service], where systemd 252 silently ignores them, so the crash-loop ceiling was not taking effect; and a Documentation URL containing %20 parsed as a specifier. Both fixed and the effective properties re-confirmed from the running units. Staging evidence: Core-first ordering, host refused when Core is absent or merely not listening, outage survival, automatic recovery, restart, graceful stop with no strays, boot simulated via multi-user.target, 3x SIGKILL contained at ~5s spacing, journald logs, and economy state byte-identical throughout (integrity ok, fk 0).
76 lines
3.6 KiB
Markdown
76 lines
3.6 KiB
Markdown
# OpenFUT systemd units
|
|
|
|
Supervision for OpenFUT Core and the FIFA17 UTAS host. Replaces the previous
|
|
`setsid nohup … nsenter …` launch, which had no restart policy, no boot
|
|
persistence and no supervisor-visible logs.
|
|
|
|
| file | scope | installed? |
|
|
|---|---|---|
|
|
| `openfut-staging-core.service` | staging Core, port 18081 | **yes** — proving ground |
|
|
| `openfut-staging-host.service` | staging host, port 8299 | **yes** |
|
|
| `openfut-netns.service` | publishes the container netns to `/run/netns/openfut` | template only |
|
|
| `openfut-core.service` | production Core, port 18080 | template only |
|
|
| `openfut-host.service` | production host, port 8099 | template only |
|
|
| `openfut-netns-bind.sh` | resolves the container netns by NAME, idempotently | helper |
|
|
| `openfut-wait-tcp.sh` | bounded readiness gate | helper |
|
|
|
|
Production templates are **not installed**. Deploy only via the plan in
|
|
`OpenFUT-Vault/06 Operations/OpenFUT Service Supervision (staging-proven).md`.
|
|
|
|
## The three decisions worth knowing
|
|
|
|
**1. `Wants=`, not `Requires=`, from host → Core.** Measured on staging:
|
|
`Requires` propagates a Core stop into a host stop, and a later Core start does
|
|
*not* bring the host back — a routine Core restart would leave the client with
|
|
no server. With `Wants`, the host survives a Core outage, answers
|
|
`503 core_unavailable` (never a Python fallback), and resumes the moment Core
|
|
returns, with no supervisor intervention.
|
|
|
|
**2. Readiness is an `ExecStartPre` TCP gate, not ordering.** `After=`/`Wants=`
|
|
order units; `Type=exec` only proves the binary exec'd. Neither means Core can
|
|
serve. Core binds its listener *after* opening the DB, running migrations and
|
|
loading the content pack, so "port open" is a genuine readiness signal. The gate
|
|
is bounded and *fails* rather than blocking: a host that waits forever looks
|
|
healthy to the supervisor while serving nobody.
|
|
|
|
**3. The netns is resolved by container NAME at every start.** Production must
|
|
run inside `openfut-fut-backend`'s network namespace. The container is
|
|
`restart=unless-stopped`, and its netns inode *changes* on restart — observed
|
|
`net:[4026539938] → net:[4026540033]`. A hardcoded pid is therefore wrong by
|
|
construction, and any process left in the old namespace keeps running with no
|
|
interfaces, silently serving nobody. `openfut-netns-bind.sh` re-resolves by name
|
|
and refreshes a stale bind mount; `NetworkNamespacePath=` then enters it
|
|
declaratively.
|
|
|
|
> **Still open:** a container restart strands *already-running* services in the
|
|
> dead namespace. Re-running `openfut-netns.service` plus restarting Core/host
|
|
> repairs it, but nothing triggers that automatically yet. See the promotion
|
|
> plan's "Residual gap".
|
|
|
|
## Operating
|
|
|
|
```bash
|
|
systemctl status openfut-staging-core openfut-staging-host
|
|
journalctl -u openfut-staging-core -f
|
|
sudo systemctl restart openfut-staging-core # host survives and recovers
|
|
sudo systemctl stop openfut-staging-host openfut-staging-core
|
|
```
|
|
|
|
Config lives in `EnvironmentFile`s (`…/systemd/core.env`, `host.env`), generated
|
|
from the live process environment so supervision changed *how* the processes
|
|
start and nothing about *what* they do. Binaries are immutable copies, so a
|
|
later `cargo build` cannot change what is running.
|
|
|
|
## Interaction with the staging lifecycle script
|
|
|
|
`scripts/sold-staging-up.py` still starts its own unsupervised processes and
|
|
refuses to run while a staging stack is up. Stop the units first:
|
|
|
|
```bash
|
|
sudo systemctl stop openfut-staging-host openfut-staging-core
|
|
python3 scripts/sold-staging-up.py --club real --variant highest …
|
|
```
|
|
|
|
Reconciling the two (having the script drive the units) is deliberately out of
|
|
scope for the supervision milestone.
|