funman300
1e8d46b258
Recreating the Docker anchor left the supervised Core and host stranded in the dead namespace while systemd still reported them active — serving nobody, invisible to any monitoring that trusts unit state. Reproduced on staging: netns 4026539938 -> 4026540033, both pids unchanged in the old one, both units "active", traffic ConnectionResetError. There is no systemd-native edge signal to bind to. Containers do appear as units, but the scope name embeds the container ID (docker-<id>.scope), which changes on every recreate, so BindsTo= has no stable target; NetworkNamespacePath= resolves once at start; a .path unit on /run/netns would watch the file this tooling maintains. So: a level-triggered reconcile on a 10s timer, comparing the namespace the services are ACTUALLY in against the anchor's CURRENT one, acting only on a real difference. That cannot miss an event while the watcher restarts or dockerd is down, and needs no debounce — a burst of three recreations produced exactly one rebind. The trigger stays separable: a docker-events unit could invoke the same script. Anchor absent stops the dependants rather than falling back to host networking; docker unavailable logs once and retries on the next tick. Two defects found while testing and fixed here: - mount --bind STACKS when the old mount is busy, silently leaking nsfs entries; the bind helper now drains stale mounts in a loop. - reconcile must stop -> rebind -> start, not rebind -> restart: a running service holds the old namespace open and makes the umount fail busy. Staging also gained a faithful anchor container so the reproduction is structural rather than mocked. Economy state was byte-identical across every lifecycle test. Production units are templates only and remain uninstalled.
93 lines
5.0 KiB
Markdown
93 lines
5.0 KiB
Markdown
# OpenFUT systemd units
|
|
|
|
Supervision for OpenFUT Core and the FIFA17 UTAS host. Replaces the previous
|
|
`setsid nohup … nsenter …` launch, which had no restart policy, no boot
|
|
persistence and no supervisor-visible logs.
|
|
|
|
| file | scope | installed? |
|
|
|---|---|---|
|
|
| `openfut-staging-netns.service` | binds the staging anchor netns | **yes** — proving ground |
|
|
| `openfut-staging-core.service` | staging Core, port 18081 | **yes** |
|
|
| `openfut-staging-host.service` | staging host, port 8299 | **yes** |
|
|
| `openfut-staging-netns-reconcile.{service,timer}` | staging netns lifecycle | **yes** |
|
|
| `openfut-netns.service` | binds the production anchor netns to `/run/netns/openfut` | template only |
|
|
| `openfut-core.service` | production Core, port 18080 | template only |
|
|
| `openfut-host.service` | production host, port 8099 | template only |
|
|
| `openfut-netns-reconcile.{service,timer}` | production netns lifecycle | template only |
|
|
| `openfut-netns-bind.sh` | resolves the anchor netns by NAME, idempotently, drains stale mounts | helper |
|
|
| `openfut-netns-reconcile.sh` | keeps services in the anchor's CURRENT netns | helper |
|
|
| `openfut-wait-tcp.sh` | bounded readiness gate | helper |
|
|
| `openfut-supervision-install.sh` | install / start / status / uninstall per environment | helper |
|
|
|
|
Production templates are **not installed**. Deploy only via the plan in
|
|
`OpenFUT-Vault/06 Operations/OpenFUT Service Supervision (staging-proven).md`.
|
|
|
|
## The three decisions worth knowing
|
|
|
|
**1. `Wants=`, not `Requires=`, from host → Core.** Measured on staging:
|
|
`Requires` propagates a Core stop into a host stop, and a later Core start does
|
|
*not* bring the host back — a routine Core restart would leave the client with
|
|
no server. With `Wants`, the host survives a Core outage, answers
|
|
`503 core_unavailable` (never a Python fallback), and resumes the moment Core
|
|
returns, with no supervisor intervention.
|
|
|
|
**2. Readiness is an `ExecStartPre` TCP gate, not ordering.** `After=`/`Wants=`
|
|
order units; `Type=exec` only proves the binary exec'd. Neither means Core can
|
|
serve. Core binds its listener *after* opening the DB, running migrations and
|
|
loading the content pack, so "port open" is a genuine readiness signal. The gate
|
|
is bounded and *fails* rather than blocking: a host that waits forever looks
|
|
healthy to the supervisor while serving nobody.
|
|
|
|
**3. The netns is resolved by container NAME at every start.** Production must
|
|
run inside `openfut-fut-backend`'s network namespace. The container is
|
|
`restart=unless-stopped`, and its netns inode *changes* on restart — observed
|
|
`net:[4026539938] → net:[4026540033]`. A hardcoded pid is therefore wrong by
|
|
construction, and any process left in the old namespace keeps running with no
|
|
interfaces, silently serving nobody. `openfut-netns-bind.sh` re-resolves by name
|
|
and refreshes a stale bind mount; `NetworkNamespacePath=` then enters it
|
|
declaratively.
|
|
|
|
**4. Stale namespaces are repaired automatically (this closes the old gap).**
|
|
Recreating the anchor strands already-running services in the dead namespace,
|
|
and — the dangerous part — *systemd still reports them `active`*. Measured before
|
|
the fix: anchor `net:[4026539938] → net:[4026540033]`, Core and host unchanged in
|
|
the old one, both units `active`, traffic `ConnectionResetError`.
|
|
|
|
`openfut-netns-reconcile.sh` on a 10s timer compares the namespace the services
|
|
are **actually in** against the anchor's **current** one and, only on a real
|
|
difference, performs one stop → rebind → start cycle. It is level-triggered, so
|
|
it cannot miss an event and needs no debounce: a burst of three back-to-back
|
|
recreations produced exactly **one** rebind.
|
|
|
|
## Operating
|
|
|
|
```bash
|
|
sudo ./openfut-supervision-install.sh status staging # units, anchor, netns agreement, mounts
|
|
journalctl -u openfut-staging-netns-reconcile -f # silent unless it acts
|
|
sudo systemctl restart openfut-staging-core # host survives and recovers
|
|
sudo ./openfut-supervision-install.sh install staging # copy + enable (does not start)
|
|
sudo ./openfut-supervision-install.sh uninstall staging # disable, stop, remove units
|
|
```
|
|
|
|
`status` is the one command worth knowing: it prints each unit's state, the
|
|
anchor's current netns, the namespace each service is *actually* in with an
|
|
`ok`/`MISMATCH` verdict, and the mount count (`1` healthy, `>1` a leaked stack).
|
|
|
|
Config lives in `EnvironmentFile`s (`…/systemd/core.env`, `host.env`), generated
|
|
from the live process environment so supervision changed *how* the processes
|
|
start and nothing about *what* they do. Binaries are immutable copies, so a
|
|
later `cargo build` cannot change what is running.
|
|
|
|
## Interaction with the staging lifecycle script
|
|
|
|
`scripts/sold-staging-up.py` still starts its own unsupervised processes and
|
|
refuses to run while a staging stack is up. Stop the units first:
|
|
|
|
```bash
|
|
sudo systemctl stop openfut-staging-host openfut-staging-core
|
|
python3 scripts/sold-staging-up.py --club real --variant highest …
|
|
```
|
|
|
|
Reconciling the two (having the script drive the units) is deliberately out of
|
|
scope for the supervision milestone.
|