ops: systemd supervision for Core and the FIFA17 host (staging-proven)
Replaces the detached `setsid nohup … nsenter …` launch, which had no restart policy, no boot persistence and no supervisor-visible logs. Staging units are installed and proven; production units are TEMPLATES and are not installed. Three decisions, each measured rather than assumed: * `Wants=`, not `Requires=`, from host to Core. With `Requires`, stopping Core stopped the host AND a later Core start did not bring it back -- a routine Core restart would leave the client with no server. With `Wants` the host survives a Core outage, answers 503 core_unavailable, never falls back to Python, and resumes the moment Core returns with no intervention. Both halves tested. * Readiness is a bounded ExecStartPre TCP gate, because ordering proves nothing about readiness and Type=exec only proves the binary exec'd. Core binds its listener after migrations and content load, so "port open" is a real signal. The gate FAILS rather than blocking: a host that waits forever looks healthy while serving nobody. * The netns is resolved by container NAME every start. The container is restart=unless-stopped and its netns inode CHANGES on restart (measured: 4026539938 -> 4026540033), so a hardcoded pid is wrong by construction and anything left in the old namespace serves nobody. Proven equivalent to today's nsenter against a scratch container, never production's namespace. `systemd-analyze verify` caught two real defects before deployment: StartLimitIntervalSec/StartLimitBurst sat in [Service], where systemd 252 silently ignores them, so the crash-loop ceiling was not taking effect; and a Documentation URL containing %20 parsed as a specifier. Both fixed and the effective properties re-confirmed from the running units. Staging evidence: Core-first ordering, host refused when Core is absent or merely not listening, outage survival, automatic recovery, restart, graceful stop with no strays, boot simulated via multi-user.target, 3x SIGKILL contained at ~5s spacing, journald logs, and economy state byte-identical throughout (integrity ok, fk 0).
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
[Unit]
|
||||
Description=OpenFUT Core (PRODUCTION) — authoritative economy state
|
||||
Documentation=file:///home/alex/OpenFUT/scripts/systemd/README.md
|
||||
# PRODUCTION TEMPLATE — NOT INSTALLED. Deploy only via the promotion plan in
|
||||
# `06 Operations/OpenFUT Service Supervision (staging-proven).md`.
|
||||
#
|
||||
# Core is the SINGLE WRITER of prod-core.db (verified by lsof: exactly one
|
||||
# process holds it open). Nothing here may be templated into a second instance.
|
||||
After=network-online.target docker.service openfut-netns.service
|
||||
Wants=network-online.target
|
||||
Requires=openfut-netns.service
|
||||
|
||||
# StartLimit* MUST live in [Unit]: systemd 252 silently IGNORES them in
|
||||
# [Service] (`systemd-analyze verify` flags it), which would have left the
|
||||
# crash-loop ceiling at the 10s/5 default instead of the intended 60s window.
|
||||
StartLimitIntervalSec=60
|
||||
StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=exec
|
||||
# root, matching the current production processes exactly. Supervision changes
|
||||
# HOW the process is started, never what it is or what it can reach.
|
||||
User=root
|
||||
|
||||
# The container's netns, published by openfut-netns.service. This replaces the
|
||||
# hand-typed `nsenter --net=/proc/<pid>/ns/net` in the runbook: same namespace,
|
||||
# no hardcoded pid, and re-resolved on every start.
|
||||
NetworkNamespacePath=/run/netns/openfut
|
||||
|
||||
EnvironmentFile=/etc/openfut/core.env
|
||||
|
||||
# An IMMUTABLE promotion artifact, not target/release. A later `cargo build`
|
||||
# must not be able to change what production is running — the same invariant
|
||||
# the promotion process already relies on.
|
||||
ExecStart=/home/alex/openfut-migration/promote-contract-20260822-184409/artifacts/openfut-core
|
||||
|
||||
KillSignal=SIGTERM
|
||||
KillMode=mixed
|
||||
# Generous, so a WAL checkpoint is never SIGKILLed mid-write. Observed shutdown
|
||||
# is sub-second.
|
||||
TimeoutStopSec=30
|
||||
|
||||
Restart=on-failure
|
||||
RestartSec=5s
|
||||
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=openfut-core
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in New Issue
Block a user