The reboot-survival gate is a test no operator can stand inside: the machine
under test is the machine running the session. So the machine records its own
recovery.
openfut-boot-evidence.service polls until the anchor, Core and host agree on
a namespace (or a 180s deadline expires), then writes a JSON file with the
boot id, anchor/Core/host pids and netns inodes, unit states, restart counts,
whether the reconciler had to act this boot, mount count, four non-mutating
probes, route ownership, and the full economy snapshot — plus the journal for
the boot so ordering is read from real timestamps rather than inferred from
unit dependencies.
It observes only; it never starts, stops or repairs anything, and carries no
Requires= or ordering that anything else waits on, so it cannot affect the
boot it is measuring. If the chain is broken the file says so, which is the
point.
Polling rather than a fixed sleep means a boot-time reconcile retry is
recorded as "settled late" rather than as a failure.
Two things surfaced by the production promotion.
`status` counted namespace mounts with an unanchored grep, so on production
"run/netns/openfut" also matched "openfut-staging" and reported a phantom
"2 = leaked stack" against a perfectly healthy host. A status command that
invents a fault is the same class of bug as a unit that reports active while
serving nobody, so it is fixed with an exact mount-point match. The bind and
reconcile logic is untouched; it always umounted an exact path.
openfut-rollback-detached.sh makes the documented rollback executable rather
than a paragraph in a runbook: it removes supervision, resolves the anchor's
CURRENT pid from Docker, and relaunches the incumbent detached pair with the
environment replayed from the captured env.json. --dry-run prints the exact
commands and touches nothing, which is how it was validated while production
was still being served by the processes it would restore.
Recreating the Docker anchor left the supervised Core and host stranded in
the dead namespace while systemd still reported them active — serving
nobody, invisible to any monitoring that trusts unit state. Reproduced on
staging: netns 4026539938 -> 4026540033, both pids unchanged in the old
one, both units "active", traffic ConnectionResetError.
There is no systemd-native edge signal to bind to. Containers do appear as
units, but the scope name embeds the container ID (docker-<id>.scope), which
changes on every recreate, so BindsTo= has no stable target;
NetworkNamespacePath= resolves once at start; a .path unit on /run/netns
would watch the file this tooling maintains. So: a level-triggered reconcile
on a 10s timer, comparing the namespace the services are ACTUALLY in against
the anchor's CURRENT one, acting only on a real difference. That cannot miss
an event while the watcher restarts or dockerd is down, and needs no
debounce — a burst of three recreations produced exactly one rebind. The
trigger stays separable: a docker-events unit could invoke the same script.
Anchor absent stops the dependants rather than falling back to host
networking; docker unavailable logs once and retries on the next tick.
Two defects found while testing and fixed here:
- mount --bind STACKS when the old mount is busy, silently leaking nsfs
entries; the bind helper now drains stale mounts in a loop.
- reconcile must stop -> rebind -> start, not rebind -> restart: a running
service holds the old namespace open and makes the umount fail busy.
Staging also gained a faithful anchor container so the reproduction is
structural rather than mocked. Economy state was byte-identical across every
lifecycle test. Production units are templates only and remain uninstalled.
Replaces the detached `setsid nohup … nsenter …` launch, which had no restart
policy, no boot persistence and no supervisor-visible logs. Staging units are
installed and proven; production units are TEMPLATES and are not installed.
Three decisions, each measured rather than assumed:
* `Wants=`, not `Requires=`, from host to Core. With `Requires`, stopping Core
stopped the host AND a later Core start did not bring it back -- a routine
Core restart would leave the client with no server. With `Wants` the host
survives a Core outage, answers 503 core_unavailable, never falls back to
Python, and resumes the moment Core returns with no intervention. Both halves
tested.
* Readiness is a bounded ExecStartPre TCP gate, because ordering proves nothing
about readiness and Type=exec only proves the binary exec'd. Core binds its
listener after migrations and content load, so "port open" is a real signal.
The gate FAILS rather than blocking: a host that waits forever looks healthy
while serving nobody.
* The netns is resolved by container NAME every start. The container is
restart=unless-stopped and its netns inode CHANGES on restart (measured:
4026539938 -> 4026540033), so a hardcoded pid is wrong by construction and
anything left in the old namespace serves nobody. Proven equivalent to today's
nsenter against a scratch container, never production's namespace.
`systemd-analyze verify` caught two real defects before deployment:
StartLimitIntervalSec/StartLimitBurst sat in [Service], where systemd 252
silently ignores them, so the crash-loop ceiling was not taking effect; and a
Documentation URL containing %20 parsed as a specifier. Both fixed and the
effective properties re-confirmed from the running units.
Staging evidence: Core-first ordering, host refused when Core is absent or
merely not listening, outage survival, automatic recovery, restart, graceful
stop with no strays, boot simulated via multi-user.target, 3x SIGKILL contained
at ~5s spacing, journald logs, and economy state byte-identical throughout
(integrity ok, fk 0).
funman300