ops(systemd): exact mount match in status, add detached rollback script

Two things surfaced by the production promotion. `status` counted namespace mounts with an unanchored grep, so on production "run/netns/openfut" also matched "openfut-staging" and reported a phantom "2 = leaked stack" against a perfectly healthy host. A status command that invents a fault is the same class of bug as a unit that reports active while serving nobody, so it is fixed with an exact mount-point match. The bind and reconcile logic is untouched; it always umounted an exact path. openfut-rollback-detached.sh makes the documented rollback executable rather than a paragraph in a runbook: it removes supervision, resolves the anchor's CURRENT pid from Docker, and relaunches the incumbent detached pair with the environment replayed from the captured env.json. --dry-run prints the exact commands and touches nothing, which is how it was validated while production was still being served by the processes it would restore.
2026-08-22 21:28:21 +00:00
parent 1e8d46b258
commit fc1fdcc5ab
2 changed files with 63 additions and 1 deletions
@@ -0,0 +1,58 @@
+#!/bin/sh
+# Roll the production supervision promotion back to the incumbent detached pair.
+#
+# This restores the process-management arrangement that ran production before
+# the supervision cutover: two nohup'd binaries entered into the anchor
+# container's network namespace via nsenter, parented to init.
+#
+# It performs NO database or schema change, because the promotion it reverses
+# performs none either. The economy is never at risk here.
+#
+# Run with --dry-run first; that prints the exact commands and touches nothing.
+#
+# usage: openfut-rollback-detached.sh [--dry-run]
+set -eu
+
+DRY=0
+[ "${1:-}" = "--dry-run" ] && DRY=1
+
+ANCHOR=openfut-fut-backend
+ART=/home/alex/openfut-migration/promote-contract-20260822-184409/artifacts
+ENVJSON=/home/alex/openfut-migration/promote-contract-20260822-184409/env.json
+LOGDIR=/home/alex/openfut-promotion/logs
+
+run() {
+    if [ "$DRY" -eq 1 ]; then printf '  DRY  %s\n' "$*"; else printf '  RUN  %s\n' "$*"; sh -c "$*"; fi
+}
+
+echo "== 1. stop and remove supervision (units only; binaries, env and DB untouched) =="
+run "systemctl disable --now openfut-host.service openfut-core.service openfut-netns.service openfut-netns-reconcile.timer || true"
+run "/home/alex/OpenFUT/scripts/systemd/openfut-supervision-install.sh uninstall production"
+
+echo "== 2. resolve the anchor's CURRENT namespace (never a remembered pid) =="
+if [ "$DRY" -eq 1 ]; then
+    APID="$(docker inspect -f '{{.State.Pid}}' "$ANCHOR" 2>/dev/null || echo '<anchor pid>')"
+    echo "  DRY  anchor $ANCHOR pid=$APID ns=$(readlink "/proc/$APID/ns/net" 2>/dev/null || echo '<ns>')"
+else
+    APID="$(docker inspect -f '{{.State.Pid}}' "$ANCHOR")"
+    [ -n "$APID" ] && [ "$APID" != "0" ] || { echo "anchor $ANCHOR is not running — cannot roll back into its namespace" >&2; exit 1; }
+    echo "  anchor $ANCHOR pid=$APID ns=$(readlink "/proc/$APID/ns/net")"
+fi
+
+# The environment is replayed from the captured live process environment, never
+# retyped. jq renders it as KEY=VALUE pairs for env(1).
+CORE_ENV="$(jq -r '.core | to_entries[] | "\(.key)=\(.value)"' "$ENVJSON" | tr '\n' ' ')"
+HOST_ENV="$(jq -r '.host | to_entries[] | "\(.key)=\(.value)"' "$ENVJSON" | tr '\n' ' ')"
+
+echo "== 3. relaunch incumbent Core (the writer first) =="
+run "mkdir -p $LOGDIR"
+run "nsenter --net=/proc/$APID/ns/net env $CORE_ENV nohup $ART/openfut-core >> $LOGDIR/prod-core.log 2>&1 &"
+run "/home/alex/OpenFUT/scripts/systemd/openfut-wait-tcp.sh 127.0.0.1 18080 30"
+
+echo "== 4. relaunch incumbent host =="
+run "nsenter --net=/proc/$APID/ns/net env $HOST_ENV nohup $ART/openfut-utas-host >> $LOGDIR/prod-host.log 2>&1 &"
+run "/home/alex/OpenFUT/scripts/systemd/openfut-wait-tcp.sh 127.0.0.1 8099 30"
+
+echo "== 5. verify =="
+run "pgrep -af 'openfut-core|openfut-utas-host'"
+echo "Rollback complete. No database or schema change was made."