blaze-host: per-gate evidence capture, separating asserted from observed
For the live FIFA gates. Records switch rules, sidecar status, log, trace and the Python contract result into a timestamped bundle, and reports CONFIGURED and OBSERVED state as two distinct sections. The separation is the whole point. 'blaze-switch.sh status = ON' is an assertion produced by the same tooling that performs the switch, and that tooling reported a successful rollback once when none had happened. The observed half comes from an unrelated source: the sidecar's own record of which peers connected to it. A non-loopback peer in that log proves the client's Blaze traffic landed on Rust without depending on reading an iptables rule correctly. Verified both ways: loopback-only traffic reports 'a FIFA session did NOT land here'; a non-loopback peer reports that it observably did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
funman300
@@ -200,3 +200,30 @@ be diffed against one against Python.
|
||||
No credential or token is logged. Volatile values are replaced before they reach
|
||||
the line, not truncated after, and a test asserts a known secret never appears
|
||||
in trace output.
|
||||
|
||||
## Evidence capture per gate
|
||||
|
||||
```bash
|
||||
./gate-evidence.sh <gate-label> # after each gate; never modifies anything
|
||||
```
|
||||
|
||||
Writes a timestamped bundle (switch rules, sidecar status, log, trace, Python
|
||||
contract result) and reports **configured** and **observed** state separately.
|
||||
|
||||
That separation is the point. `blaze-switch.sh status = ON` is an assertion from
|
||||
the same tooling that performs the switch — and that tooling reported a
|
||||
successful rollback once when it had not happened. The observed half comes from
|
||||
a different source: the sidecar's own record of which peers connected to it. A
|
||||
non-loopback peer in the sidecar log is proof the client's Blaze traffic landed
|
||||
on Rust that does not depend on reading an iptables rule correctly.
|
||||
|
||||
The script says so explicitly, in one of two forms:
|
||||
|
||||
```
|
||||
REMOTE peer(s) reached the Rust sidecar: 10.10.0.x
|
||||
=> the client's Blaze traffic observably landed on Rust
|
||||
```
|
||||
```
|
||||
no remote peer connected — only loopback (or nothing) reached Rust
|
||||
=> a FIFA session did NOT land here
|
||||
```
|
||||
|
||||
Executable
+127
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env bash
|
||||
# Capture evidence for one live-FIFA gate.
|
||||
#
|
||||
# gate-evidence.sh <gate-label> [outdir]
|
||||
#
|
||||
# Records what the system was configured to do AND what it observably did, and
|
||||
# reports them separately.
|
||||
#
|
||||
# WHY BOTH
|
||||
#
|
||||
# `blaze-switch.sh status = ON` is an assertion produced by the same tooling
|
||||
# that performs the switch. If that tooling is wrong — and it has been once
|
||||
# already, reporting a rollback that had not happened — the assertion is
|
||||
# worthless. The observed half comes from a different source entirely: the
|
||||
# sidecar's own record of which peers connected to it. A remote peer appearing
|
||||
# in the sidecar log is proof the client reached Rust that does not depend on
|
||||
# reading an iptables rule correctly.
|
||||
#
|
||||
# Run it after each gate. It never modifies anything.
|
||||
set -uo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$(readlink -f "$0")")" && pwd)"
|
||||
ROOT="$(cd "$HERE/.." && pwd)"
|
||||
RUNDIR="${OPENFUT_SIDECAR_RUNDIR:-${TMPDIR:-/tmp}/openfut-sidecar}"
|
||||
LOGFILE="${OPENFUT_SIDECAR_LOG:-$RUNDIR/sidecar.log}"
|
||||
TRACE="${OPENFUT_BLAZE_TRACE:-}"
|
||||
|
||||
LABEL="${1:-}"
|
||||
OUT="${2:-$ROOT/gate-evidence}"
|
||||
[[ -n "$LABEL" ]] || { echo "usage: gate-evidence.sh <gate-label> [outdir]" >&2; exit 2; }
|
||||
|
||||
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
DEST="$OUT/${STAMP}-${LABEL}"
|
||||
mkdir -p "$DEST"
|
||||
|
||||
say() { echo "$@"; }
|
||||
both() { echo "$@" | tee -a "$DEST/summary.txt" >/dev/null; echo "$@"; }
|
||||
|
||||
both "=== gate evidence: $LABEL ($STAMP) ==="
|
||||
both ""
|
||||
|
||||
# ---------------------------------------------------- configured (asserted)
|
||||
both "--- CONFIGURED (asserted by tooling) ---"
|
||||
{
|
||||
"$HERE/blaze-switch.sh" status 2>&1
|
||||
echo
|
||||
"$HERE/sidecar.sh" status 2>&1
|
||||
} > "$DEST/configured.txt"
|
||||
sed 's/^/ /' "$DEST/configured.txt" | tee -a "$DEST/summary.txt"
|
||||
|
||||
# Raw rules, straight from the kernel, not via our parser.
|
||||
{ sudo iptables -t nat -S 2>/dev/null || true; } > "$DEST/iptables-nat.txt"
|
||||
both ""
|
||||
|
||||
# ------------------------------------------------------ observed (measured)
|
||||
both "--- OBSERVED (measured from the sidecar's own record) ---"
|
||||
|
||||
if [[ ! -f "$LOGFILE" ]]; then
|
||||
both " no sidecar log at $LOGFILE — nothing observed"
|
||||
else
|
||||
cp "$LOGFILE" "$DEST/sidecar.log" 2>/dev/null
|
||||
|
||||
banner="$(grep -m1 'openfut-blaze-host v' "$LOGFILE" 2>/dev/null || true)"
|
||||
both " build: ${banner:-<none>}"
|
||||
if grep -q 'tree=DIRTY' <<<"$banner"; then
|
||||
both " !! DIRTY BUILD — this run is NOT parity evidence"
|
||||
fi
|
||||
|
||||
conns="$(grep -c 'CONNECT from' "$LOGFILE" 2>/dev/null || echo 0)"
|
||||
both " connections accepted: $conns"
|
||||
|
||||
# THE INDEPENDENT ASSERTION: which peers actually reached this process.
|
||||
peers="$(grep -o 'CONNECT from [0-9.]*' "$LOGFILE" 2>/dev/null | awk '{print $3}' | sort -u || true)"
|
||||
remote="$(grep -v '^127\.' <<<"$peers" | grep -v '^$' || true)"
|
||||
both " peers: $(tr '\n' ' ' <<<"$peers")"
|
||||
if [[ -n "$remote" ]]; then
|
||||
both " REMOTE peer(s) reached the Rust sidecar: $(tr '\n' ' ' <<<"$remote")"
|
||||
both " => the client's Blaze traffic observably landed on Rust"
|
||||
else
|
||||
both " no remote peer connected — only loopback (or nothing) reached Rust"
|
||||
both " => a FIFA session did NOT land here"
|
||||
fi
|
||||
|
||||
# Session shape, straight from the log.
|
||||
logins="$(grep -c 'Authentication::login .*REPLY' "$LOGFILE" 2>/dev/null || echo 0)"
|
||||
notifs="$(grep -c 'UserSessions::<' "$LOGFILE" 2>/dev/null || echo 0)"
|
||||
both " login replies: $logins UserSessions notifications: $notifs"
|
||||
both " close reasons:"
|
||||
grep -o 'CLOSE after [0-9]* frame(s): .*' "$LOGFILE" 2>/dev/null \
|
||||
| sort | uniq -c | sed 's/^/ /' | tee -a "$DEST/summary.txt" || true
|
||||
|
||||
# Anything that looks wrong.
|
||||
probs="$(grep -E 'DECODE FAILED|REJECT|FAILED|absurd' "$LOGFILE" 2>/dev/null | head -20 || true)"
|
||||
if [[ -n "$probs" ]]; then
|
||||
both " ANOMALIES:"
|
||||
sed 's/^/ /' <<<"$probs" | tee -a "$DEST/summary.txt"
|
||||
else
|
||||
both " no anomalies in the log"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------------ trace
|
||||
both ""
|
||||
both "--- TRACE ---"
|
||||
if [[ -n "$TRACE" && -f "$TRACE" ]]; then
|
||||
cp "$TRACE" "$DEST/rust-blaze.trace"
|
||||
frames="$(grep -c '^conn-' "$TRACE" 2>/dev/null || echo 0)"
|
||||
both " $TRACE -> $DEST/rust-blaze.trace ($frames traced frames)"
|
||||
both " routes seen:"
|
||||
grep -o '^conn-[0-9]* \(RX\|TX\) [^ ]* [A-Za-z]*::[^ ]*' "$TRACE" 2>/dev/null \
|
||||
| awk '{print $2, $4}' | sort | uniq -c | sort -rn | head -20 \
|
||||
| sed 's/^/ /' | tee -a "$DEST/summary.txt" || true
|
||||
else
|
||||
both " no trace configured (set OPENFUT_BLAZE_TRACE before starting the sidecar)"
|
||||
fi
|
||||
|
||||
# ----------------------------------------------------------- python health
|
||||
both ""
|
||||
both "--- PYTHON BACKEND (must stay healthy throughout) ---"
|
||||
if contract="$(cd "$ROOT/fifa17-recon" && timeout 120 python3 tools/test_fut_contract.py 2>&1 | tail -1)"; then
|
||||
both " contract suite: $contract"
|
||||
else
|
||||
both " contract suite: FAILED TO RUN"
|
||||
fi
|
||||
|
||||
both ""
|
||||
both "evidence bundle: $DEST"
|
||||
Reference in New Issue
Block a user