funman300/OpenFUT
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
db743ffd1f4a67b5012db0c9d5b0ce240d3926ac
OpenFUT/scripts/sold-staging-up.py
T
funman300 db743ffd1f feat(fifa17): project owned kits with active home/away designation 
Closes the server side of the FUT kit selector. Ownership stays generic in
Core (submodule bump: club_kit_assignments + GET/PUT /club/kits); this
commit adds the FIFA17 representation, the host projection and importer
support.

adapter:
* ContentKind::Kit ("kit") so kits are classified alongside player/staff/
  consumable instead of being mistaken for 0-rated players.
* Fifa17CardIdentity carries card_asset_id and team_id; RawCard keeps both
  optional because the emitted catalog writes null for non-kit definitions.
* shape_kit_item emits only the fields the client's kit path reads
  (id/resourceId/assetId/cardassetid/cardsubtypeid/itemState/owners/
  untradeable/teamid) — no attributeList, no itemType.
* itemState on the wire is the STRING token activeHomeKit/activeAwayKit;
  the 101/102 integers are the client's post-deserialisation runtime enum
  (item+0x5c) and are never emitted.
* club_stats S_KITS (0x28) now counts owned kits instead of a hard zero.

host:
* CoreKitAssignments + CoreAccess::get_active_kits (GET /club/kits),
  defaulting to no active kits so a Core without the endpoint degrades
  instead of fabricating a designation.
* handle_club classifies type=player|kit, rejects any other type with an
  empty page and outcome=unsupported_type, and now always fetches
  unpaginated from Core: kind and transfer-pile membership are host-side
  concepts Core cannot express, so filtering and pagination must both
  happen after shaping or pages come back short.

importer:
* ItemClass::Kit (cardsubtypeid == 9 and resourceId in 6_300_000..=6_400_654),
  kit counts/balances, and card_asset_id/team_id carried into the emitted
  catalog and manifest.
* a kit group missing cardassetid == 35 or teamid is DEFERRED
  (missing_kit_render_metadata) rather than defaulted; conflicting render
  metadata across instances defers as render_metadata_conflict.

staging: sold-staging-up.py seeds two owned kits (6300006 home / 6400003
away, team 21) plus both active designations so the projection can be
verified over HTTP before involving the client.
2026-08-21 03:17:57 +00:00

1106 lines
44 KiB
Python
Executable File
Raw Blame History

#!/usr/bin/env python3
"""Bring up a COMPLETE, ISOLATED FIFA-17 staging stack for the seller-facing SOLD
A/B experiment, so a real FIFA 17 client can be pointed at it without production
being touched in any way.
ONE entry point. It starts, in order:
1. staging Core 127.0.0.1:18081 throwaway sqlite under the staging dir
2. staging utas-host 0.0.0.0:8299 own market/pile/identity/clientdata files
3. staging Blaze 0.0.0.0:42327 (redirector) / 42330 (main) / 42331 (nucleus)
and then prints the three `openfut.cfg` lines the operator must put on the client.
Tear the whole thing down with `scripts/sold-staging-down.py`.
WHY the client only needs Blaze ports: FIFA 17 learns both the UTAS base URL and
the roster URL from Blaze. This script copies the responder into the staging dir,
rewrites the hardcoded UTAS port, and passes the independently configurable roster
host through `OPENFUT_ROSTER_HOST`.
ISOLATION, enforced not assumed:
* production ports 8099 8199 18080 8443 42127 42130 42131 4216 8080 8081 8094 are
a hard deny-list: never bound, never connected to, and every chosen staging port
is checked against it AND checked free before anything is launched;
* nothing under /home/alex/openfut-promotion/state/ is ever opened -- every path
this script touches goes through `safe_path()`, which refuses that prefix;
* the Core/utas-host binaries are COPIED into the staging dir and run from there,
so (a) a later `cargo build` cannot change what staging is running and (b) every
staging process's /proc cmdline provably contains the staging directory, which is
what the down script requires before it will signal anything;
* `OPENFUT_UTAS_PYTHON_URL` points at an unused loopback port, so any Python
fallback fails closed and loudly instead of silently serving production data;
* nothing on the FIFA client machine (10.10.0.105) is modified -- the operator
edits `openfut.cfg` by hand, using the block this script prints.
Read-only reuse of production: staging Blaze advertises the production roster
service through a certificate dNSName (`winter15.gosredirector.ea.com:8081` by
default) and advertises the production POW content/API hosts
(`10.10.0.120:8085` / `:8094`) verbatim. These services hold NO economy state.
The client must resolve the roster hostname to this server without changing the
advertised URL.
python3 scripts/sold-staging-up.py --variant highest
python3 scripts/sold-staging-up.py --variant buyNow --coins-processed 1 \
--count-mode active_plus_sold
python3 scripts/sold-staging-up.py --variant off # sold projection disabled
"""
from __future__ import annotations
import argparse
import http.client
import json
import os
import re
import shutil
import signal
import socket
import sqlite3
import subprocess
import sys
import time
# --- fixed facts -------------------------------------------------------------------
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Production. Never bind, never connect, never open.
FORBIDDEN_PORTS = frozenset(
{8099, 8199, 18080, 8443, 42127, 42130, 42131, 4216, 8080, 8081, 8094}
)
FORBIDDEN_PATHS = ("/home/alex/openfut-promotion/state",)
PROD_PIDS = {3631953: "prod utas-host", 3374264: "prod Core"}
# The staging port block. One obvious place; every one of these is asserted free.
# 42227 (the first choice for the redirector) is permanently occupied by
# openfut-redirector-host, so the redirector sits at 42327.
CORE_PORT = 18081
HOST_PORT = 8299
BLAZE_REDIR_PORT = 42327
BLAZE_MAIN_PORT = 42330
BLAZE_NUCLEUS_PORT = 42331
# Deliberately dead: the utas-host requires a Python upstream, and this one must
# never resolve to the production oracle on 8199.
DEAD_PYTHON_PORT = 8399
STAGING_PORTS = {
"staging Core": CORE_PORT,
"staging utas-host": HOST_PORT,
"staging blaze redirector": BLAZE_REDIR_PORT,
"staging blaze main": BLAZE_MAIN_PORT,
"staging blaze nucleus": BLAZE_NUCLEUS_PORT,
}
DEFAULT_STAGING_DIR = "/home/alex/openfut-sold-staging"
# What staging Blaze tells the client about itself and about the economy-free
# auxiliary services. Same advertise IP and same POW hosts production Blaze uses.
ADVERTISE = "10.10.0.120"
BIND = "0.0.0.0"
POW_CONTENT_HOST = "10.10.0.120:8085"
POW_HOST = "10.10.0.120:8094"
DEFAULT_ROSTER_HOST = "winter15.gosredirector.ea.com:8081"
BLAZE_SRC = os.path.join(REPO, "fifa17-recon", "tools", "blaze_responder_v3b.py")
BLAZE_ASSETS = ("redir_cert.pem", "redir_key.pem")
# Emitted FIFA17 content, from a build tree OUTSIDE the forbidden production state
# directory. `cards` is Core's content pack (a bare JSON array of CardDefinition);
# `catalog` is the adapter's identity catalog (card id -> asset id/version/rareflag).
CONTENT_SRC = "/home/alex/openfut-post-p1/staging/emit/content"
CARDS_NAME = "fifa17-production-cards.json"
CATALOG_NAME = "fifa17-production-catalog.json"
GAME = "fifa17"
PERSONA_ID = "33068179"
PERSONA_NAME = "CAGE"
TS = "2026-01-01T00:00:00Z"
# Seller A is the real FIFA persona, so the retail client logs into THIS profile
# (Core resolves the active profile by game_id, and X-OpenFUT-Game is `fifa17`).
SELLER_PROFILE = "prof-seller-a-cage"
SELLER_CLUB = "club-seller-a-cage"
SELLER_COINS = 1_000
SELLER_SQUAD = "squad-seller-a"
# Buyer B is a synthetic second club. It is parked on its OWN game_id so it can
# never become the active `fifa17` profile -- Core is single-profile-per-game.
BUYER_PROFILE = "prof-buyer-b"
BUYER_CLUB = "club-buyer-b"
BUYER_GAME = "fifa17-buyer-b"
BUYER_COINS = 20_000
# 11 starters + 1 disposable item for the seller. Every card_id here exists in BOTH
# the content pack (so Core's content preflight passes) and the identity catalog (so
# the host can resolve a resourceId); both memberships are asserted before launch.
SELLER_SQUAD_CARDS = [
("owned-a-gk", "fifa17_84053575"), # Manuel Neuer GK 97
("owned-a-lb", "fifa17_151192389"), # David Alaba LB 91
("owned-a-cb1", "fifa17_134381968"), # Thiago Silva CB 92
("owned-a-cb2", "fifa17_151177437"), # Diego Godin CB 92
("owned-a-rb", "fifa17_100785235"), # Philipp Lahm RB 90
("owned-a-cm1", "fifa17_151171947"), # Luka Modric CM 93
("owned-a-cm2", "fifa17_84054731"), # Ivan Rakitic CM 92
("owned-a-cm3", "fifa17_134400249"), # Toni Kroos CM 91
("owned-a-lw", "fifa17_83906881"), # Cristiano Ronaldo LW 99
("owned-a-st", "fifa17_117617092"), # Luis Suarez ST 95
("owned-a-rw", "fifa17_84044103"), # Lionel Messi RW 98
]
# THE disposable item: what the operator lists and sells during the experiment.
DISPOSABLE_ITEM = "owned-a-disposable"
DISPOSABLE_CARD = "fifa17_232273" # Nelson Atiagli LB 51, rareflag 1
# Two authoritative modern kit-card definitions for one real source team. These
# are staging fixtures derived from fcc_kitcards, not synthetic FIFA identities.
KIT_TEAM_ID = 21
STAGING_KITS = [
("home", "owned-a-kit-home", "fifa17_6300006", 6_300_006),
("away", "owned-a-kit-away", "fifa17_6400003", 6_400_003),
]
READY_TIMEOUT_S = 60.0
# --- output ------------------------------------------------------------------------
def banner(title: str) -> None:
print()
print("=" * 78)
print(f"== {title}")
print("=" * 78)
def step(msg: str) -> None:
print(f" {msg}")
def ok(msg: str) -> None:
print(f" [ OK ] {msg}")
class Fatal(RuntimeError):
"""Anything that must abort bring-up loudly rather than degrade."""
# --- isolation guards --------------------------------------------------------------
def safe_path(path: str) -> str:
"""Every filesystem path in this script goes through here. Refuses the live
production state directory outright -- a typo cannot reach prod-core.db."""
real = os.path.realpath(path)
for bad in FORBIDDEN_PATHS:
if real == bad or real.startswith(bad + os.sep):
raise Fatal(f"REFUSING to touch production state: {path} -> {real}")
return path
def check_port_allowed(port: int, what: str) -> None:
if port in FORBIDDEN_PORTS:
raise Fatal(f"REFUSING: {what} port {port} is a PRODUCTION port")
def listening_ports() -> set[int]:
"""Every TCP port in state LISTEN in this network namespace, read straight from
the kernel socket table.
A trial bind() is the wrong test: after a server exits, its accepted sockets sit
in TIME_WAIT holding the same local port, so bind() reports EADDRINUSE for a
minute even though nothing is serving -- and every server here sets SO_REUSEADDR
and would bind fine. This mirrors `ss -ltn` (and host-lifecycle.sh's
hl_port_listening), which is the question actually being asked."""
ports: set[int] = set()
for path in ("/proc/net/tcp", "/proc/net/tcp6"):
try:
with open(path) as fh:
next(fh, None) # header
for line in fh:
fields = line.split()
if len(fields) < 4 or fields[3] != "0A": # TCP_LISTEN
continue
ports.add(int(fields[1].rsplit(":", 1)[1], 16))
except OSError:
continue
return ports
def port_free(port: int) -> bool:
return port not in listening_ports()
def assert_ports_free() -> None:
busy = []
for what, port in STAGING_PORTS.items():
check_port_allowed(port, what)
if not port_free(port):
busy.append(f"{what} {port}")
check_port_allowed(DEAD_PYTHON_PORT, "dead python upstream")
if not port_free(DEAD_PYTHON_PORT):
busy.append(
f"dead python upstream {DEAD_PYTHON_PORT} (it MUST stay unbound so the "
"Python fallback fails closed)"
)
if busy:
raise Fatal(
"REFUSING to start -- these staging ports are not free:\n "
+ "\n ".join(busy)
+ "\n Nothing was launched. Free them, or edit the port block at the "
"top of this script."
)
ok(
"staging ports free and none is a production port: "
+ ", ".join(str(p) for p in STAGING_PORTS.values())
)
def pid_alive(pid: int) -> bool:
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except PermissionError:
return True # exists, owned by root (the prod stack runs under sudo)
return True
def cmdline_of(pid: int) -> str:
try:
with open(f"/proc/{pid}/cmdline", "rb") as fh:
return fh.read().replace(b"\0", b" ").decode(errors="replace").strip()
except OSError:
return ""
def assert_prod_alive(where: str) -> None:
for pid, what in PROD_PIDS.items():
if not pid_alive(pid):
raise Fatal(f"{what} pid {pid} is NOT alive at {where} -- stop and investigate")
ok(
f"production untouched at {where}: "
+ ", ".join(f"{what} pid {pid} alive" for pid, what in PROD_PIDS.items())
)
# --- HTTP ---------------------------------------------------------------------------
def http_get(port: int, path: str, timeout: float = 5.0):
check_port_allowed(port, "HTTP request")
conn = http.client.HTTPConnection("127.0.0.1", port, timeout=timeout)
try:
conn.request("GET", path, headers={"X-OpenFUT-Game": GAME})
resp = conn.getresponse()
raw = resp.read()
return resp.status, raw.decode("utf-8", "replace")
finally:
conn.close()
def wait_http(port: int, path: str, proc, log_path: str, label: str) -> None:
deadline = time.monotonic() + READY_TIMEOUT_S
last = "no attempt"
while time.monotonic() < deadline:
if proc.poll() is not None:
raise Fatal(f"{label} exited {proc.returncode} during startup\n{tail(log_path)}")
try:
status, _ = http_get(port, path)
if status < 500:
return
last = f"HTTP {status}"
except OSError as exc:
last = f"{type(exc).__name__}: {exc}"
time.sleep(0.15)
raise Fatal(
f"{label} never answered {path} on 127.0.0.1:{port} within "
f"{READY_TIMEOUT_S:.0f}s (last: {last})\n{tail(log_path)}"
)
def wait_tcp(ports: list[int], proc, log_path: str, label: str) -> None:
deadline = time.monotonic() + READY_TIMEOUT_S
pending = list(ports)
while time.monotonic() < deadline and pending:
if proc.poll() is not None:
raise Fatal(f"{label} exited {proc.returncode} during startup\n{tail(log_path)}")
still = []
for port in pending:
check_port_allowed(port, f"{label} readiness probe")
s = socket.socket()
s.settimeout(1.0)
try:
s.connect(("127.0.0.1", port))
except OSError:
still.append(port)
finally:
s.close()
pending = still
if pending:
time.sleep(0.15)
if pending:
raise Fatal(f"{label} never listened on {pending}\n{tail(log_path)}")
def tail(log_path: str, lines: int = 30) -> str:
try:
with open(log_path, errors="replace") as fh:
body = fh.read().splitlines()
except OSError:
return f"(no log at {log_path})"
return f"--- {os.path.basename(log_path)} tail ---\n" + "\n".join(body[-lines:])
# --- staging directory --------------------------------------------------------------
class Layout:
def __init__(self, root: str) -> None:
self.root = safe_path(os.path.abspath(root))
self.bin = os.path.join(self.root, "bin")
self.content = os.path.join(self.root, "content")
self.blaze = os.path.join(self.root, "blaze")
self.logs = os.path.join(self.root, "logs")
self.core_db = os.path.join(self.root, "staging-core.db")
self.market_db = os.path.join(self.root, "staging-market.db")
self.pile_db = os.path.join(self.root, "staging-pile.db")
self.identity = os.path.join(self.root, "staging-identity.json")
self.clientdata = os.path.join(self.root, "staging-clientdata.json")
self.cards = os.path.join(self.content, CARDS_NAME)
self.catalog = os.path.join(self.content, CATALOG_NAME)
self.blaze_script = os.path.join(self.blaze, "blaze_responder_staging.py")
self.blaze_rx = os.path.join(self.blaze, "rx")
self.manifest = os.path.join(self.root, "manifest.json")
self.core_bin = os.path.join(self.bin, "openfut-core")
self.host_bin = os.path.join(self.bin, "openfut-utas-host")
self.core_log = os.path.join(self.logs, "core.log")
self.host_log = os.path.join(self.logs, "utas-host.log")
self.blaze_log = os.path.join(self.logs, "blaze.log")
# The responder's OWN log(), kept separate from its stdout/stderr file so
# two writers never interleave in one file.
self.blaze_responder_log = os.path.join(self.logs, "blaze-responder.log")
for p in vars(self).values():
safe_path(p)
def make_dirs(self) -> None:
for d in (self.root, self.bin, self.content, self.blaze, self.logs,
self.blaze_rx):
os.makedirs(safe_path(d), exist_ok=True)
def refuse_if_up(lay: Layout) -> None:
"""A previous stack still running must be torn down by the down script, never
stepped on: two stacks would fight over the same sqlite files."""
if not os.path.exists(lay.manifest):
return
try:
with open(safe_path(lay.manifest)) as fh:
manifest = json.load(fh)
except (OSError, ValueError):
return
live = []
for proc in manifest.get("processes", []):
pid = int(proc["pid"])
if pid_alive(pid) and lay.root in cmdline_of(pid):
live.append(f"{proc['name']} pid {pid}")
if live:
raise Fatal(
"a previous staging stack is STILL UP:\n "
+ "\n ".join(live)
+ "\n Run: python3 scripts/sold-staging-down.py"
+ "\n Nothing was launched and nothing was signalled."
)
def reset_throwaway_state(lay: Layout) -> None:
"""Every staging database is throwaway by definition, so a re-`up` after a clean
`down` starts from nothing rather than refusing. Only files this script created
are removed, and only after `refuse_if_up` proved no stack is running."""
removed = []
for path in (lay.core_db, lay.market_db, lay.pile_db, lay.identity,
lay.clientdata, lay.manifest, lay.manifest + ".stopped"):
for candidate in (path, path + "-wal", path + "-shm"):
if os.path.exists(safe_path(candidate)):
os.remove(safe_path(candidate))
removed.append(os.path.basename(candidate))
if removed:
ok("removed stale throwaway state from a previous run: " + ", ".join(removed))
def materialise(lay: Layout) -> None:
lay.make_dirs()
for src, dst in (
(os.path.join(REPO, "target", "release", "openfut-core"), lay.core_bin),
(os.path.join(REPO, "target", "release", "openfut-utas-host"), lay.host_bin),
):
if not os.path.isfile(src):
raise Fatal(
f"not built: {src}\n cargo build --release -p openfut-core "
"-p openfut-utas-host"
)
shutil.copy2(safe_path(src), safe_path(dst))
os.chmod(dst, 0o755)
ok(f"binaries copied into {lay.bin} (a later rebuild cannot change staging)")
for name, dst in ((CARDS_NAME, lay.cards), (CATALOG_NAME, lay.catalog)):
src = os.path.join(CONTENT_SRC, name)
if not os.path.isfile(src):
raise Fatal(f"missing FIFA17 content source {src}")
shutil.copy2(safe_path(src), safe_path(dst))
ok(f"FIFA17 content copied from {CONTENT_SRC} (outside production state)")
with open(safe_path(lay.cards)) as fh:
definitions = json.load(fh)
with open(safe_path(lay.catalog)) as fh:
catalog = json.load(fh)
existing = {definition["id"] for definition in definitions}
for _slot, _owned_id, card_id, resource_id in STAGING_KITS:
if card_id not in existing:
definitions.append({
"id": card_id,
"name": "Kit",
"overall": 0,
"position": "",
"nation": "",
"league": "",
"club": "",
"pace": 0,
"shooting": 0,
"passing": 0,
"dribbling": 0,
"defending": 0,
"physical": 0,
"rarity": "bronze",
"image_path": None,
})
catalog["cards"][card_id] = {
"asset_id": resource_id,
"version": 0,
"rareflag": 0,
"kind": "kit",
"subtype": 9,
"card_asset_id": 35,
"team_id": KIT_TEAM_ID,
}
with open(safe_path(lay.cards), "w") as fh:
json.dump(definitions, fh, indent=2)
fh.write("\n")
with open(safe_path(lay.catalog), "w") as fh:
json.dump(catalog, fh, indent=2)
fh.write("\n")
ok("added ownership-backed home/away kit fixtures from fcc_kitcards")
def assert_seed_cards_resolvable(lay: Layout) -> None:
"""Core's content preflight rejects any owned card whose card_id is not a loaded
CardDefinition, and the host refuses to shape a /club item with no catalog
identity. Prove BOTH memberships now, not via a startup crash later."""
with open(safe_path(lay.cards)) as fh:
pack_ids = {c["id"] for c in json.load(fh)}
with open(safe_path(lay.catalog)) as fh:
catalog_ids = set(json.load(fh)["cards"])
wanted = (
[card for _, card in SELLER_SQUAD_CARDS]
+ [DISPOSABLE_CARD]
+ [card for _, _, card, _ in STAGING_KITS]
)
missing_pack = sorted(set(wanted) - pack_ids)
missing_cat = sorted(set(wanted) - catalog_ids)
if missing_pack or missing_cat:
raise Fatal(
"seed card ids are not resolvable -- Core or the host would fail at "
f"startup.\n absent from content pack: {missing_pack}"
f"\n absent from identity catalog: {missing_cat}"
)
ok(
f"all {len(wanted)} seed card ids present in BOTH the content pack "
f"({len(pack_ids)} defs) and the identity catalog ({len(catalog_ids)} entries)"
)
# --- blaze patching -----------------------------------------------------------------
# Every substitution is anchored to the whole assignment line and must apply exactly
# once. A silent no-op here would leave staging Blaze advertising PRODUCTION UTAS.
def blaze_patches(lay: Layout) -> list[tuple[str, str, str]]:
return [
("REDIR_PORT", r"^REDIR_PORT = 42127$", f"REDIR_PORT = {BLAZE_REDIR_PORT}"),
("BLAZE_PORT", r"^BLAZE_PORT = 42130$", f"BLAZE_PORT = {BLAZE_MAIN_PORT}"),
("NUCLEUS_PORT", r"^NUCLEUS_PORT = 42131$",
f"NUCLEUS_PORT = {BLAZE_NUCLEUS_PORT}"),
("UTAS_BASE", r'^UTAS_BASE = "http://%s:8099/" % _ADVERTISE$',
f'UTAS_BASE = "http://%s:{HOST_PORT}/" % _ADVERTISE'),
# Not protocol values, but the responder's two hardcoded /tmp paths: left
# alone, a staging run would write its frames and log into the shared
# host /tmp and make a capture ambiguous about which stack produced it.
("LOG", r'^LOG = "/tmp/blaze_responder\.log"$',
f'LOG = "{lay.blaze_responder_log}"'),
("RXDIR", r'^RXDIR = "/tmp/blaze_rx"$', f'RXDIR = "{lay.blaze_rx}"'),
]
def patch_blaze(lay: Layout) -> None:
if not os.path.isfile(BLAZE_SRC):
raise Fatal(f"missing blaze responder {BLAZE_SRC}")
with open(safe_path(BLAZE_SRC)) as fh:
text = fh.read()
for name, pattern, replacement in blaze_patches(lay):
text, n = re.subn(pattern, replacement.replace("\\", "\\\\"), text,
flags=re.MULTILINE)
if n != 1:
raise Fatal(
f"blaze patch {name} applied {n} times, expected exactly 1 "
f"(pattern {pattern!r}). The responder changed shape -- REFUSING to "
"run a half-patched copy that could point at production."
)
step(f"patched {name:<12} -> {replacement.split(' = ', 1)[1]}")
with open(safe_path(lay.blaze_script), "w") as fh:
fh.write(text)
for asset in BLAZE_ASSETS:
src = os.path.join(os.path.dirname(BLAZE_SRC), asset)
if not os.path.isfile(src):
raise Fatal(f"missing blaze TLS asset {src}")
# The responder resolves CERT/KEY relative to its own directory, so the
# assets are copied rather than patched.
shutil.copy2(safe_path(src), safe_path(os.path.join(lay.blaze, asset)))
verify_blaze_patch(lay)
def verify_blaze_patch(lay: Layout) -> None:
"""Re-read the file from disk and prove the patched copy cannot reach production
Blaze ports or production UTAS."""
with open(safe_path(lay.blaze_script)) as fh:
lines = fh.read().splitlines()
def assignment(name: str) -> str:
hits = [ln for ln in lines if re.match(rf"^{name} = ", ln)]
if len(hits) != 1:
raise Fatal(f"patched blaze copy has {len(hits)} `{name} =` lines")
return hits[0]
expected = {
"REDIR_PORT": f"REDIR_PORT = {BLAZE_REDIR_PORT}",
"BLAZE_PORT": f"BLAZE_PORT = {BLAZE_MAIN_PORT}",
"NUCLEUS_PORT": f"NUCLEUS_PORT = {BLAZE_NUCLEUS_PORT}",
"UTAS_BASE": f'UTAS_BASE = "http://%s:{HOST_PORT}/" % _ADVERTISE',
"LOG": f'LOG = "{lay.blaze_responder_log}"',
"RXDIR": f'RXDIR = "{lay.blaze_rx}"',
}
for name, want in expected.items():
got = assignment(name)
if got != want:
raise Fatal(f"patched blaze {name} is {got!r}, expected {want!r}")
utas = assignment("UTAS_BASE")
if ":8099" in utas:
raise Fatal(f"patched blaze STILL advertises production UTAS: {utas!r}")
ok(f"patched blaze UTAS_BASE = {utas.split(' = ', 1)[1]} (no :8099)")
ok(
"patched blaze ports: redirector "
f"{BLAZE_REDIR_PORT} / main {BLAZE_MAIN_PORT} / nucleus "
f"{BLAZE_NUCLEUS_PORT} (no 42127/42130/42131)"
)
# --- seeding ------------------------------------------------------------------------
def seed_core_db(lay: Layout) -> None:
"""Two identities by direct SQL, against the schema Core just migrated.
Column sets are from openfut-core/migrations/0001_initial.sql plus 0016's
profiles.game_id. Seller A carries game_id `fifa17` so the retail client (which
sends X-OpenFUT-Game: fifa17) resolves to it; Buyer B is parked on its own
game_id so it can never shadow the seller as the active fifa17 profile.
"""
conn = sqlite3.connect(safe_path(lay.core_db), timeout=15)
try:
conn.execute("PRAGMA busy_timeout = 15000")
with conn:
conn.executemany(
"INSERT INTO profiles (id, username, level, xp, created_at, "
"updated_at, game_id) VALUES (?, ?, 1, 0, ?, ?, ?)",
[
(SELLER_PROFILE, PERSONA_NAME, TS, TS, GAME),
(BUYER_PROFILE, "BUYER-B", TS, TS, BUYER_GAME),
],
)
conn.executemany(
"INSERT INTO clubs (id, profile_id, name, coins, level, created_at, "
"updated_at) VALUES (?, ?, ?, ?, 1, ?, ?)",
[
(SELLER_CLUB, SELLER_PROFILE, f"{PERSONA_NAME} FC", SELLER_COINS,
TS, TS),
(BUYER_CLUB, BUYER_PROFILE, "Buyer B FC", BUYER_COINS, TS, TS),
],
)
conn.executemany(
"INSERT INTO owned_cards (id, club_id, card_id, is_loan, "
"acquired_at) VALUES (?, ?, ?, 0, ?)",
[(item, SELLER_CLUB, card, TS) for item, card in SELLER_SQUAD_CARDS]
+ [(DISPOSABLE_ITEM, SELLER_CLUB, DISPOSABLE_CARD, TS)]
+ [
(owned_id, SELLER_CLUB, card_id, TS)
for _slot, owned_id, card_id, _resource_id in STAGING_KITS
],
)
conn.execute(
"INSERT INTO squads (id, club_id, name, formation, created_at, "
"updated_at) VALUES (?, ?, ?, ?, ?, ?)",
(SELLER_SQUAD, SELLER_CLUB, "Staging XI", "4-3-3", TS, TS),
)
conn.executemany(
"INSERT INTO squad_players (id, squad_id, owned_card_id, "
"position_index, is_captain, is_on_bench) VALUES (?, ?, ?, ?, ?, 0)",
[
(f"sp-{idx}", SELLER_SQUAD, item, idx, 1 if idx == 0 else 0)
for idx, (item, _) in enumerate(SELLER_SQUAD_CARDS)
],
)
conn.executemany(
"INSERT INTO club_kit_assignments (club_id, slot, owned_card_id, updated_at) "
"VALUES (?, ?, ?, ?)",
[
(SELLER_CLUB, slot, owned_id, TS)
for slot, owned_id, _card_id, _resource_id in STAGING_KITS
],
)
finally:
conn.close()
ok(
f"seeded Seller A ({PERSONA_NAME}, persona {PERSONA_ID}, {SELLER_COINS} coins, "
f"{len(SELLER_SQUAD_CARDS)} starters + 1 disposable + "
f"{len(STAGING_KITS)} active kits) and Buyer B ({BUYER_COINS} coins)"
)
def db_summary(lay: Layout) -> str:
conn = sqlite3.connect(safe_path(lay.core_db), timeout=15)
try:
clubs = conn.execute("SELECT id, coins FROM clubs ORDER BY id").fetchall()
owned = conn.execute(
"SELECT club_id, COUNT(*) FROM owned_cards GROUP BY club_id"
).fetchall()
finally:
conn.close()
return f"clubs={dict(clubs)} owned={dict(owned)}"
# --- launching ----------------------------------------------------------------------
class Launched:
def __init__(self, name: str, proc, log: str, port_note: str) -> None:
self.name = name
self.proc = proc
self.log = log
self.port_note = port_note
def record(self) -> dict:
return {
"name": self.name,
"pid": self.proc.pid,
"pgid": os.getpgid(self.proc.pid),
"cmdline": cmdline_of(self.proc.pid),
"log": self.log,
"ports": self.port_note,
}
def spawn(argv: list[str], env: dict, cwd: str, log_path: str, append=False):
"""Start detached (own session) so the stack survives this script exiting, and
so the down script can signal exactly this process group and nothing else."""
log = open(safe_path(log_path), "a" if append else "w", buffering=1)
try:
log.write(f"\n---- launch {' '.join(argv)} @ {time.strftime('%FT%TZ')} ----\n")
return subprocess.Popen(
argv,
cwd=safe_path(cwd),
env=env,
stdout=log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
finally:
log.close() # the child holds its own dup of the fd
def core_env(lay: Layout) -> dict:
return dict(
os.environ,
LISTEN_ADDR=f"127.0.0.1:{CORE_PORT}",
DATABASE_URL=f"sqlite://{lay.core_db}",
DATA_DIR=os.path.join(REPO, "openfut-core", "data"),
OPENFUT_CONTENT_PACKS=lay.cards,
RUST_LOG="openfut_core=info",
)
def migrate_core(lay: Layout) -> None:
"""Core owns its schema and has no migrate-only subcommand, so the fixture
cannot be written into an empty file: start it once, let it migrate, stop it,
seed, then start the long-lived instance."""
proc = spawn([lay.core_bin], core_env(lay), lay.root, lay.core_log)
try:
wait_http(CORE_PORT, "/health", proc, lay.core_log, "staging Core (migrate)")
ok(f"staging Core migrated {os.path.basename(lay.core_db)}")
finally:
proc.terminate()
try:
proc.wait(timeout=20)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=20)
def start_core(lay: Layout) -> Launched:
proc = spawn([lay.core_bin], core_env(lay), lay.root, lay.core_log, append=True)
wait_http(CORE_PORT, "/health", proc, lay.core_log, "staging Core")
ok(f"staging Core ready on 127.0.0.1:{CORE_PORT} (pid {proc.pid})")
return Launched("staging-core", proc, lay.core_log, f"127.0.0.1:{CORE_PORT}")
def start_host(lay: Layout, variant: str, coins_processed: str, count_mode: str) -> Launched:
env = dict(
os.environ,
OPENFUT_UTAS_HOST_ADDR=f"{BIND}:{HOST_PORT}",
OPENFUT_CORE_URL=f"http://127.0.0.1:{CORE_PORT}",
# NOT the production oracle on 8199. Nothing listens here, so any Python
# fallback fails closed and shows up in the log.
OPENFUT_UTAS_PYTHON_URL=f"http://127.0.0.1:{DEAD_PYTHON_PORT}",
OPENFUT_FIFA17_TABLES_DIR=os.path.join(REPO, "fifa17-recon", "data", "tables"),
OPENFUT_FIFA17_CATALOG=lay.catalog,
OPENFUT_IDENTITY_STORE=lay.identity,
OPENFUT_CLIENTDATA_DB=lay.clientdata,
OPENFUT_PERSONA_ID=PERSONA_ID,
OPENFUT_MARKET_DB=lay.market_db,
OPENFUT_PILE_DB=lay.pile_db,
RUST_LOG="info",
)
# OFF must mean "unset", not "set to something unrecognised": the host treats an
# unrecognised token as OFF, but leaving the variable behind invites confusion.
for key in ("OPENFUT_FIFA17_SOLD_EXPERIMENT", "OPENFUT_FIFA17_SOLD_COINS_PROCESSED",
"OPENFUT_FIFA17_SOLD_COUNT_MODE"):
env.pop(key, None)
if variant != "off":
env["OPENFUT_FIFA17_SOLD_EXPERIMENT"] = variant
env["OPENFUT_FIFA17_SOLD_COINS_PROCESSED"] = coins_processed
env["OPENFUT_FIFA17_SOLD_COUNT_MODE"] = count_mode
proc = spawn([lay.host_bin], env, lay.root, lay.host_log)
wait_http(HOST_PORT, f"/ut/game/{GAME}/tradePile/counts", proc, lay.host_log,
"staging utas-host")
ok(f"staging utas-host ready on {BIND}:{HOST_PORT} (pid {proc.pid})")
return Launched("staging-utas-host", proc, lay.host_log, f"{BIND}:{HOST_PORT}")
def start_blaze(lay: Layout, roster_host: str) -> Launched:
env = dict(
os.environ,
OPENFUT_ADVERTISE=ADVERTISE,
OPENFUT_BIND=BIND,
OPENFUT_ROSTER_HOST=roster_host,
# Read-only reuse of the production auxiliary services: static content, no
# economy state, and staging never connects to them -- it only advertises
# the same strings production Blaze advertises.
POW_CONTENT_HOST=POW_CONTENT_HOST,
POW_HOST=POW_HOST,
# The responder does `sys.path.insert(0, dirname(__file__))` to reach its
# pure sibling modules (`heat2` TDF codec, `fut_account` identity). The copy
# lives elsewhere, so the originals are made importable read-only instead of
# duplicated -- identity MUST stay byte-identical to what LSX and Blaze
# already assert for this persona.
PYTHONPATH=os.path.dirname(BLAZE_SRC),
)
env.pop("FUT_POW", None)
env.pop("FUT_SBC", None)
proc = spawn(["python3", "-u", lay.blaze_script], env, lay.blaze, lay.blaze_log)
wait_tcp([BLAZE_REDIR_PORT, BLAZE_MAIN_PORT, BLAZE_NUCLEUS_PORT], proc,
lay.blaze_log, "staging blaze")
want = (
f"RESPONDER v3 START (redir {BLAZE_REDIR_PORT} / blaze {BLAZE_MAIN_PORT} "
f"/ nucleus {BLAZE_NUCLEUS_PORT})"
)
responder_log = ""
if os.path.exists(lay.blaze_responder_log):
with open(safe_path(lay.blaze_responder_log), errors="replace") as fh:
responder_log = fh.read()
if want not in responder_log:
raise Fatal(
f"staging blaze did not log {want!r} -- it is not the patched copy.\n"
f"{tail(lay.blaze_responder_log)}\n{tail(lay.blaze_log)}"
)
ok(
f"staging blaze ready: redirector {BLAZE_REDIR_PORT}, main {BLAZE_MAIN_PORT}, "
f"nucleus {BLAZE_NUCLEUS_PORT} (pid {proc.pid}); roster {roster_host}; "
f"logged {want!r}"
)
return Launched(
"staging-blaze", proc, lay.blaze_log,
f"{BIND}:{BLAZE_REDIR_PORT},{BLAZE_MAIN_PORT},{BLAZE_NUCLEUS_PORT}",
)
def stop_launched(items: list[Launched]) -> None:
"""Roll back a partial bring-up: signal only the process groups we created."""
for item in reversed(items):
if item.proc.poll() is not None:
continue
try:
os.killpg(os.getpgid(item.proc.pid), signal.SIGTERM)
except OSError:
pass
try:
item.proc.wait(timeout=10)
except subprocess.TimeoutExpired:
try:
os.killpg(os.getpgid(item.proc.pid), signal.SIGKILL)
except OSError:
pass
# --- verification -------------------------------------------------------------------
def host_banner_line(lay: Layout) -> str:
with open(safe_path(lay.host_log), errors="replace") as fh:
hits = [ln.strip() for ln in fh if "sold-experiment=" in ln]
if not hits:
raise Fatal(f"staging utas-host printed no sold-experiment banner\n{tail(lay.host_log)}")
return hits[-1]
def verify(lay: Layout, variant: str) -> None:
line = host_banner_line(lay)
ok(f"host banner: {line}")
if variant == "off":
if "sold-experiment=OFF" not in line:
raise Fatal(f"expected an OFF banner, got {line!r}")
else:
want = f"sold-experiment=ON bidState={variant}"
if want not in line:
raise Fatal(f"banner does not say {want!r}: {line!r}")
status, body = http_get(HOST_PORT, f"/ut/game/{GAME}/tradePile/counts")
ok(f"GET 127.0.0.1:{HOST_PORT}/ut/game/{GAME}/tradePile/counts -> HTTP {status} {body}")
if status != 200:
raise Fatal("staging /tradePile/counts did not answer 200")
status, body = http_get(CORE_PORT, "/health")
ok(f"GET 127.0.0.1:{CORE_PORT}/health -> HTTP {status} {body}")
# Nothing this process ever opened may live under the production state dir.
leaked = []
for fd in os.listdir(f"/proc/{os.getpid()}/fd"):
try:
target = os.readlink(f"/proc/{os.getpid()}/fd/{fd}")
except OSError:
continue
if any(target.startswith(bad) for bad in FORBIDDEN_PATHS):
leaked.append(target)
if leaked:
raise Fatal(f"this script has open handles on production state: {leaked}")
ok(f"no open handle under {FORBIDDEN_PATHS[0]}/")
assert_prod_alive("end of bring-up")
# --- summary ------------------------------------------------------------------------
def cfg_block() -> list[str]:
return [
f"host={ADVERTISE}",
f"blaze_redirector_port={BLAZE_REDIR_PORT}",
f"blaze_main_port={BLAZE_MAIN_PORT}",
]
def print_summary(lay: Layout, variant: str, coins_processed: str, count_mode: str,
roster_host: str, records: list[dict]) -> None:
banner("STAGING STACK IS UP")
rows = [
("staging Core", f"127.0.0.1:{CORE_PORT}", "loopback only; client never talks to it"),
("staging utas-host", f"{BIND}:{HOST_PORT}", "UTAS the client reaches"),
("staging blaze redirector", f"{BIND}:{BLAZE_REDIR_PORT}", "TLS; EA :10041 / :42230"),
("staging blaze main", f"{BIND}:{BLAZE_MAIN_PORT}", "EA :42127"),
("staging blaze nucleus", f"{BIND}:{BLAZE_NUCLEUS_PORT}", "OAuth stub"),
("dead python upstream", f"127.0.0.1:{DEAD_PYTHON_PORT}", "UNBOUND on purpose: fallback fails closed"),
]
print(f" {'SERVICE':<26} {'BIND':<24} NOTE")
for name, bind, note in rows:
print(f" {name:<26} {bind:<24} {note}")
print()
print(f" {'STATE FILE':<26} PATH")
for name, path in (
("Core sqlite", lay.core_db),
("market sqlite", lay.market_db),
("pile sqlite", lay.pile_db),
("identity store", lay.identity),
("clientdata blobs", lay.clientdata),
("content pack", lay.cards),
("identity catalog", lay.catalog),
("patched blaze", lay.blaze_script),
("manifest", lay.manifest),
):
print(f" {name:<26} {path}")
print()
print(f" {'PROCESS':<26} {'PID':<8} {'PGID':<8} LOG")
for rec in records:
print(f" {rec['name']:<26} {rec['pid']:<8} {rec['pgid']:<8} {rec['log']}")
print()
print(f" experiment variant : {variant}")
print(f" coinsProcessed : {coins_processed}")
print(f" count mode : {count_mode}")
print(f" roster host : {roster_host}")
print(f" banner : {host_banner_line(lay)}")
print(f" seeded state : {db_summary(lay)}")
print(f" disposable item to sell : {DISPOSABLE_ITEM} ({DISPOSABLE_CARD})")
banner("OPERATOR: EDIT openfut.cfg ON THE FIFA CLIENT (10.10.0.105)")
print(' File: "/mnt/games/FIFA 17/openfut.cfg" (back it up first:')
print(' cp openfut.cfg openfut.cfg.prod)')
print()
print(" Replace these three lines with EXACTLY:")
print()
for line in cfg_block():
print(f" {line}")
print()
print(" Leave https_port=8443 UNCHANGED (Bridge; no economy state).")
print(" Then RELAUNCH the FIFA 17 client -- a running client caches its UTAS")
print(" session and will not re-auth against a different stack.")
print()
print(" REVERT to production (production values, unchanged on this host):")
print()
print(" host=10.10.0.120")
print(" blaze_redirector_port=42127")
print(" blaze_main_port=42130")
print()
print(" Full detail: docs/SOLD_STAGING_RUNBOOK.md")
print()
print(" Tear down: python3 scripts/sold-staging-down.py")
# --- main ---------------------------------------------------------------------------
def main() -> int:
ap = argparse.ArgumentParser(
description=__doc__.splitlines()[0],
formatter_class=argparse.RawDescriptionHelpFormatter,
)
ap.add_argument("--variant", choices=["highest", "buyNow", "off"], default="highest",
help="bidState token emitted on a sold seller row (default: highest)")
ap.add_argument("--coins-processed", choices=["0", "1"], default="0",
help="coinsProcessed on the sold row (default: 0)")
ap.add_argument("--count-mode", choices=["active", "active_plus_sold"],
default="active",
help="what /tradePile/counts.count reports (default: active)")
ap.add_argument(
"--roster-host",
default=os.environ.get("OPENFUT_ROSTER_HOST", DEFAULT_ROSTER_HOST),
help=f"host:port advertised for roster HTTPS (default: {DEFAULT_ROSTER_HOST})",
)
ap.add_argument("--dir", default=os.environ.get("OPENFUT_SOLD_STAGING_DIR",
DEFAULT_STAGING_DIR),
help=f"staging directory (default: {DEFAULT_STAGING_DIR})")
args = ap.parse_args()
lay = Layout(args.dir)
started: list[Launched] = []
try:
banner("PREFLIGHT (nothing is launched until every check passes)")
step(f"staging dir : {lay.root}")
step(f"repo : {REPO}")
step(f"forbidden ports : {sorted(FORBIDDEN_PORTS)}")
step(f"forbidden paths : {list(FORBIDDEN_PATHS)}")
assert_prod_alive("preflight")
refuse_if_up(lay)
assert_ports_free()
banner("MATERIALISE STAGING DIRECTORY")
materialise(lay)
reset_throwaway_state(lay)
assert_seed_cards_resolvable(lay)
banner("PATCH THE BLAZE RESPONDER COPY")
patch_blaze(lay)
banner("STAGING CORE")
if os.path.exists(lay.core_db):
raise Fatal(f"{lay.core_db} should have been removed by the state reset")
migrate_core(lay)
seed_core_db(lay)
started.append(start_core(lay))
banner("STAGING UTAS-HOST")
started.append(start_host(lay, args.variant, args.coins_processed,
args.count_mode))
banner("STAGING BLAZE")
started.append(start_blaze(lay, args.roster_host))
records = [item.record() for item in started]
for rec in records:
if lay.root not in rec["cmdline"]:
raise Fatal(
f"{rec['name']} pid {rec['pid']} cmdline does not contain the "
f"staging dir -- the down script would refuse to stop it: "
f"{rec['cmdline']!r}"
)
with open(safe_path(lay.manifest), "w") as fh:
json.dump(
{
"created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"staging_dir": lay.root,
"variant": args.variant,
"coins_processed": args.coins_processed,
"count_mode": args.count_mode,
"roster_host": args.roster_host,
"ports": {
"core": CORE_PORT,
"utas_host": HOST_PORT,
"blaze_redirector": BLAZE_REDIR_PORT,
"blaze_main": BLAZE_MAIN_PORT,
"blaze_nucleus": BLAZE_NUCLEUS_PORT,
"dead_python": DEAD_PYTHON_PORT,
},
"state_files": {
"core_db": lay.core_db,
"market_db": lay.market_db,
"pile_db": lay.pile_db,
"identity": lay.identity,
"clientdata": lay.clientdata,
},
"client_cfg": cfg_block(),
"processes": records,
},
fh,
indent=2,
)
ok(f"manifest written: {lay.manifest}")
banner("VERIFY ISOLATION")
verify(lay, args.variant)
print_summary(lay, args.variant, args.coins_processed, args.count_mode,
args.roster_host, records)
return 0
except Fatal as exc:
print(f"\nFATAL: {exc}", file=sys.stderr)
if started:
print(" rolling back the partial bring-up...", file=sys.stderr)
stop_launched(started)
print(" rolled back (only this script's own process groups were "
"signalled)", file=sys.stderr)
return 1
if __name__ == "__main__":
sys.exit(main())
Reference in New Issue View Git Blame Copy Permalink