funman300/OpenFUT
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(staging): let staging serve the operator's real club, not just the fixture

Browse Source 
The operator could not field a starting XI because staging has only ever held the
14-item synthetic fixture (11 auto-picked starters, one disposable, two kits). The
real club -- the 1986-item CAGE import, 29,843,976 coins -- was never lost, but it
sits in `/home/alex/openfut-promotion/state`, which BOTH staging lifecycle scripts
list in FORBIDDEN_PATHS and refuse to open. That guard is correct and stays.

Worth recording while looking for the club: the LIVE production Core container
serves an EMPTY database (0 owned cards, schema predating even the game_id column).
The real club is not being served anywhere right now; it exists as state on disk.
So restoring it into staging is not a convenience, it is the only way to play it.

Two pieces:

`scripts/club-snapshot.py` is the ONE place allowed to read production state, and
it is read-only by construction: the Core database is opened `mode=ro` and copied
with sqlite's online backup API (a plain file copy can tear a database with a hot
WAL), every destination is asserted to be outside the production directory before
anything is opened for writing, and the sha256 of every source is compared before
and after -- a mismatch aborts, because that would mean the snapshot modified
production. It then proves the copy is faithful (same counts, coins, squad) and
that the identity store maps EVERY owned card to a wire id, since an unmapped card
would reappear under a freshly minted id and break the client's cached squad.

`sold-staging-up.py --club real` installs that snapshot. It is installed BEFORE
Core first starts, so Core migrates the copy forward from schema v19 through
match_completions, squad managers and kit assignments. Seller A is then already
present -- it IS the imported persona -- so only Buyer B is seeded, the kit
fixtures are attached to the real club so the kit work stays exercisable, and the
real squad is left alone. The up script still never reads production state: the
snapshot lives outside it, which is precisely what makes `--club real` compatible
with the `safe_path()` refusal.

The resolvability preflight now covers whichever club will actually be served. This
is the check that matters most for the real one: Core does not fail on an owned card
whose definition is missing, it silently filter_map-drops it, so a gap shows up as
an EMPTY club with all 1986 rows still in the database. Verified: all 1712 distinct
card ids resolve in both the content pack and the identity catalog, 0 missing.

`sold-staging-seed-squad.py` now REFUSES to run when the manifest says the real club
is installed. `PUT /squad/0` is a full replacement, so the fixture seeder would have
overwritten the operator's own lineup with an auto-picked XI -- destructive and not
recoverable in place. `--show` still works in every mode; `--force` overrides.

Verified end to end against the restored club: /club 29,843,976 coins, /collection
1988, 1966 players + 2 kits served over the UTAS wire, squad 'OpenFUT' (f433) rated
90 with 11 players carrying contract 7 / fitness 99, and every wire id stable from
the snapshot identity store. The fixture path was re-run afterwards and still seeds
exactly 14 items, so the SOLD experiment is unaffected.
This commit is contained in:
funman300
2026-08-21 16:35:06 +00:00
parent dddcfb917c
commit de747b79e6
3 changed files with 534 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/sold-staging-up.py
+190 -56
View File
@@ -39,6 +39,15 @@ default) and advertises the production POW content/API hosts
The client must resolve the roster hostname to this server without changing the
advertised URL.
WHICH CLUB the seller plays with is chosen by `--club`:
* `fixture` (default) seeds a 14-item synthetic club -- 11 starters, one
disposable item to sell, two kits -- which is all the SOLD experiment needs;
* `real` installs the operator's own imported club from the snapshot produced by
`scripts/club-snapshot.py`, including its coins, its squad and the identity
store that keeps every item's wire id stable. This script still never reads
production state: the snapshot lives outside it, which is what makes the
`safe_path()` refusal above compatible with playing the real club.
python3 scripts/sold-staging-up.py --variant highest
python3 scripts/sold-staging-up.py --variant buyNow --coins-processed 1 \
--count-mode active_plus_sold
@@ -119,6 +128,16 @@ CONTENT_SRC = "/home/alex/openfut-post-p1/staging/emit/content"
CARDS_NAME = "fifa17-production-cards.json"
CATALOG_NAME = "fifa17-production-catalog.json"
# The operator's REAL club (the 1986-item CAGE import), as staged by
# scripts/club-snapshot.py. That snapshot lives OUTSIDE the production state
# directory precisely so this script never has to read production state:
# FORBIDDEN_PATHS stays absolute and safe_path() still refuses the live directory.
CLUB_SNAPSHOT_DIR = "/home/alex/openfut-club-snapshot"
SNAP_CORE_DB = "core.db"
SNAP_IDENTITY = "identity.json"
SNAP_CLIENTDATA = "clientdata.json"
SNAP_MANIFEST = "snapshot.json"
GAME = "fifa17"
PERSONA_ID = "33068179"
PERSONA_NAME = "CAGE"
@@ -489,30 +508,89 @@ def materialise(lay: Layout) -> None:
ok("added ownership-backed home/away kit fixtures from fcc_kitcards")
def assert_seed_cards_resolvable(lay: Layout) -> None:
def install_real_club(lay: Layout) -> dict:
"""Install the operator's real club from the snapshot in place of the fixture.
The snapshot database is a schema generation behind (it predates
match_completions, squad managers and kit assignments), so it is installed
BEFORE migrate_core and Core brings the COPY forward. Production's own files
are never opened here: scripts/club-snapshot.py already took them out, which is
why this script can keep refusing the production state directory outright.
"""
manifest_path = os.path.join(CLUB_SNAPSHOT_DIR, SNAP_MANIFEST)
if not os.path.isfile(manifest_path):
raise Fatal(
f"no club snapshot at {CLUB_SNAPSHOT_DIR}.\n"
" Take one first (it is read-only on production state):\n"
" python3 scripts/club-snapshot.py"
)
with open(manifest_path) as fh:
manifest = json.load(fh)
for name, dst in ((SNAP_CORE_DB, lay.core_db),
(SNAP_IDENTITY, lay.identity),
(SNAP_CLIENTDATA, lay.clientdata)):
src = os.path.join(CLUB_SNAPSHOT_DIR, name)
if not os.path.isfile(src):
raise Fatal(f"club snapshot is incomplete: missing {src}")
shutil.copy2(src, safe_path(dst))
os.chmod(safe_path(dst), 0o644)
club = manifest["club"]
ok(
f"installed the real club from {CLUB_SNAPSHOT_DIR} (taken "
f"{manifest['taken_at']}): {club['username']}, {club['owned_cards']} items, "
f"{club['coins']:,} coins, schema v{club['schema_version']}"
)
ok(
"wire ids come from the snapshot identity store, so item ids the client "
f"already cached still resolve ({manifest['identity_rows']} rows, "
f"watermarks {manifest['watermarks']})"
)
return club
def assert_seed_cards_resolvable(lay: Layout, real_club: dict | None) -> None:
"""Core's content preflight rejects any owned card whose card_id is not a loaded
CardDefinition, and the host refuses to shape a /club item with no catalog
identity. Prove BOTH memberships now, not via a startup crash later."""
identity. Prove BOTH memberships now, not via a startup crash later.
For the real club this is the check that matters most: a card_id missing from
the pack is not an error at read time, it is silently filter_map-dropped, so the
symptom is an EMPTY /collection with all the rows still sitting in the database.
"""
with open(safe_path(lay.cards)) as fh:
pack_ids = {c["id"] for c in json.load(fh)}
with open(safe_path(lay.catalog)) as fh:
catalog_ids = set(json.load(fh)["cards"])
wanted = (
[card for _, card in SELLER_SQUAD_CARDS]
+ [DISPOSABLE_CARD]
+ [card for _, _, card, _ in STAGING_KITS]
)
missing_pack = sorted(set(wanted) - pack_ids)
missing_cat = sorted(set(wanted) - catalog_ids)
if real_club is None:
wanted = set(
[card for _, card in SELLER_SQUAD_CARDS]
+ [DISPOSABLE_CARD]
+ [card for _, _, card, _ in STAGING_KITS]
)
what = f"all {len(wanted)} fixture seed card ids"
else:
conn = sqlite3.connect(f"file:{safe_path(lay.core_db)}?mode=ro", uri=True)
try:
wanted = {row[0] for row in
conn.execute("SELECT DISTINCT card_id FROM owned_cards")}
finally:
conn.close()
wanted |= {card for _, _, card, _ in STAGING_KITS}
what = (f"all {len(wanted)} distinct card ids owned by the real club "
"(plus the kit fixtures)")
missing_pack = sorted(wanted - pack_ids)
missing_cat = sorted(wanted - catalog_ids)
if missing_pack or missing_cat:
raise Fatal(
"seed card ids are not resolvable -- Core or the host would fail at "
f"startup.\n absent from content pack: {missing_pack}"
f"\n absent from identity catalog: {missing_cat}"
"card ids are not resolvable. Core drops an owned card whose definition "
"is missing instead of failing, so this would surface as an EMPTY club "
"with every row still in the database.\n"
f" absent from content pack ({len(missing_pack)}): {missing_pack[:10]}\n"
f" absent from identity catalog ({len(missing_cat)}): {missing_cat[:10]}"
)
ok(
f"all {len(wanted)} seed card ids present in BOTH the content pack "
f"({len(pack_ids)} defs) and the identity catalog ({len(catalog_ids)} entries)"
f"{what} present in BOTH the content pack ({len(pack_ids)} defs) and the "
f"identity catalog ({len(catalog_ids)} entries)"
)
@@ -607,73 +685,96 @@ def verify_blaze_patch(lay: Layout) -> None:
# --- seeding ------------------------------------------------------------------------
def seed_core_db(lay: Layout) -> None:
"""Two identities by direct SQL, against the schema Core just migrated.
def seed_core_db(lay: Layout, real_club: dict | None) -> None:
"""Seed the identities the experiment needs, against the schema Core just
migrated.
Column sets are from openfut-core/migrations/0001_initial.sql plus 0016's
profiles.game_id. Seller A carries game_id `fifa17` so the retail client (which
sends X-OpenFUT-Game: fifa17) resolves to it; Buyer B is parked on its own
game_id so it can never shadow the seller as the active fifa17 profile.
With the real club installed, Seller A already exists -- it IS the imported
persona, with its own club id, coins, items and squad -- so only Buyer B is
added, and the kit fixtures are attached to the real club so the kit work stays
exercisable. The real squad is never touched: it is the operator's own.
"""
conn = sqlite3.connect(safe_path(lay.core_db), timeout=15)
try:
conn.execute("PRAGMA busy_timeout = 15000")
with conn:
profiles = [(BUYER_PROFILE, "BUYER-B", TS, TS, BUYER_GAME)]
clubs = [(BUYER_CLUB, BUYER_PROFILE, "Buyer B FC", BUYER_COINS, TS, TS)]
if real_club is None:
profiles.insert(0, (SELLER_PROFILE, PERSONA_NAME, TS, TS, GAME))
clubs.insert(0, (SELLER_CLUB, SELLER_PROFILE, f"{PERSONA_NAME} FC",
SELLER_COINS, TS, TS))
conn.executemany(
"INSERT INTO profiles (id, username, level, xp, created_at, "
"updated_at, game_id) VALUES (?, ?, 1, 0, ?, ?, ?)",
[
(SELLER_PROFILE, PERSONA_NAME, TS, TS, GAME),
(BUYER_PROFILE, "BUYER-B", TS, TS, BUYER_GAME),
],
profiles,
)
conn.executemany(
"INSERT INTO clubs (id, profile_id, name, coins, level, created_at, "
"updated_at) VALUES (?, ?, ?, ?, 1, ?, ?)",
[
(SELLER_CLUB, SELLER_PROFILE, f"{PERSONA_NAME} FC", SELLER_COINS,
TS, TS),
(BUYER_CLUB, BUYER_PROFILE, "Buyer B FC", BUYER_COINS, TS, TS),
],
clubs,
)
seller_club = SELLER_CLUB if real_club is None else real_club["club_id"]
owned = [
(owned_id, seller_club, card_id, TS)
for _slot, owned_id, card_id, _resource_id in STAGING_KITS
]
if real_club is None:
owned = (
[(item, SELLER_CLUB, card, TS) for item, card in SELLER_SQUAD_CARDS]
+ [(DISPOSABLE_ITEM, SELLER_CLUB, DISPOSABLE_CARD, TS)]
+ owned
)
conn.executemany(
"INSERT INTO owned_cards (id, club_id, card_id, is_loan, "
"acquired_at) VALUES (?, ?, ?, 0, ?)",
[(item, SELLER_CLUB, card, TS) for item, card in SELLER_SQUAD_CARDS]
+ [(DISPOSABLE_ITEM, SELLER_CLUB, DISPOSABLE_CARD, TS)]
+ [
(owned_id, SELLER_CLUB, card_id, TS)
for _slot, owned_id, card_id, _resource_id in STAGING_KITS
],
)
conn.execute(
"INSERT INTO squads (id, club_id, name, formation, created_at, "
"updated_at) VALUES (?, ?, ?, ?, ?, ?)",
(SELLER_SQUAD, SELLER_CLUB, "Staging XI", "4-3-3", TS, TS),
)
conn.executemany(
"INSERT INTO squad_players (id, squad_id, owned_card_id, "
"position_index, is_captain, is_on_bench) VALUES (?, ?, ?, ?, ?, 0)",
[
(f"sp-{idx}", SELLER_SQUAD, item, idx, 1 if idx == 0 else 0)
for idx, (item, _) in enumerate(SELLER_SQUAD_CARDS)
],
owned,
)
if real_club is None:
conn.execute(
"INSERT INTO squads (id, club_id, name, formation, created_at, "
"updated_at) VALUES (?, ?, ?, ?, ?, ?)",
(SELLER_SQUAD, SELLER_CLUB, "Staging XI", "4-3-3", TS, TS),
)
conn.executemany(
"INSERT INTO squad_players (id, squad_id, owned_card_id, "
"position_index, is_captain, is_on_bench) VALUES (?, ?, ?, ?, ?, 0)",
[
(f"sp-{idx}", SELLER_SQUAD, item, idx, 1 if idx == 0 else 0)
for idx, (item, _) in enumerate(SELLER_SQUAD_CARDS)
],
)
conn.executemany(
"INSERT INTO club_kit_assignments (club_id, slot, owned_card_id, updated_at) "
"VALUES (?, ?, ?, ?)",
[
(SELLER_CLUB, slot, owned_id, TS)
(seller_club, slot, owned_id, TS)
for slot, owned_id, _card_id, _resource_id in STAGING_KITS
],
)
finally:
conn.close()
ok(
f"seeded Seller A ({PERSONA_NAME}, persona {PERSONA_ID}, {SELLER_COINS} coins, "
f"{len(SELLER_SQUAD_CARDS)} starters + 1 disposable + "
f"{len(STAGING_KITS)} active kits) and Buyer B ({BUYER_COINS} coins)"
)
if real_club is None:
ok(
f"seeded Seller A ({PERSONA_NAME}, persona {PERSONA_ID}, {SELLER_COINS} "
f"coins, {len(SELLER_SQUAD_CARDS)} starters + 1 disposable + "
f"{len(STAGING_KITS)} active kits) and Buyer B ({BUYER_COINS} coins)"
)
else:
ok(
f"kept the real club untouched ({real_club['owned_cards']} items, "
f"{real_club['coins']:,} coins, squad {real_club['squads'][0]['name']!r} "
f"with {real_club['squad_players']} players); added "
f"{len(STAGING_KITS)} active kits and Buyer B ({BUYER_COINS} coins)"
)
def db_summary(lay: Layout) -> str:
@@ -919,8 +1020,23 @@ def cfg_block() -> list[str]:
def print_summary(lay: Layout, variant: str, coins_processed: str, count_mode: str,
roster_host: str, records: list[dict]) -> None:
roster_host: str, records: list[dict],
real_club: dict | None) -> None:
banner("STAGING STACK IS UP")
if real_club is None:
print(" club: the 14-item synthetic fixture "
"(--club real installs the operator's own)")
else:
squad = real_club["squads"][0] if real_club["squads"] else None
print(f" club: THE REAL ONE -- {real_club['username']} "
f"({real_club['club_name']}), {real_club['owned_cards']} items, "
f"{real_club['coins']:,} coins")
if squad is not None:
print(f" squad {squad['name']!r} ({squad['formation']}) with "
f"{real_club['squad_players']} players")
print(" DO NOT run sold-staging-seed-squad.py: it would overwrite "
"this squad with the fixture XI")
print()
rows = [
("staging Core", f"127.0.0.1:{CORE_PORT}", "loopback only; client never talks to it"),
("staging utas-host", f"{BIND}:{HOST_PORT}", "UTAS the client reaches"),
@@ -1009,6 +1125,12 @@ def main() -> int:
ap.add_argument("--dir", default=os.environ.get("OPENFUT_SOLD_STAGING_DIR",
DEFAULT_STAGING_DIR),
help=f"staging directory (default: {DEFAULT_STAGING_DIR})")
ap.add_argument(
"--club", choices=["fixture", "real"], default="fixture",
help="which club the seller plays with: the 14-item synthetic fixture "
"(default) or the operator's real imported club, installed from the "
"snapshot taken by scripts/club-snapshot.py",
)
args = ap.parse_args()
lay = Layout(args.dir)
@@ -1019,6 +1141,7 @@ def main() -> int:
step(f"repo : {REPO}")
step(f"forbidden ports : {sorted(FORBIDDEN_PORTS)}")
step(f"forbidden paths : {list(FORBIDDEN_PATHS)}")
step(f"club : {args.club}")
assert_prod_alive("preflight")
refuse_if_up(lay)
assert_ports_free()
@@ -1026,16 +1149,25 @@ def main() -> int:
banner("MATERIALISE STAGING DIRECTORY")
materialise(lay)
reset_throwaway_state(lay)
assert_seed_cards_resolvable(lay)
# The real club is installed BEFORE Core first runs, so Core migrates the
# snapshot forward (it is a schema generation behind) rather than being
# handed a database it has already opened.
real_club = install_real_club(lay) if args.club == "real" else None
assert_seed_cards_resolvable(lay, real_club)
banner("PATCH THE BLAZE RESPONDER COPY")
patch_blaze(lay)
banner("STAGING CORE")
if os.path.exists(lay.core_db):
raise Fatal(f"{lay.core_db} should have been removed by the state reset")
if real_club is None:
if os.path.exists(lay.core_db):
raise Fatal(
f"{lay.core_db} should have been removed by the state reset"
)
elif not os.path.exists(lay.core_db):
raise Fatal(f"{lay.core_db} should have been installed from the snapshot")
migrate_core(lay)
seed_core_db(lay)
seed_core_db(lay, real_club)
started.append(start_core(lay))
banner("STAGING UTAS-HOST")
@@ -1059,6 +1191,8 @@ def main() -> int:
"created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"staging_dir": lay.root,
"variant": args.variant,
"club": args.club,
"real_club": real_club,
"coins_processed": args.coins_processed,
"count_mode": args.count_mode,
"roster_host": args.roster_host,
@@ -1089,7 +1223,7 @@ def main() -> int:
verify(lay, args.variant)
print_summary(lay, args.variant, args.coins_processed, args.count_mode,
args.roster_host, records)
args.roster_host, records, real_club)
return 0
except ProductionError as exc:
print(f"\nFATAL: {exc}", file=sys.stderr)