fifa17-recon: card-family enumeration round -- managers cracked, consumables need no ids
11-agent round, every investigation adversarially reviewed. The headline is that this
was never five id hunts: the client's database is resident in ordinary heap as a
self-describing catalog of bit-packed fixed-stride row arrays, walkable READ-ONLY, so
the id sets fall out at zero cost in human club visits.
MEASURED:
managercards carddbid 1000001..1001455, assetid == carddbid; two agents using two
different block locators agreed 417/417 (docs/managercards_ids.txt). This is why
the earlier sweeps of 1..5000 and 6000..8000 were silent.
staff bands: headcoach 2000004+, fitnesscoach 3000019+, physio 4000002+,
gkcoach 9000001+, corroborated by the four miss-fallback assetids hard-coded in
FUN_180141660 each landing inside its own table's decoded range.
all four coach branches write a LOUD miss-fill: firstname/lastname "DB Error",
rating 0x32, rare 1, attrs[0] 0xf, plus a table-unique assetid. Managers write
none, so a wrong manager id is silent and a wrong coach id labels itself.
consumables have NO table and NO id space: cardtype 6 has no arm in the merge,
FUN_18013f4d0's only callees are a range clamp and an enum map, and every string
is a hardcoded FUT_CONSUMABLE_* literal. A contract is three JSON keys.
the ?type= taxonomy is 29 explicit arms plus a default: badge 11, kit 12, stadium
13, ball 14, equippables 15, leaguelogos 16, misc 26. club/stats kits and
badgeDBid are PLAIN COUNTS, not ids.
fancards is a boolean column of the fixtures table; newcards is FUT atom 0x1d7.
NEITHER is a card family, so two of the five hunts never existed.
REFUTED, and worth keeping: the live table-directory walk was off by one entry
(descriptor for table T is at entry-0x20, not entry+0x08), which had mislabelled
managercards as factory_teams and shifted every column count. managercards names are
32-bit string-pool offsets and the pool was never located -- we get ids, not names,
and we do not need names because the client supplies them.
TRAPS RECORDED: rareflag=1 silently converts a Player Fitness card (219) into Squad
Fitness, and fut_store._item() hardcodes rareflag 1 on every item.
Four proposed club-item sweep windows were killed by review as invariant by
construction: with no merge arm there is no miss-fill, so every id yields a
byte-identical record and the probe cannot discriminate. That is a wasted human action
correctly caught before it cost one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUT92pz6RWKih9dSr8ZpxW
This commit is contained in:
funman300
@@ -0,0 +1,417 @@
|
||||
1000001
|
||||
1000004
|
||||
1000005
|
||||
1000012
|
||||
1000017
|
||||
1000018
|
||||
1000019
|
||||
1000020
|
||||
1000022
|
||||
1000023
|
||||
1000026
|
||||
1000028
|
||||
1000031
|
||||
1000032
|
||||
1000033
|
||||
1000034
|
||||
1000035
|
||||
1000038
|
||||
1000041
|
||||
1000052
|
||||
1000054
|
||||
1000057
|
||||
1000058
|
||||
1000063
|
||||
1000065
|
||||
1000066
|
||||
1000067
|
||||
1000068
|
||||
1000072
|
||||
1000073
|
||||
1000074
|
||||
1000075
|
||||
1000081
|
||||
1000083
|
||||
1000087
|
||||
1000089
|
||||
1000090
|
||||
1000096
|
||||
1000097
|
||||
1000102
|
||||
1000104
|
||||
1000107
|
||||
1000109
|
||||
1000110
|
||||
1000114
|
||||
1000115
|
||||
1000118
|
||||
1000119
|
||||
1000121
|
||||
1000124
|
||||
1000127
|
||||
1000128
|
||||
1000129
|
||||
1000133
|
||||
1000135
|
||||
1000138
|
||||
1000146
|
||||
1000148
|
||||
1000156
|
||||
1000158
|
||||
1000161
|
||||
1000165
|
||||
1000167
|
||||
1000172
|
||||
1000174
|
||||
1000175
|
||||
1000178
|
||||
1000181
|
||||
1000183
|
||||
1000186
|
||||
1000196
|
||||
1000198
|
||||
1000199
|
||||
1000202
|
||||
1000203
|
||||
1000206
|
||||
1000208
|
||||
1000213
|
||||
1000214
|
||||
1000218
|
||||
1000219
|
||||
1000221
|
||||
1000222
|
||||
1000227
|
||||
1000230
|
||||
1000243
|
||||
1000247
|
||||
1000252
|
||||
1000256
|
||||
1000257
|
||||
1000259
|
||||
1000260
|
||||
1000263
|
||||
1000264
|
||||
1000266
|
||||
1000270
|
||||
1000272
|
||||
1000273
|
||||
1000275
|
||||
1000276
|
||||
1000281
|
||||
1000292
|
||||
1000302
|
||||
1000309
|
||||
1000314
|
||||
1000316
|
||||
1000321
|
||||
1000322
|
||||
1000324
|
||||
1000340
|
||||
1000343
|
||||
1000345
|
||||
1000353
|
||||
1000361
|
||||
1000380
|
||||
1000414
|
||||
1000417
|
||||
1000433
|
||||
1000445
|
||||
1000451
|
||||
1000452
|
||||
1000462
|
||||
1000470
|
||||
1000495
|
||||
1000496
|
||||
1000497
|
||||
1000500
|
||||
1000503
|
||||
1000505
|
||||
1000506
|
||||
1000507
|
||||
1000509
|
||||
1000512
|
||||
1000517
|
||||
1000519
|
||||
1000522
|
||||
1000523
|
||||
1000527
|
||||
1000529
|
||||
1000530
|
||||
1000532
|
||||
1000534
|
||||
1000535
|
||||
1000543
|
||||
1000544
|
||||
1000546
|
||||
1000548
|
||||
1000550
|
||||
1000551
|
||||
1000564
|
||||
1000599
|
||||
1000600
|
||||
1000603
|
||||
1000608
|
||||
1000609
|
||||
1000611
|
||||
1000612
|
||||
1000616
|
||||
1000622
|
||||
1000623
|
||||
1000624
|
||||
1000628
|
||||
1000630
|
||||
1000656
|
||||
1000664
|
||||
1000666
|
||||
1000680
|
||||
1000694
|
||||
1000696
|
||||
1000698
|
||||
1000702
|
||||
1000705
|
||||
1000712
|
||||
1000720
|
||||
1000726
|
||||
1000731
|
||||
1000732
|
||||
1000733
|
||||
1000743
|
||||
1000758
|
||||
1000762
|
||||
1000763
|
||||
1000789
|
||||
1000794
|
||||
1000800
|
||||
1000802
|
||||
1000811
|
||||
1000815
|
||||
1000817
|
||||
1000819
|
||||
1000825
|
||||
1000830
|
||||
1000832
|
||||
1000833
|
||||
1000836
|
||||
1000842
|
||||
1000846
|
||||
1000857
|
||||
1000858
|
||||
1000861
|
||||
1000871
|
||||
1000873
|
||||
1000874
|
||||
1000878
|
||||
1000879
|
||||
1000895
|
||||
1000899
|
||||
1000902
|
||||
1000903
|
||||
1000907
|
||||
1000913
|
||||
1000914
|
||||
1000915
|
||||
1000916
|
||||
1000920
|
||||
1000924
|
||||
1000925
|
||||
1000927
|
||||
1000929
|
||||
1000940
|
||||
1000948
|
||||
1000954
|
||||
1000957
|
||||
1000969
|
||||
1000972
|
||||
1000982
|
||||
1001008
|
||||
1001010
|
||||
1001021
|
||||
1001032
|
||||
1001037
|
||||
1001038
|
||||
1001039
|
||||
1001045
|
||||
1001046
|
||||
1001047
|
||||
1001052
|
||||
1001053
|
||||
1001072
|
||||
1001081
|
||||
1001099
|
||||
1001104
|
||||
1001107
|
||||
1001113
|
||||
1001119
|
||||
1001121
|
||||
1001122
|
||||
1001129
|
||||
1001133
|
||||
1001135
|
||||
1001137
|
||||
1001156
|
||||
1001167
|
||||
1001178
|
||||
1001181
|
||||
1001182
|
||||
1001184
|
||||
1001228
|
||||
1001229
|
||||
1001247
|
||||
1001257
|
||||
1001269
|
||||
1001292
|
||||
1001299
|
||||
1001303
|
||||
1001309
|
||||
1001311
|
||||
1001320
|
||||
1001326
|
||||
1001331
|
||||
1001333
|
||||
1001334
|
||||
1001335
|
||||
1001338
|
||||
1001339
|
||||
1001340
|
||||
1001341
|
||||
1001342
|
||||
1001345
|
||||
1001346
|
||||
1001347
|
||||
1001348
|
||||
1001354
|
||||
1001360
|
||||
1001362
|
||||
1001369
|
||||
1001370
|
||||
1001371
|
||||
1001373
|
||||
1001374
|
||||
1001376
|
||||
1001390
|
||||
1001393
|
||||
1001396
|
||||
1001408
|
||||
1001409
|
||||
1001421
|
||||
1001432
|
||||
1001433
|
||||
1001434
|
||||
1001435
|
||||
1001436
|
||||
1001437
|
||||
1001438
|
||||
1001439
|
||||
1001440
|
||||
1001441
|
||||
1001442
|
||||
1001443
|
||||
1001444
|
||||
1001445
|
||||
1001446
|
||||
1001447
|
||||
1001448
|
||||
1001449
|
||||
1001450
|
||||
1001451
|
||||
1001452
|
||||
1001453
|
||||
1001454
|
||||
1001455
|
||||
1001456
|
||||
1001457
|
||||
1001458
|
||||
1001459
|
||||
1001460
|
||||
1001461
|
||||
1001462
|
||||
1001463
|
||||
1001464
|
||||
1001465
|
||||
1001466
|
||||
1001467
|
||||
1001468
|
||||
1001469
|
||||
1001470
|
||||
1001471
|
||||
1001472
|
||||
1001473
|
||||
1001474
|
||||
1001475
|
||||
1001476
|
||||
1001477
|
||||
1001478
|
||||
1001479
|
||||
1001480
|
||||
1001481
|
||||
1001482
|
||||
1001483
|
||||
1001484
|
||||
1001485
|
||||
1001486
|
||||
1001487
|
||||
1001488
|
||||
1001489
|
||||
1001490
|
||||
1001491
|
||||
1001492
|
||||
1001493
|
||||
1001494
|
||||
1001495
|
||||
1001496
|
||||
1001497
|
||||
1001498
|
||||
1001499
|
||||
1001500
|
||||
1001501
|
||||
1001502
|
||||
1001504
|
||||
1001505
|
||||
1001506
|
||||
1001507
|
||||
1001508
|
||||
1001509
|
||||
1001510
|
||||
1001511
|
||||
1001512
|
||||
1001513
|
||||
1001514
|
||||
1001515
|
||||
1001516
|
||||
1001517
|
||||
1001518
|
||||
1001519
|
||||
1001520
|
||||
1001521
|
||||
1001522
|
||||
1001523
|
||||
1001524
|
||||
1001525
|
||||
1001526
|
||||
1001527
|
||||
1001528
|
||||
1001529
|
||||
1001530
|
||||
1001531
|
||||
1001532
|
||||
1001533
|
||||
1001534
|
||||
1001535
|
||||
1001536
|
||||
1001537
|
||||
1001538
|
||||
1001539
|
||||
1001540
|
||||
1001541
|
||||
1001542
|
||||
1001543
|
||||
1001544
|
||||
1001545
|
||||
1001546
|
||||
1001547
|
||||
1001548
|
||||
1001549
|
||||
1001550
|
||||
1001551
|
||||
1001552
|
||||
@@ -0,0 +1,225 @@
|
||||
# Card families: the 2026-08-04 enumeration round
|
||||
|
||||
Workflow wf_3afc63b3-994, 11 agents, every investigation paired with an adversarial
|
||||
reviewer. Verbatim synthesis. Where a reviewer refuted an investigator the refutation
|
||||
is recorded rather than smoothed over, because several of those corrections are the
|
||||
reason the plan reads the way it does.
|
||||
|
||||
# The next round: one enumeration, not five id hunts
|
||||
|
||||
Written 2026-08-04. Facts below are attributed to the investigation that produced them and to the reviewer that confirmed or refuted them. Anything not settled is marked TODO/CONFIRM.
|
||||
|
||||
Live state at the time of writing, checked directly: `FIFA17.exe` is running as **pid 52703**, `utas_server.py` is running as **pid 37291**, and `/tmp/fut_id_sweep` is **empty (0 bytes)**, so no sweep is armed and the real club is being served. That matters for the ordering below: the single most valuable measurement costs zero human actions and the window for it is open right now.
|
||||
|
||||
---
|
||||
|
||||
## 1. What was established
|
||||
|
||||
**Managers.** The investigator re-derived `FUN_1801356c0` (len 1587) verbatim and both the reviewer and the taxonomy pair confirmed it independently: `SELECT firstname,lastname,assetid,value,talkrating,negotiation,rare FROM managercards WHERE carddbid == *(u32*)(record+0x18)`, key read **raw**, no `&0xffffff`, no else-branch, therefore no miss-fill. Writes: firstname to +0xb8 (0x10 bytes), lastname to +0xc8 (0x15), assetid to +0x20, `value` to +0xb4 (this is the rating byte), talkrating +0xe2, negotiation +0xe3, `rare==1` to +0x58. **Where the reviewer refuted the investigator:** the investigator's live table-directory walk was off by exactly one entry (descriptor for table T is at `entry-0x20`, not `entry+0x08`), so its headline "managercards has 62 columns in a 128-byte row" was actually `factory_teams`; the real row is 20 bytes, 10 columns. Its per-table column counts (18/107/9/11) were all shifted. Its 1M band was a guess that happened to land right. The reviewer then measured the ids.
|
||||
|
||||
**The four coach tables.** Verified and reproduced by its reviewer: all four inline branches of `FUN_180141660` (len 7937) write a loud miss-fill on `rowcount < 1`: firstname and lastname both `"DB Error"`, rating +0xb4 = 0x32, rare +0x58 = 1, attrs[0] +0x98 = 0xf, and a table-unique assetid at +0x20 (headcoach 2000148, fitnesscoach 3000259, physio 4000146, gkcoach 9000258, each appearing exactly once in the whole binary, at its own branch). This is the single most useful fact of the round: **a wrong coach id is self-labelling, a wrong manager id is silent.** The jump table (1, 2, 3, 4, 5, 10 live; 6/7/8/9 to the shared tail) was read dword by dword by both. **Where the reviewer refuted the investigator:** "the tail always runs" is false, `FUN_180141660` returns immediately if `record+8` is null; `FUN_1801362e0` is a partial overwrite, not a sniffer that derives cardtype from the id alone; and the investigator missed that the headcoach and gkcoach branches also write the attribute array, so `card_identity_probe`'s `attrs` column is not trustworthy for staff either (and `attrs[0]==15` is a free second miss fingerprint).
|
||||
|
||||
**Consumables.** Confirmed structurally by three independent routes: cardtype 6 has no arm in the merge; `FUN_18013f4d0` (len 8354) has exactly two callees, a range clamp and an enum map, and touches no DB handle; and every string the UI shows is a hardcoded `FUT_CONSUMABLE_*` literal referenced from exactly one function, `FUN_1801bfac0`. **Consumables have no table and no id space to crack.** Category comes from `cardsubtypeid` alone into record+0xb8; artwork is a fixed client-side constant per category, so `resourceId` never reaches the screen. **Where the reviewer refuted the investigator:** the dead-zone list was wrong twice (137-200 is cardtype 0, never reaches `FUN_18013f4d0` at all, and 145-150 inside it is cardtype 9; the 87-88 gap is really 87-90); the formation map's 8th entry is 0x06 not 0x00; the play-style artwork field is record+0xbe not +0xc1; the atom-name table index is `0x1802d2760 + 8*atom` with no `+6`; and there are 24 `FUT_CONSUMABLE_*` strings, not 26. None of that touches the headline. Two real traps survive: `rareflag=1` silently converts a Player Fitness card (219) into a Squad Fitness card, and `fut_store._item()` hardcodes `"rareflag": 1` on every item (confirmed at line 56).
|
||||
|
||||
**Club items.** The `type=` taxonomy is real and complete: 29 explicit arms in `FUN_18012ec50` plus a default that returns `any`, including badge(11), kit(12), stadium(13), ball(14), equippables(15), leaguelogos(16), misc(26). The `equippables` value has a second selector at `this+0x30` (10 badge, 11 kit, 21 stadium, 22 ball). `state=` carries activeBadge/activeHomeKit/activeAwayKit/activeBall/activeStadium. `club/stats` `kits` and `badgeDBid` are consumed as **plain counts** by the same getter and publisher shape as the already-live `PLAYERS_EMPLOYED` rows, so that family needs no id at all. The registry correction is solid and the reviewer re-did it properly: `GET ut/%s/club` is **ClubSearch**, not `GetClubInfo`; `GetClubInfo` is on `ut/%s/user/list`. **Where the reviewer refuted the investigator:** the registry census was 47 of 125 records (the conclusion survived a full re-walk); "`misc` belongs to the market vocabulary" is exactly backwards; the "no `RS4:FutClubSearchServerResponse` therefore it reuses another class" argument is invalid, because `FutClubStatsServerResponse` also does not exist and club/stats works today; and **all four proposed club-item sweep windows are invariant by construction** and would be wasted human actions.
|
||||
|
||||
**Taxonomy, and the finding that reorders the round.** `newcards` is FUT atom 0x1d7, not a table. `fancards` is a boolean column of the `fixtures` table, next to `fanflags`. Neither is a card family. And the client's database is **resident in the ordinary process heap as a self-describing catalog of bit-packed, fixed-stride row arrays, walkable entirely read-only.** Table descriptor 0x28 bytes, column descriptor 0x30 bytes, both anchored on the shared string-class constant `0x07C20760`; layout block header carries the row count, entries are `{bitOffset, colTag, width, type}`, the row pointer sits at `hdr-0x48`, values are `raw + min`. About 150 tables with complete schemas. **Where the reviewer refuted the investigator:** "no caller omits the WHERE" is false (7 functions call FROM and never WHERE; `FUN_180042440` literally enumerates `fcc_myclubscategories` with no predicate, so the client itself does full-table scans); the "one staff id space at 4,000,000 with stride 4, tiled by residue mod 4" is wrong (four disjoint bases); `fitnesscoachcards` does have a layout block; and the header-signature locator found 90 blocks for 151 tables, while a tag-set-run locator finds 183 blocks covering 145 tables. The investigator also overclaimed "verified end to end by decoding manager names": the names it decoded were the `manager` (career staff) table's inline names. **managercards names are stored as 32-bit string-pool offsets and the pool was never located.** We get ids, not names, and we do not need names because the client supplies them.
|
||||
|
||||
---
|
||||
|
||||
## 2. The route
|
||||
|
||||
**This is one enumeration problem. Commit to it.**
|
||||
|
||||
The alternative, five per-table id hunts driven from the club sweep, was costed honestly by the investigations themselves: the coach investigation priced an exhaustive single-family walk at roughly 480 club visits, and the manager investigation priced an exhaustive 1M-band walk at roughly 29. Even the cheap banded windows come to a dozen or more human tab re-entries across the five families, and for managers alone two of those visits already produced 2000+ records and no interpretable signal.
|
||||
|
||||
The enumeration route has already produced, at zero human tab cost, the exact id sets for four of the five card tables plus the consumable definition tables plus the full 107-column `players` table, and it has done so twice, by two agents using two different block locators, agreeing 417/417 on managercards against `fifa17-recon/docs/managercards_ids.txt`. It also answered the two questions the sweeps could never have answered: `fancards` and `newcards` are not families at all, so two of the five hunts do not exist.
|
||||
|
||||
**Say it plainly: enumeration is reachable read-only, and it makes the per-table id hunts moot.** Live sweeps are demoted from "how we find ids" to "how we confirm the decode, one visit, once."
|
||||
|
||||
I would switch back to sweeps only on one of three triggers:
|
||||
|
||||
1. The `width == ceil(log2(max-min+1))` assertion (see Q0) fails on a table we care about. That would mean the layout-entry field order is wrong, which is genuinely in dispute: the taxonomy pair read `{bitOffset, colTag, width, type}` and the manager reviewer read `{type, bit_offset, hash4, bit_width}`. The assertion is the discriminator.
|
||||
2. A family we need has no resident block and no second route to it. Nothing currently in scope is in that state, but the block census is a coverage claim, not a proof of absence.
|
||||
3. The one live confirmation in Q3 comes back disagreeing with the decoded ids in a way that is not a simple row-count over-read. Then the decode is a plausible byte run after all and everything below it is a guess.
|
||||
|
||||
---
|
||||
|
||||
## 3. The queue
|
||||
|
||||
Ordered by the cost of the measurement that settles the item, cheapest first. "Live test" means a human leaves a menu, re-enters a tab, and reports what is on screen.
|
||||
|
||||
### Q0. Dump the whole resident store to disk. Live tests: 0.
|
||||
|
||||
Do this first and do it now, while pid 52703 is alive.
|
||||
|
||||
**What it changes:** turns every remaining per-table id question in this round into arithmetic on a file.
|
||||
|
||||
**Code change:** new `fifa17-recon/tools/dbstore_dump.py`, built from the existing `fifa17-recon/tools/db_catalog_walk.py` (160 lines, already walks the catalog and writes `layouts.json`, which is not currently on disk). Three changes to it:
|
||||
- replace the header-signature block locator with the tag-set-run locator (scan for 4-aligned dwords equal to known column tags, group 16-byte-stride runs, match run tag-sets against the catalog). The signature version finds 90 blocks for 151 tables and missed `fitnesscoachcards`; the run version finds 183 for 145.
|
||||
- add a row decoder that emits values (`raw + min`) and writes one TSV per table under `fifa17-recon/data/dbstore/`, plus a manifest with `{table, rows, stride, rowptr, hdr, columns[]}`.
|
||||
- add three assertions, and fail loudly rather than writing a file: (a) for every int column, `width == ceil(log2(max-min+1))`; (b) the directory route, with the manager reviewer's `entry-0x20` correction, resolves the same descriptor as the string-pointer route; (c) each staff family's hardcoded DB-Error fallback assetid (2000148, 3000259, 4000146, 9000258) lands inside that family's own decoded id range.
|
||||
|
||||
No sweep, no `/tmp/fut_id_sweep` write, no server restart. `/proc/PID/mem` opened `'rb'`, seek and read only.
|
||||
|
||||
**POSITIVE looks like:** all three assertions pass; `managercards` decodes 417 ids matching `docs/managercards_ids.txt` exactly; `players` decodes 17563 ids that are a superset of the 17547 in `fifa17-recon/data/roster.json`; every `fcc_*` table lands on disk.
|
||||
|
||||
**A NEGATIVE proves:** assertion (a) failing proves the layout-entry field order is wrong, which invalidates every id in this plan and sends us back to the binary before any live test is spent. That is the whole reason to run it before Q3. Assertion (b) failing proves the two routes disagree and one of them is mis-anchored, which is precisely the error that produced the `factory_teams` mix-up. Assertion (c) failing proves the block-to-table attribution is coincidence, and the staff bands in Q3 should not be fired.
|
||||
|
||||
### Q1. Settle 417 versus 321 offline. Live tests: 0.
|
||||
|
||||
**What it changes:** the exact top of the manager id range, and more importantly which row-count field is authoritative for every table.
|
||||
|
||||
**The conflict:** the manager reviewer read `rows = 321`, `carddbid` topping out at **1,001,455**, from the count `u32` at `descriptor-0x10`. The taxonomy investigator and its reviewer both read `rows = 417`, topping out at **1,001,552**, from the layout header at `hdr+0x04`, and the reviewer reproduced all 417 against the committed file. Two different fields, two different pids (52703 and 11864), one static database. TODO/CONFIRM.
|
||||
|
||||
**Code change:** none beyond Q0. In the dump tool, print both counters for `managercards` side by side, then decode rows 321 through 416 and check that `carddbid` stays strictly increasing, that `assetid == carddbid` still holds, that `value` stays in a plausible rating range and `rare` stays 0/1. Junk in that tail means 321 is right and the layout header over-reads.
|
||||
|
||||
**POSITIVE:** the tail rows are well-formed, 417 stands, and `hdr+0x04` is the authoritative counter.
|
||||
|
||||
**A NEGATIVE proves:** if the tail decodes as noise, `descriptor-0x10` is authoritative and **every** row count in the Q0 dump needs re-reading from that field, including the staff families. That is a broad correction and it is much cheaper to catch here than in a live session.
|
||||
|
||||
### Q2. Batch every tool change into one server restart. Live tests: 0.
|
||||
|
||||
Restarting `utas_server.py` mid-session is the action that once produced "error connecting to FIFA 17 Ultimate Team", per the comment in `sweep_window()`. So all of this ships together, before the human launches.
|
||||
|
||||
**(a) Window lists in the sweep control file.** `_parse_window()` currently takes one window. Extend `sweep_window()`/`sweep_items()` to accept a `;`-separated or newline-separated list and advance one window per club fetch, exactly as `auto:` advances one chunk per fetch. **This is the highest-leverage change in the round**, because a club visit issues roughly 7 club fetches (recorded in the `_SWEEP_POS` comment), so it converts Q3 from five re-aims and five tab re-entries into one visit. Requirement: give each window its own item-id base. `sweep_items()` currently derives ids from `SWEEP_ID_BASE + (pid - lo) * len(subs) + si`, and two windows with overlapping offsets would collide in the CardsDb map and let one family overwrite another's readback.
|
||||
|
||||
**(b) `card_identity_probe.py`: print the consumable descriptor.** It already reads `cardtype` (+0x4c) and `subtype` (+0x50), which is what makes the consumable windows interpretable at all. Add `category = u32(+0xb8)`, `i16(+0xbc)`, `u8(+0xbe)`, `u8(+0xbf)`, `u8(+0xc0)`, `u8(+0xc1)`. Note that +0xb8 is firstname-as-string for players and staff and category-as-int32 for consumables; print both and let `cardtype` select which one is meaningful. Also surface `attrs[0]` prominently, since `attrs[0]==15` is the second staff-miss fingerprint.
|
||||
|
||||
**(c) `fut_store._item()`: sweep-only overrides.** Add optional `rareflag`, `amount` (atom 0x1b) and `contract` (atom 0xb8) overrides, applied **only** when `sweep_items()` passes them. The default item dict must not change by one byte: it is the live-proven player path. `amount` in particular has never been sent by us in any item ever, and it defaults to -1 in the parser, which stamps 0xFF into +0xbf for training/healing/fitness and +0xbe for play-style.
|
||||
|
||||
**POSITIVE:** the server restarts, the existing player club renders exactly as before with an empty control file, and a one-window control file still behaves identically to today.
|
||||
|
||||
**A NEGATIVE proves:** if the player club changes at all with an empty control file, one of these three edits leaked into the default path, and it must be reverted before any session.
|
||||
|
||||
### Q3. THE live session: managers plus the four staff bands. Live tests: 1 visit (5 fetches) if Q2(a) ships, otherwise 5.
|
||||
|
||||
**What it changes:** confirms or kills the entire enumeration route, and if it confirms, managers and all four coach families become servable in the same round.
|
||||
|
||||
**Control file** (`/tmp/fut_id_sweep`), in this order:
|
||||
|
||||
```
|
||||
t4@1000001-1001552
|
||||
t*@2000001-2000330
|
||||
t*@3000001-3000330
|
||||
t*@4000001-4000260
|
||||
t*@9000001-9000330
|
||||
```
|
||||
|
||||
Item counts: 1552, 1650, 1650, 1300, 1650. All under the 5000 ingest ceiling. `t*@` fans subtypes 4, 5, 6, 7, 8 per `_STAFF_SUBTYPES`.
|
||||
|
||||
**Predicted family mapping**, from the reviewer's attribution (fallback assetid in range, plus `fitnesscoachcards` pinned by its unique 9-column schema as the control, plus `physiocards` pinned independently by its 6-bit `amount` column):
|
||||
|
||||
| band | ids | rows | family | cardsubtypeid |
|
||||
|---|---|---|---|---|
|
||||
| 1M | 1000001-1001552 (or -1001455, see Q1) | 417 or 321 | managercards | 4 |
|
||||
| 2M | 2000004-2000328 | 124 | headcoachcards | 5 |
|
||||
| 3M | 3000019-3000328 | 115 | fitnesscoachcards | 8 |
|
||||
| 4M | 4000002-4000259 | 51 | physiocards | 7 |
|
||||
| 9M | 9000001-9000324 | 121 | gkcoachcards | 6 |
|
||||
|
||||
**POSITIVE, window 1:** exactly 417 records come back with non-blank first/last and rating in **57..88** (the decoded `value` range), matching `docs/managercards_ids.txt` id for id, with exactly 135 carrying rare. The other 1135 keep sentinel rating 7 and blank names. The rating range is what makes the manager window interpretable despite the missing miss-fill: sentinel 7 is nowhere near 57.
|
||||
|
||||
**POSITIVE, windows 2-5:** in each band, exactly **one** subtype returns real names and the other three return `firstname == "DB Error"`, `assetId ==` that family's fallback constant, rating 0x32, `attrs[0] == 15`. Subtype 4 stays untouched throughout, which is a free control.
|
||||
|
||||
**A NEGATIVE proves, and this is why these particular windows are worth a visit:**
|
||||
- Window 1 silent but windows 2-5 showing "DB Error": the merge ran, the plumbing works, and the failure is specific to how `resourceId` reaches `carddbid` on the manager branch. Actionable.
|
||||
- Window 1 hitting exactly 321 with 1001456-1001552 silent: Q1's conflict resolves to 321 and every row count in the dump needs re-reading. Actionable.
|
||||
- Windows 2-5 all "DB Error" with no real names anywhere: the branches execute, the subtype map is right, the plumbing is right, and only the decoded id sets are wrong. That is a clean, fully interpretable negative and it isolates the failure to Q0's decoder.
|
||||
- Nothing changes anywhere, in any window, including no "DB Error": the sweep never ingested. Check first that *any* record changed before concluding anything; if none did, both of the earlier manager sweeps were vacuous rather than negative, and the bug is upstream of every id in this document.
|
||||
|
||||
**Do not fire** `t*@6000000-...` or `t9@6000000-...`. The 6,000,000 to 6,499,999 band is set by `FUN_1801362e0`, which is not on the club path (`FUN_18013fe00` does not call it), so serving ids in that band through the club changes nothing at all and the negative proves nothing.
|
||||
|
||||
### Q4. Observe the club-item tabs, same session, zero extra launches.
|
||||
|
||||
**What it changes:** gives us the request half of a club-item wire shape, which is the only half we can obtain without inventing anything.
|
||||
|
||||
**No code change, no sweep.** While the human is already in the club for Q3 (with the control file **emptied**, so the real club is showing), have them open the My Club categories and club-items screens. We read `/tmp/utas_server.log` afterwards for any `type=` value never seen before (badge, kit, ball, stadium, equippables, leaguelogos, misc, unlocks, development, trophy variants) and for any `state=activeBadge|activeHomeKit|activeAwayKit|activeBall|activeStadium`. Today the log contains only `type=player`, `type=manager`, `type=custom&league=N`, and no-type with `team=N`, all under `User-Agent: ProtoHttp 1.3/DS 15.1.2.1.0 (Windows)`. (Be careful reading that log: it also contains `Python-urllib/3.14` requests from our own probes, and one earlier investigation nearly refuted the URL builder on the strength of those.)
|
||||
|
||||
**POSITIVE:** a new `type=` appears in the log. Then club items have a real request to answer and the family moves up the queue.
|
||||
|
||||
**A NEGATIVE proves:** that screen does not fetch from us, which means club items are further away than their id spaces suggest and the ranking in Q7 is right.
|
||||
|
||||
### Q5. Consumables: one fetch, ten categories. Live tests: 1 fetch (can share the Q3 visit if Q2(a) ships).
|
||||
|
||||
**What it changes:** confirms the "pure enum card" model and tells us whether a contract card is three JSON keys.
|
||||
|
||||
**Control file**, appended to the Q3 list or fired on its own:
|
||||
|
||||
```
|
||||
t201@1-1;t202@1-1;t211@1-1;t219@1-1;t91@1-1;t251@1-1;t300@1-1;t51@1-1;t61@1-1;t71@1-1
|
||||
```
|
||||
|
||||
with Q2(c) emitting `amount` on cardtype-6 subtypes and `contract` on 201/202.
|
||||
|
||||
**POSITIVE:** `cardtype` (+0x4c) reads **6** on all ten, and `category` (+0xb8) reads 2, 3, 4, 5, 8, 9, 10, 0, 0, and 6-or-7 respectively. On t91, `+0xbf == 7` and `+0xbc == 8`, both derived from the subtype alone with nothing sent, which is the strongest available test of the no-data-needed claim. On t300, `+0xbc` carries the league id from `amount`.
|
||||
|
||||
**A NEGATIVE proves:** `+0x4c` reading 0 proves the `FUN_1800d8330` range read is wrong; `+0x4c` reading 6 with category 0 everywhere proves `FUN_18013f4d0` is not reached on this path. Both are sharp, and they are sharp only because the probe prints `+0x4c`. Reading `+0xb8` alone cannot distinguish "cardtype 6, dead zone, category 0" from "cardtype 0, resolver never called", since the parser memsets +0xb8..+0x157 to zero.
|
||||
|
||||
**One exception, state it to the human:** t219 tests the `rareflag` trap, and the trap is a loc-key choice made at render time inside `FUN_1801bfac0` that leaves **no trace in the record**. The probe cannot answer it. It needs a human reading the card on screen: "Squad Fitness" confirms the trap, "Player Fitness" refutes the read of case 5.
|
||||
|
||||
### Q6. cardtype 7 (cardsubtypeid 9, 10, 11). Live tests: 1, low priority.
|
||||
|
||||
No DB merge, no table, no id space. The only observable is the parser's cardtype-7 arm copying `name`/`localizedName` and `description` (0x1f bytes each) straight off the wire, plus a `cardassetid` default of 0x23 for subtype 9. Those are scalar string atoms and cannot desync the tokenizer, so the risk is low, but the value is also low: this family currently has no known use. Fire it only if a session has a spare fetch.
|
||||
|
||||
### Q7. Club items: badges, balls, kits, stadium. Ranked LAST, and here is the reason.
|
||||
|
||||
The id spaces come free from Q0 (`fcc_badgecards` 11 columns, `fcc_kitcards` 12, `fcc_balls` 9, `fcc_stadium` 13, `fcc_leaguelogos` 8, `fcc_misccards` 10 are all in the catalog), so nothing is being deferred out of ignorance. What is missing is everything else: cardtype 9 has no merge arm, so no name, no miss-fill, no readback signal of any kind; no badge/kit/ball/stadium request has ever been observed from this client; no response shape has ever been observed; and the artwork path (probably keyed on `teamid` 0x306 or `stadiumid` 0x2db) was never read, which is the one place a hidden id space could still be lurking.
|
||||
|
||||
Every sweep window anyone has proposed for this family is **invariant by construction**: with no merge arm and no miss-fill, every id in the window produces a byte-identical record, so the probe's readback cannot discriminate. The only signal on offer is a human eyeballing the UI, and that is confounded twice: `sweep_items()` deliberately keeps `itemType: "player"` while `itemType` is one of the two wire discriminators, and `club_route` returns the sweep payload **before** the `?type=` filter, so the items arrive in answer to a `type=player` fetch.
|
||||
|
||||
Do not spend a live test here. Revisit only if Q4 produces a real `type=badge` (or kit/ball/stadium/equippables) request in the log. The honest cheap win in this family is not items at all: it is adding the `kits`, `kitsHome`, `kitsAway`, `badges`, `badgeDBid`, `balls`, `stadia`, `leagueLogos` and trophy rows to the existing, live-proven `club/stats` payload, where the values are plain counts and the row shape is already exercised. That is a data change with no new wire shape and freeze risk none.
|
||||
|
||||
### Q8. Documentation corrections, zero cost, commit alongside Q0.
|
||||
|
||||
`docs/ENDPOINT_MAP.md`: `GET ut/%s/club` is **ClubSearch** (registry index 3, confirmed by a full 125-record walk), not `FutGetClubInfo`; `GetClubInfo` is on `ut/%s/user/list`, `GetClubUsers` on `ut/%s/clubUser`. Row 12: `FutConsumablesSearch` (deser 0x180130d10) has no `displayGroupUseDefaultImage`; its only top-level atom is `itemData` (0x16b), whose elements are `{count 0xbc, discardValue 0xd7, item 0x16a, resourceId 0x287, untradeableCount 0x362}`. Also record: `fancards` is a `fixtures` column, `newcards` is atom 0x1d7, and neither is a card family.
|
||||
|
||||
---
|
||||
|
||||
## 4. The next live session
|
||||
|
||||
**One launch. Do Q0, Q1 and Q2 before the human touches anything.**
|
||||
|
||||
Preconditions to verify from the terminal, not from memory: `FIFA17.exe` alive; `utas_server.py` alive on its port; `/tmp/fut_id_sweep` exists and is **empty**; `fifa17-recon/data/dbstore/` written by Q0 with all three assertions passing; `utas_server.py` restarted **once** with the Q2 changes and the empty control file still rendering the real club.
|
||||
|
||||
Then, in order:
|
||||
|
||||
1. **Human launches FIFA 17 and reaches the FUT club UI.** Record: that the club looks normal, and the game's pid. If Q0 has not run yet, run it now against this pid before arming anything.
|
||||
2. **Arm window 1.** Write `t4@1000001-1001552` (or the full 5-window list, if Q2(a) shipped) into `/tmp/fut_id_sweep`. No restart.
|
||||
3. **Human leaves the club tab and re-enters it.** Record: does the club fill with cards at all; do any of them show real manager names; roughly how many named versus blank. If Q2(a) shipped, the human should page through the club so the client issues its ~7 fetches and all five windows advance in this one visit.
|
||||
4. **Run `card_identity_probe`.** Record to a file, not to the eye: the count of records with rating != 7, the count with non-blank first/last, the exact set of `resourceId` values that resolved, the count with rare set, and separately the count with `first == "DB Error"` and their `assetId` values.
|
||||
5. **Diff the resolved id set against `docs/managercards_ids.txt`.** Record the three numbers: matched, expected-but-missing, resolved-but-unexpected. Any id above 1001455 that resolved settles Q1 in favour of 417.
|
||||
6. **For each staff band, record which single cardsubtypeid returned real names** and confirm the other three returned "DB Error" with the right fallback assetid. That is the table attribution, confirmed from outside, and it is the one thing the memory walk could not do on its own for the three schema-identical families.
|
||||
7. **Optionally append the Q5 consumables list** and have the human re-enter the tab once more. Record `cardtype` and `category` for all ten, plus what the fitness card (t219) literally says on screen, in words.
|
||||
8. **Empty `/tmp/fut_id_sweep`.** Human re-enters the club tab. **Record that the real club is back and unchanged.** Do not end the session without this step.
|
||||
9. **Q4, last, with the control file empty:** human opens the My Club categories and club-items screens. We read the log for new `type=` values. Nothing is served, nothing is judged on screen.
|
||||
|
||||
Everything the human is asked to report is a count or a literal string, never a judgement about whether something "looks right".
|
||||
|
||||
---
|
||||
|
||||
## 5. What could make this plan wrong
|
||||
|
||||
**The decode could be a plausible byte run rather than a mapped structure.** Three independent signals say otherwise: the `width == ceil(log2(max-min+1))` prediction held 8 of 8 on managercards' integer columns, the four hardcoded DB-Error fallback assetids each land inside their own family's decoded range, and two agents using two different block locators agreed 417 of 417 on managercards. If any of those three stops holding on a table, treat that table's ids as a guess again. This exact failure mode has already burned two sessions: pointer fragments read as sequential ids, and a strictly-increasing 18-bit field in the `manager` table that turned out to be `teamid`.
|
||||
|
||||
**The two row-count fields disagree and we do not know which is authoritative.** 417 versus 321 is unresolved (TODO/CONFIRM). If `descriptor-0x10` wins, every row count in the Q0 dump is wrong, including all four staff bands, and Q3's windows are mis-sized.
|
||||
|
||||
**The layout-entry field order is in dispute.** `{bitOffset, colTag, width, type}` versus `{type, bit_offset, hash4, bit_width}`. Same 16 bytes, different reading. The `ceil(log2)` assertion is the only thing standing between us and silently decoding the wrong bits.
|
||||
|
||||
**Q0 can kill the oracle.** `/proc/PID/mem` is the whole route, and the game is a live oracle that costs a launch to recreate. A stray `'r+b'`, a mistaken write path, or an out-of-range seek in a hardened version of the tool takes FIFA17.exe down. Loading gigabytes in 16 MB chunks can also push the machine into swap and make the game hitch. Open `'rb'` only, read while the game sits at a menu, never mid-match.
|
||||
|
||||
**Q0 can break the one thing that is fully working: the roster.** `fifa17-recon/data/roster.json` holds 17,547 players and was cross-validated 573 of 573 against an independent oracle. The decoded `players` table has **17,563** rows, a 16-id delta with no explanation (TODO/CONFIRM). It will be tempting to regenerate the roster from the richer 107-column table. Do not. Diff the id sets first, only ever add columns, and keep the validated file until a fresh cross-validation passes on the new one.
|
||||
|
||||
**Q2(c) can break every player card in the club.** `fut_store._item()` hardcodes `"rareflag": 1`, the merge reads +0x58 for the rare tier, and the parser's `discardValue` fallback query keys on `rare` as well as `cardtype` and `rating`. Any change to that default re-tiers and re-prices every card in an existing save. Keep the overrides sweep-only and default-off, and verify with an empty control file before the human launches.
|
||||
|
||||
**Q2(a) can silently corrupt its own readback.** `sweep_items()` derives item ids from the candidate's offset within the range so chunks accumulate in the map without colliding. A window **list** breaks that invariant unless each window gets its own id base: two windows would write different families to the same map keys and one would overwrite the other, producing a clean-looking result that is simply the last window five times.
|
||||
|
||||
**Q3 transiently breaks the club tab fix that just shipped.** `club_route` returns the sweep payload before the `?type=` filter, so while any window is armed, the player tab, the staff tab and the league and team drill-downs all show sweep items. That is precisely the regression class the `type=` filter was added to fix (footballers displayed as coaching staff). It reverses on the next fetch once the file is emptied, but the human must never be asked to judge anything about their real club while a window is armed, and step 8 of the session is not optional.
|
||||
|
||||
**Q3's item counts are under a ceiling that was proven for a different payload.** 5000 items is live-proven for a single-subtype player payload; 20000 was served and silently not ingested. A `t*@` payload at 1650 has never been ingested at any size (4995 was proposed, never fired). Before drawing any conclusion from silence, confirm that at least one record in the window changed. TODO/CONFIRM.
|
||||
|
||||
**Q5 can freeze the club screen.** Consumables are the first cardtype-6 items this client will ever have been served, and the highest `cardsubtypeid` it has seen to date is 8. If a category the UI cannot lay out arrives, the club tab is where it fails. Serve one item per subtype, never thousands, and never a subtype outside the ranges read out of `FUN_18013f4d0`.
|
||||
|
||||
**Q7, if anyone rushes it, breaks the staff tab.** Adding `type=badge` handling means touching the narrow filter in `club_route`, whose default for unrecognised types is deliberately "filter, not unfiltered". Flipping that default is exactly how footballers reappear in the coaching staff list.
|
||||
|
||||
**And the honest limit of the whole round:** an id space buys names and ratings, not pictures. The merge writes manager firstname, lastname, assetid, value, talkrating, negotiation and rare, and nothing else. Nothing in this plan establishes that a manager card will render a portrait, or that a badge will render a crest. Manager names are stored in the client's own tables as 32-bit string-pool offsets whose pool was never located, so we cannot even preview offline what the client will show. Artwork is the next problem, and it should not be promised as part of this one.
|
||||
@@ -0,0 +1,160 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""READ-ONLY: enumerate every resident FIFA 17 DB table and dump its rows.
|
||||
|
||||
See dbwalk.py header for the on-heap format. This version locates a table's
|
||||
layout block by its header signature (ncols<<16 | 0xffff) and matches on the
|
||||
column tag set. Opens /proc/PID/mem 'rb'; only seek()/read().
|
||||
"""
|
||||
import glob, struct, sys, json
|
||||
|
||||
CONST = 0x07C20760
|
||||
|
||||
def find_pid():
|
||||
for d in glob.glob("/proc/[0-9]*"):
|
||||
try:
|
||||
if open(d + "/comm").read().strip() == "FIFA17.exe":
|
||||
return int(d.rsplit("/", 1)[-1])
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
pid = find_pid()
|
||||
f = open("/proc/%d/mem" % pid, "rb")
|
||||
|
||||
def rd(va, n):
|
||||
try:
|
||||
f.seek(va); b = f.read(n)
|
||||
return b if b and len(b) == n else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def q(va):
|
||||
b = rd(va, 8)
|
||||
return struct.unpack("<Q", b)[0] if b else None
|
||||
|
||||
def cstr(va, m=64):
|
||||
b = rd(va, m)
|
||||
if not b:
|
||||
return None
|
||||
z = b.find(b"\x00")
|
||||
if z <= 0:
|
||||
return None
|
||||
try:
|
||||
return b[:z].decode("ascii")
|
||||
except UnicodeDecodeError:
|
||||
return None
|
||||
|
||||
def regions(lo_lim=0, hi_lim=1 << 62):
|
||||
out = []
|
||||
for line in open("/proc/%d/maps" % pid):
|
||||
p = line.split()
|
||||
a, b = p[0].split("-")
|
||||
if "r" not in p[1] or "w" not in p[1]:
|
||||
continue
|
||||
lo, hi = int(a, 16), int(b, 16)
|
||||
if hi - lo > 512 << 20 or hi < lo_lim or lo > hi_lim:
|
||||
continue
|
||||
out.append((max(lo, lo_lim), min(hi, hi_lim)))
|
||||
return out
|
||||
|
||||
# ------- load the DB heap once -------
|
||||
CHUNKS = []
|
||||
for lo, hi in regions(0x06000000, 0x48000000):
|
||||
off = lo
|
||||
while off < hi:
|
||||
n = min(16 << 20, hi - off)
|
||||
d = rd(off, n)
|
||||
if d:
|
||||
CHUNKS.append((off, d))
|
||||
off += n
|
||||
print("loaded %d chunks, %.0f MB" % (len(CHUNKS), sum(len(c[1]) for c in CHUNKS) / 1e6))
|
||||
|
||||
def scan(pat):
|
||||
out = []
|
||||
for base, d in CHUNKS:
|
||||
i = d.find(pat)
|
||||
while i != -1:
|
||||
out.append(base + i)
|
||||
i = d.find(pat, i + 1)
|
||||
return out
|
||||
|
||||
# ------- catalog -------
|
||||
tables = {}
|
||||
for c in scan(struct.pack("<Q", CONST)):
|
||||
if c % 8:
|
||||
continue
|
||||
nm = cstr(q(c + 8) or 0)
|
||||
if not nm:
|
||||
continue
|
||||
h = rd(c - 0x10, 0x10)
|
||||
if not h:
|
||||
continue
|
||||
tag, cnt, colarr = struct.unpack("<IIQ", h)
|
||||
if not (1 <= cnt <= 200) or colarr < 0x1000 or q(colarr + 0x20) != CONST:
|
||||
continue
|
||||
cols = []
|
||||
for i in range(cnt):
|
||||
d = rd(colarr + i * 0x30, 0x30)
|
||||
if not d:
|
||||
break
|
||||
ty, ctag, mn, mx, ln = struct.unpack_from("<IIiII", d, 0)
|
||||
cols.append(dict(name=cstr(struct.unpack_from("<Q", d, 0x28)[0]),
|
||||
type=ty, tag=ctag, min=mn, max=mx, len=ln))
|
||||
tables.setdefault(nm, []).append(dict(desc=c - 0x10, ncols=cnt, cols=cols))
|
||||
print("tables: %d" % len(tables))
|
||||
|
||||
# ------- layout blocks by header signature -------
|
||||
ncounts = sorted({t["ncols"] for v in tables.values() for t in v})
|
||||
blocks = []
|
||||
for n in ncounts:
|
||||
sig = struct.pack("<I", (n << 16) | 0xFFFF)
|
||||
for a in scan(sig):
|
||||
if a % 4:
|
||||
continue
|
||||
hdr = a - 8 # hdr = {cap, rowcount, sig, ?}
|
||||
ents = []
|
||||
ok = True
|
||||
for k in range(n):
|
||||
e = rd(hdr + 0x10 + k * 16, 16)
|
||||
if not e:
|
||||
ok = False; break
|
||||
off, tag, w, ty = struct.unpack("<4I", e)
|
||||
tb = struct.pack("<I", tag)
|
||||
if not (all(0x30 <= x < 0x7B for x in tb) and 0 < w <= 512 and off < 16384):
|
||||
ok = False; break
|
||||
ents.append((off, tag, w, ty))
|
||||
if ok:
|
||||
blocks.append((hdr, struct.unpack("<I", rd(hdr + 4, 4))[0], ents))
|
||||
print("layout blocks: %d" % len(blocks))
|
||||
byset = {}
|
||||
for hdr, rc, ents in blocks:
|
||||
byset.setdefault(frozenset(e[1] for e in ents), []).append((hdr, rc, ents))
|
||||
|
||||
def align4(x):
|
||||
return (x + 3) & ~3
|
||||
|
||||
out = {}
|
||||
for name in sorted(tables):
|
||||
for t in tables[name]:
|
||||
tset = frozenset(c["tag"] for c in t["cols"])
|
||||
for hdr, rc, ents in byset.get(tset, []):
|
||||
lay = {e[1]: (e[0], e[2], e[3]) for e in ents}
|
||||
maxbit = max(o + w for o, w, ty in
|
||||
[(lay[c["tag"]][0], 32 if c["type"] == 1 else lay[c["tag"]][1],
|
||||
0) for c in t["cols"]])
|
||||
stride = align4((maxbit + 7) // 8)
|
||||
rowptr = q(hdr - 0x48)
|
||||
out.setdefault(name, []).append(
|
||||
dict(hdr=hdr, rows=rc, rowptr=rowptr, stride=stride,
|
||||
cols=[(c["name"], c["type"], lay[c["tag"]][0],
|
||||
lay[c["tag"]][1], c["min"], c["max"]) for c in t["cols"]]))
|
||||
|
||||
for name in sorted(out):
|
||||
for b in out[name]:
|
||||
print("\n== %-24s rows=%-7d stride=%-3d rowptr=%#x hdr=%#x"
|
||||
% (name, b["rows"], b["stride"], b["rowptr"] or 0, b["hdr"]))
|
||||
for cn, ty, o, w, mn, mx in sorted(b["cols"], key=lambda x: x[2]):
|
||||
print(" bit %-5d w=%-4d %-26s type=%d [%d..%d]" % (o, w, cn, ty, mn, mx))
|
||||
json.dump({k: [{kk: vv for kk, vv in b.items()} for b in v] for k, v in out.items()},
|
||||
open("layouts.json", "w"), indent=1)
|
||||
print("\nwrote layouts.json for %d tables" % len(out))
|
||||
Executable
+242
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Read FIFA 17's LOADED database schema (and any table's rows) out of a live
|
||||
FIFA17.exe. READ-ONLY: /proc/PID/mem is opened 'rb' and only ever seek()/read().
|
||||
There is no write path in this file.
|
||||
|
||||
WHY THIS EXISTS
|
||||
---------------
|
||||
`managercards` (and headcoachcards / gkcoachcards / physiocards /
|
||||
fitnesscoachcards) are queried by CardsDLL through a column called `carddbid`,
|
||||
and the set of legal carddbid values was unknown -- two blind sweeps (1..5000 and
|
||||
6000..8000) resolved nothing, and the manager branch writes no miss-fill so a
|
||||
wrong id is completely silent. dbdata.dll on disk is packed (see
|
||||
dbdata_extract.py's docstring: uniform ~7.5 bits/byte entropy, one export that
|
||||
returns a ~759-byte attestation blob), so the tables cannot be read off disk.
|
||||
|
||||
They ARE fully resident in the running game, and unlike the player roster they
|
||||
are reachable through the DB's own SCHEMA rather than by content-anchoring.
|
||||
|
||||
THE STRUCTURES (resolved live, 2026-08-04, pid 11864, sitting in the FUT club UI)
|
||||
--------------------------------------------------------------------------------
|
||||
Three cooperating structures, all in the game's rw-p heap:
|
||||
|
||||
* IDENTIFIER INTERN POOL, seen at 0x0770_0000..0x07d0_0000. Every table name
|
||||
and every column name in the whole database, interned once. Entry:
|
||||
+0x00 char* next-in-bucket
|
||||
+0x08 u64 strlen+1
|
||||
+0x10 u64 hash
|
||||
+0x18 char[] the NUL-terminated identifier, INLINE
|
||||
e.g. "managercards" at 0x07be63a0, "carddbid" at 0x078eefc8,
|
||||
"talkrating" at 0x078ec188, "headcoachcards" at 0x07be5b80.
|
||||
|
||||
* TABLE DIRECTORY, seen at 0x4254_7ff0.., 573 entries of 40 bytes. The
|
||||
reliable signature is the CONSTANT QWORD 0x07c20760 that every entry carries
|
||||
at +0x20 -- anchor on that, not on any fixed address:
|
||||
+0x00 char* table name (into the intern pool)
|
||||
+0x08 void* table descriptor ("p1")
|
||||
+0x10 u32 name hash4 (4 bytes, also repeated inside the descriptor)
|
||||
+0x14 u32 column count
|
||||
+0x18 void* column-descriptor bucket ("p2")
|
||||
+0x20 void* 0x07c20760 (the anchor)
|
||||
Live values read this session:
|
||||
managercards p1=0x42482f08 hash='xIfB' ncol=62
|
||||
headcoachcards p1=0x427429d8 hash='WTdJ' ncol=18
|
||||
gkcoachcards p1=0x42e46648 hash='CZUM' ncol=107
|
||||
physiocards p1=0x42482478 hash='AThf' ncol=9
|
||||
fitnesscoachcards p1=0x42d668e8 hash='MmoU' ncol=11
|
||||
|
||||
* TABLE DESCRIPTOR (at p1). Self-identifies by repeating the table's own
|
||||
hash4; immediately after that hash come
|
||||
u32 row_size_bytes, u32 max_bit_index
|
||||
(managercards: 0x80 and 0x3ff -- 128 bytes == 1024 bits, consistent), and
|
||||
then a column array of 16-byte records:
|
||||
u32 type (3 for the scalar columns seen), u32 bit_offset,
|
||||
u32 name hash4, u32 bit_width
|
||||
First managercards columns read live (bit_offset, width):
|
||||
(0x260,7) (0x20,8) (0x267,15) (0x28,8) (0x276,14) (0x284,5)
|
||||
(0x30,8) (0x289,7) (0x290,2) (0x38,8) (0x292,18) (0x2a4,...)
|
||||
Note the bit offsets run past 416, so a row is NOT the 0x34-byte
|
||||
firstname/lastname record seen near 0x42530928 -- that is a different table.
|
||||
|
||||
* COLUMN-NAME entry (0x30 bytes, in the p2 buckets, also anchored by
|
||||
0x07c20760 at +0x20): +0x04 hash4, +0x28 char* name. This is what turns a
|
||||
column's hash4 back into "carddbid".
|
||||
|
||||
WHAT THIS TOOL DOES NOT YET DO
|
||||
------------------------------
|
||||
It stops at the schema. It does not locate the ROW STORAGE for a table -- that
|
||||
pointer was not identified before the game exited. Candidate leads recorded at
|
||||
the time: managercards' descriptor holds 0x426cb278 at p1+0x08 and 0x07822918 at
|
||||
p1+0x30, and the 16 bytes before p1 read `... 81 04 00 00` (0x481 = 1153, a
|
||||
plausible row count). Once row storage is found, carddbid for every row is a
|
||||
mechanical bit-extract with (bit_offset, bit_width) from --table.
|
||||
|
||||
USAGE
|
||||
-----
|
||||
./dbschema_probe.py # list the FUT card tables
|
||||
./dbschema_probe.py --table managercards # full column list for one table
|
||||
./dbschema_probe.py --list # every table in the directory
|
||||
|
||||
Requires ptrace access to the FIFA process (ptrace_scope=1 + same uid suffices)
|
||||
and FIFA17.exe actually running.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
|
||||
SHARED_ANCHOR = 0x07C20760 # the qword every directory/column entry carries at +0x20
|
||||
POOL_LO, POOL_HI = 0x07700000, 0x07D00000
|
||||
HEAP_LO, HEAP_HI = 0x42000000, 0x43000000
|
||||
IDENT = re.compile(r'[A-Za-z][A-Za-z0-9_]*\Z')
|
||||
|
||||
FUT_TABLES = ("managercards", "headcoachcards", "gkcoachcards", "physiocards",
|
||||
"fitnesscoachcards", "players", "teams", "leagues", "nations")
|
||||
|
||||
|
||||
def find_pid():
|
||||
for d in os.listdir('/proc'):
|
||||
if not d.isdigit():
|
||||
continue
|
||||
try:
|
||||
if open('/proc/%s/comm' % d).read().strip() == 'FIFA17.exe':
|
||||
return int(d)
|
||||
except OSError:
|
||||
pass
|
||||
raise SystemExit("FIFA17.exe is not running (this tool needs the live game)")
|
||||
|
||||
|
||||
class Mem(object):
|
||||
"""Read-only window onto /proc/PID/mem, with the two regions pre-slurped."""
|
||||
|
||||
def __init__(self, pid):
|
||||
self.fh = open('/proc/%d/mem' % pid, 'rb')
|
||||
self.pool = self.read(POOL_LO, POOL_HI - POOL_LO)
|
||||
self.heap = self.read(HEAP_LO, HEAP_HI - HEAP_LO)
|
||||
if not self.pool or not self.heap:
|
||||
raise SystemExit("could not read the DB regions -- is the game past the "
|
||||
"main menu? (the DB is loaded lazily)")
|
||||
|
||||
def read(self, addr, n):
|
||||
if addr < 0 or addr > (1 << 47):
|
||||
return b''
|
||||
try:
|
||||
self.fh.seek(addr)
|
||||
return self.fh.read(n)
|
||||
except (OSError, ValueError):
|
||||
return b''
|
||||
|
||||
def cstr(self, addr, maxlen=64):
|
||||
for base, buf in ((POOL_LO, self.pool), (HEAP_LO, self.heap)):
|
||||
if base <= addr < base + len(buf):
|
||||
i = addr - base
|
||||
j = buf.find(b'\x00', i, i + maxlen)
|
||||
if j <= i:
|
||||
return None
|
||||
s = buf[i:j]
|
||||
return s.decode('latin1') if re.fullmatch(rb'[ -~]+', s) else None
|
||||
return None
|
||||
|
||||
|
||||
def hash4_to_names(mem):
|
||||
"""Every column-name entry in the DB, keyed by its 4-byte name hash."""
|
||||
out = {}
|
||||
for base, buf in ((HEAP_LO, mem.heap), (POOL_LO, mem.pool)):
|
||||
for off in range(0, len(buf) - 0x30, 8):
|
||||
if struct.unpack_from('<Q', buf, off + 0x20)[0] != SHARED_ANCHOR:
|
||||
continue
|
||||
nm = mem.cstr(struct.unpack_from('<Q', buf, off + 0x28)[0], 48)
|
||||
if not nm or not IDENT.match(nm):
|
||||
continue
|
||||
out.setdefault(struct.unpack_from('<I', buf, off + 4)[0], set()).add(nm)
|
||||
return out
|
||||
|
||||
|
||||
def table_directory(mem):
|
||||
"""name -> (entry_addr, descriptor, hash4, ncol, colbucket)."""
|
||||
out = {}
|
||||
buf = mem.heap
|
||||
for off in range(0, len(buf) - 0x28, 8):
|
||||
if struct.unpack_from('<Q', buf, off + 0x20)[0] != SHARED_ANCHOR:
|
||||
continue
|
||||
nm = mem.cstr(struct.unpack_from('<Q', buf, off)[0], 48)
|
||||
if not nm or not IDENT.match(nm):
|
||||
continue
|
||||
out[nm] = (HEAP_LO + off,
|
||||
struct.unpack_from('<Q', buf, off + 8)[0],
|
||||
struct.unpack_from('<I', buf, off + 0x10)[0],
|
||||
struct.unpack_from('<I', buf, off + 0x14)[0],
|
||||
struct.unpack_from('<Q', buf, off + 0x18)[0])
|
||||
return out
|
||||
|
||||
|
||||
def h4str(v):
|
||||
return struct.pack('<I', v).decode('latin1')
|
||||
|
||||
|
||||
def describe(mem, tables, h2n, name):
|
||||
if name not in tables:
|
||||
print("### %-20s NOT IN DIRECTORY" % name)
|
||||
return
|
||||
ent, p1, h4, ncol, p2 = tables[name]
|
||||
print("\n### %s dir@%#x descriptor=%#x hash=%r columns=%d"
|
||||
% (name, ent, p1, h4str(h4), ncol))
|
||||
blk = mem.read(p1, 0x80 + ncol * 16 + 0x80)
|
||||
if not blk:
|
||||
print(" descriptor unreadable")
|
||||
return
|
||||
pos = blk.find(struct.pack('<I', h4))
|
||||
if pos < 0:
|
||||
print(" self-hash not found in the descriptor -- layout changed")
|
||||
return
|
||||
rowsz, maxbit = struct.unpack_from('<II', blk, pos + 4)
|
||||
print(" self-hash at +%#x row_size=%#x bytes max_bit=%#x" % (pos, rowsz, maxbit))
|
||||
# the column array begins at the first 16-byte record whose hash4 is a known
|
||||
# column name and whose width is sane
|
||||
st = pos + 12
|
||||
while st < len(blk) - 16:
|
||||
t, boff, ch, w = struct.unpack_from('<IIII', blk, st)
|
||||
if t == 3 and ch in h2n and 0 < w <= 64:
|
||||
break
|
||||
st += 4
|
||||
print(" column array at +%#x" % st)
|
||||
cols = []
|
||||
for i in range(ncol):
|
||||
o = st + i * 16
|
||||
if o + 16 > len(blk):
|
||||
break
|
||||
t, boff, ch, w = struct.unpack_from('<IIII', blk, o)
|
||||
cols.append((boff, w, t, ch,
|
||||
"|".join(sorted(h2n.get(ch, {"?" + h4str(ch)})))))
|
||||
for boff, w, t, ch, nm in sorted(cols):
|
||||
print(" bit %5d width %-3d type %-2d %s" % (boff, w, t, nm))
|
||||
return cols
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--pid', type=int)
|
||||
ap.add_argument('--table', action='append')
|
||||
ap.add_argument('--list', action='store_true')
|
||||
a = ap.parse_args()
|
||||
|
||||
pid = a.pid or find_pid()
|
||||
print("FIFA17.exe pid %d" % pid)
|
||||
mem = Mem(pid)
|
||||
h2n = hash4_to_names(mem)
|
||||
tables = table_directory(mem)
|
||||
print("column-name hashes: %d tables in directory: %d" % (len(h2n), len(tables)))
|
||||
if a.list:
|
||||
for nm in sorted(tables):
|
||||
ent, p1, h4, ncol, p2 = tables[nm]
|
||||
print(" %-34s desc=%#012x hash=%r cols=%d" % (nm, p1, h4str(h4), ncol))
|
||||
return
|
||||
for nm in (a.table or FUT_TABLES):
|
||||
describe(mem, tables, h2n, nm)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user