fifa17-recon: real positions, clubs and the six card attributes for all 17,563 players
The pool now comes from the game's OWN resident database, not from a rating index plus guesses. tools/db_dump.py walked the client's self-describing table catalog read-only and wrote data/tables/ (149 tables, 55MB); tools/build_player_facts.py turned it into data/player_facts.json; data/pool.json is the compact form fut_cards loads. MEASURED, per player: position (players.preferredposition1), nationality, teamid, leagueid (via leagueteamlinks), and the six card attributes. The six attributes are NOT columns -- they are a weighted sum of the 29 base attributes, and the weights are read out of the game's own playerattributesmapping table rather than from published formulas. Checked against real FIFA 17 cards: Messi 89/90/86/96/26/61 and Ibrahimovic 72/90/81/85/31/86 are EXACT, Suarez is one off on physical, Ronaldo within two on pace and shooting. Keepers come out directly from the gk* columns. What this fixes on screen: Kaka was a CDM, Bale a CM, Suarez a GK, and every attribute was derived from the rating. Now Bale is RW, Boateng is a CB with 90 defending, De Gea is a GK, and a bronze pack deals real bronze players in real positions. REVERSAL, deliberate: nation/team/league were being sent as ZERO so the client would fill its own values (the merge fills those three only when they arrive zero). Now that we hold the game's own numbers there is nothing to gain from zeros, and they actively hurt -- club-stats drill-downs bucket by the item's own nation and leagueId, so a club full of zeros would have quietly emptied the per-nation and per-league panels fixed the day before. Send the real values. The old rating-index path is kept as a fallback so the pool still builds without data/pool.json, and it now says out loud which of the two it used, because one is a measurement and the other is a guess. 439 + 61 checks green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUT92pz6RWKih9dSr8ZpxW
This commit is contained in:
funman300
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Turn the raw table dumps written by db_dump.py into ONE file the FUT card
|
||||
pool can consume: data/player_facts.json.
|
||||
|
||||
Input data/tables/{players,teamplayerlinks,teams,leagues,nations,
|
||||
leagueteamlinks,playerattributesmapping}.json
|
||||
Output data/player_facts.json
|
||||
|
||||
{"meta": {...},
|
||||
"players": [ {"id":20801,"rating":94,"pos":27,"pos2":16,"pos3":25,"pos4":-1,
|
||||
"nation":38,"team":243,"league":53,
|
||||
"attrs":[90,93,82,91,33,80], # PAC SHO PAS DRI DEF PHY
|
||||
"gk":false}, ... ]}
|
||||
|
||||
THE SIX CARD ATTRIBUTES ARE NOT COLUMNS. `players` stores the 29 base
|
||||
attributes; the six numbers a FUT card shows are a weighted sum of them. The
|
||||
weights are not guessed here -- they are read out of the game's own
|
||||
`playerattributesmapping` table, which maps each base attribute id to its
|
||||
percentage contribution to speed / shooting / passing / dribbling / defending /
|
||||
physical (and to the five gk* stats). Every column of that table sums to
|
||||
exactly 100.
|
||||
|
||||
The one inference this file makes is attributeid -> players column name, and it
|
||||
is safe for this purpose: wherever two attributes could be swapped (marking vs
|
||||
standingtackle at 30 each, shotpower vs longshots at 20 each, ...) they carry
|
||||
EQUAL weight, so the computed six are identical either way. The assignments
|
||||
that actually move a number -- 45/55 acceleration/sprintspeed, 45 finishing,
|
||||
50 dribbling, 35 shortpassing, 30 ballcontrol, 50 strength, 25 stamina,
|
||||
20 aggression, 20 interceptions, 15 longpassing -- are each the unique
|
||||
attribute with that weight.
|
||||
|
||||
VALIDATION (see the report): overallrating in `players` agrees with
|
||||
data/roster.json, extracted from a completely different memory structure, on
|
||||
17547 of 17547 shared players.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
TABLES = os.path.join(HERE, '..', 'data', 'tables')
|
||||
OUT = os.path.join(HERE, '..', 'data', 'player_facts.json')
|
||||
|
||||
# FUT card slot -> [(players column, percent)], read off playerattributesmapping.
|
||||
OUTFIELD = [
|
||||
('pace', [('acceleration', 45), ('sprintspeed', 55)]),
|
||||
('shooting', [('finishing', 45), ('shotpower', 20), ('longshots', 20),
|
||||
('positioning', 5), ('volleys', 5), ('penalties', 5)]),
|
||||
('passing', [('shortpassing', 35), ('vision', 20), ('crossing', 20),
|
||||
('longpassing', 15), ('freekickaccuracy', 5), ('curve', 5)]),
|
||||
('dribbling', [('dribbling', 50), ('ballcontrol', 30), ('agility', 10),
|
||||
('balance', 5), ('reactions', 5)]),
|
||||
('defending', [('marking', 30), ('standingtackle', 30), ('interceptions', 20),
|
||||
('headingaccuracy', 10), ('slidingtackle', 10)]),
|
||||
('physical', [('strength', 50), ('stamina', 25), ('aggression', 20),
|
||||
('jumping', 5)]),
|
||||
]
|
||||
|
||||
# Keeper card face. The five gk* columns are used at 100% -- they ARE the card
|
||||
# numbers, no arithmetic. The speed slot is the only weighted one.
|
||||
GK = [
|
||||
('diving', [('gkdiving', 100)]),
|
||||
('handling', [('gkhandling', 100)]),
|
||||
('kicking', [('gkkicking', 100)]),
|
||||
('reflexes', [('gkreflexes', 100)]),
|
||||
('speed', [('acceleration', 60), ('sprintspeed', 40)]),
|
||||
('positioning', [('gkpositioning', 100)]),
|
||||
]
|
||||
|
||||
POSITION_NAMES = ['GK', 'SW', 'RWB', 'RB', 'RCB', 'CB', 'LCB', 'LB', 'LWB',
|
||||
'RDM', 'CDM', 'LDM', 'RM', 'RCM', 'CM', 'LCM', 'LM',
|
||||
'RAM', 'CAM', 'LAM', 'RF', 'CF', 'LF', 'RW', 'RS', 'ST',
|
||||
'LS', 'LW']
|
||||
|
||||
|
||||
def load(name):
|
||||
with open(os.path.join(TABLES, name + '.json')) as fh:
|
||||
return json.load(fh)
|
||||
|
||||
|
||||
def weighted(row, spec):
|
||||
return int(round(sum(row[c] * w for c, w in spec) / 100.0))
|
||||
|
||||
|
||||
def main():
|
||||
players = load('players')['rows']
|
||||
tpl = load('teamplayerlinks')['rows']
|
||||
teams = {t['teamid']: t for t in load('teams')['rows']}
|
||||
nations = {n['nationid']: n['nationname'] for n in load('nations')['rows']}
|
||||
|
||||
# A player appears in teamplayerlinks once per club AND once per national
|
||||
# side. The club is the row whose team is not a national team; nations
|
||||
# and teams share an id space only through teamnationlinks, so the cheap,
|
||||
# reliable discriminator is: the club link is the one with the lowest
|
||||
# teamid that is not the player's own nation-team. Keep every link too.
|
||||
nat_team = set()
|
||||
for t in load('teamnationlinks')['rows']:
|
||||
nat_team.add(t['teamid'])
|
||||
clubs = {}
|
||||
alllinks = {}
|
||||
for l in tpl:
|
||||
alllinks.setdefault(l['playerid'], []).append(l)
|
||||
if l['teamid'] in nat_team:
|
||||
continue
|
||||
prev = clubs.get(l['playerid'])
|
||||
if prev is None or l['teamid'] < prev['teamid']:
|
||||
clubs[l['playerid']] = l
|
||||
|
||||
out = []
|
||||
for p in players:
|
||||
pid = p['playerid']
|
||||
gk = p['preferredposition1'] == 0
|
||||
spec = GK if gk else OUTFIELD
|
||||
club = clubs.get(pid)
|
||||
out.append({
|
||||
'id': pid,
|
||||
'rating': p['overallrating'],
|
||||
'potential': p['potential'],
|
||||
'pos': p['preferredposition1'],
|
||||
'pos2': p['preferredposition2'],
|
||||
'pos3': p['preferredposition3'],
|
||||
'pos4': p['preferredposition4'],
|
||||
'posname': POSITION_NAMES[p['preferredposition1']]
|
||||
if 0 <= p['preferredposition1'] < len(POSITION_NAMES) else None,
|
||||
'nation': p['nationality'],
|
||||
'nationname': nations.get(p['nationality']),
|
||||
'team': club['teamid'] if club else 0,
|
||||
'teamname': teams.get(club['teamid'], {}).get('teamname') if club else None,
|
||||
'jersey': club['jerseynumber'] if club else 0,
|
||||
'foot': p['preferredfoot'],
|
||||
'skillmoves': p['skillmoves'],
|
||||
'weakfoot': p['weakfootabilitytypecode'],
|
||||
'height': p['height'],
|
||||
'weight': p['weight'],
|
||||
'gk': gk,
|
||||
'attrs': [weighted(p, s) for _, s in spec],
|
||||
})
|
||||
|
||||
doc = {
|
||||
'meta': {
|
||||
'source': 'FIFA17.exe resident database, via tools/db_dump.py',
|
||||
'players': len(out),
|
||||
'attr_order_outfield': [k for k, _ in OUTFIELD],
|
||||
'attr_order_gk': [k for k, _ in GK],
|
||||
'weights_outfield': {k: dict(v) for k, v in OUTFIELD},
|
||||
'weights_gk': {k: dict(v) for k, v in GK},
|
||||
'position_enum': POSITION_NAMES,
|
||||
},
|
||||
'players': out,
|
||||
}
|
||||
with open(OUT, 'w') as fh:
|
||||
json.dump(doc, fh, ensure_ascii=False, separators=(',', ':'))
|
||||
sys.stderr.write("wrote %s: %d players (%d keepers)\n"
|
||||
% (OUT, len(out), sum(1 for x in out if x['gk'])))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,569 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Dump FIFA 17's LOADED relational database (schema + every row) out of a live
|
||||
FIFA17.exe, or out of a saved memory image.
|
||||
|
||||
READ-ONLY. /proc/PID/mem is opened 'rb' and only ever seek()/read(). There is
|
||||
no write path in this file.
|
||||
|
||||
WHY THIS EXISTS
|
||||
---------------
|
||||
dbdata.dll on disk is packed (see dbdata_extract.py's docstring), so the game's
|
||||
tables cannot be read off disk. They ARE fully resident in the running process,
|
||||
in a self-describing form: the DB carries its own catalogue, its own column
|
||||
names, and its own bit-level row layout. This tool walks that catalogue and
|
||||
decodes the rows.
|
||||
|
||||
dbschema_probe.py got as far as the catalogue but had two defects, both fixed
|
||||
here and both worth writing down because they are easy to re-introduce:
|
||||
|
||||
1. THE NAME IS AT anchor+0x08, NOT AT anchor-0x20. Every catalogue record --
|
||||
table records and column records alike -- carries the constant qword
|
||||
0x07c20760 (SHARED_ANCHOR). Reading the name from the start of the
|
||||
32-byte-aligned slot picks up the PREVIOUS record's name, which silently
|
||||
mislabels every table by one slot: what the old tool called "gkcoachcards"
|
||||
is really `players`, what it called "teams" is really `teamplayerlinks`,
|
||||
and so on. The mislabelling is invisible because both names are real.
|
||||
The check that catches it: a table's column set must match its name
|
||||
(`nations` must contain nationname, `players` must contain acceleration).
|
||||
|
||||
2. --list MIXED COLUMNS INTO THE TABLE LIST. Column records and table
|
||||
records share the same anchor, so an anchor scan alone yields both. The
|
||||
discriminator used here is the TABLE VTABLE: a record is a table if and
|
||||
only if its descriptor's first qword is 0x07c31028. That takes 4126
|
||||
anchor hits down to exactly 149 tables, with zero column names among them.
|
||||
|
||||
THE STRUCTURES (resolved live 2026-08-04, pid 52703, game in the FUT club UI)
|
||||
------------------------------------------------------------------------------
|
||||
CATALOGUE RECORD (table). Found by scanning for SHARED_ANCHOR; the record
|
||||
starts 0x18 before it:
|
||||
+0x00 void* descriptor
|
||||
+0x08 u32 table name hash4 (repeated at descriptor+0x40)
|
||||
+0x0c u32 column count
|
||||
+0x10 void* column-definition array ("p2")
|
||||
+0x18 u64 SHARED_ANCHOR 0x07c20760
|
||||
+0x20 char* table name
|
||||
(stride 0x30)
|
||||
|
||||
COLUMN-DEFINITION RECORD (0x30 bytes, p2[i]) -- the human-readable half:
|
||||
+0x00 u32 kind: 1 = string, 2 = integer, 4 = date
|
||||
+0x04 u32 name hash4
|
||||
+0x08 u32 min value (as u32; may be negative, e.g. -1 for a position)
|
||||
+0x0c u32 max value
|
||||
+0x20 u64 SHARED_ANCHOR
|
||||
+0x28 char* column name
|
||||
|
||||
TABLE DESCRIPTOR (at the record's descriptor pointer):
|
||||
+0x00 u64 0x07c31028 (the table vtable -- the discriminator)
|
||||
+0x30 void* ROW BLOCK
|
||||
+0x40 u32 table name hash4 (self-identification)
|
||||
+0x44 u32 row size in BYTES
|
||||
+0x48 u32 max bit index (== rowsize*8 - 1)
|
||||
+0x7c u32 ROW COUNT
|
||||
+0x82 u16 column count
|
||||
+0x88 ... column layout array, `ncol` records of 16 bytes:
|
||||
u32 bit_offset, u32 name hash4, u32 bit_width, u32 flags
|
||||
(sorted by hash4, so it needs sorting by bit_offset to read)
|
||||
|
||||
ROW BLOCK (pointed at by descriptor+0x30) is preceded by a 16-byte header:
|
||||
-0x10 u32 byte size of the block
|
||||
-0x08 u64 0x2c020e60 (a second constant, a useful check)
|
||||
Rows are `rowsize` bytes, densely packed, no gaps.
|
||||
|
||||
FIELD DECODING
|
||||
--------------
|
||||
value = (int.from_bytes(row, 'little') >> bit_offset) & ((1 << width) - 1)
|
||||
Integer columns then add `min`, so a column declared min=-1 max=32 stores 0..33
|
||||
and reads back -1..32. This was confirmed on managercards.carddbid (bit 64,
|
||||
width 24) reading 1000001 on row 0 -- the exact value docs/managercards_ids.txt
|
||||
recorded from a completely independent live sweep.
|
||||
|
||||
STRINGS come in two forms and the tool decides per column:
|
||||
* INLINE when bit_offset + width <= the next column's bit_offset: the field
|
||||
is a fixed-size NUL-terminated char array inside the row. This is how
|
||||
nations.nationname, leagues.leaguename and teams.teamname are stored, and
|
||||
all three decode to real names.
|
||||
* OFFSET otherwise: the field is a 32-bit offset into a string pool that is
|
||||
NOT resident as a flat blob (searched for; not found). Those columns are
|
||||
emitted as raw integers and flagged `"storage": "offset-unresolved"` in the
|
||||
schema block. managercards.firstname/lastname and playernames.name are of
|
||||
this kind. Player names are already available from data/roster.json via
|
||||
dbdata_extract.py, so nothing depends on resolving them.
|
||||
|
||||
USAGE
|
||||
-----
|
||||
./db_dump.py --list # every table, row count, columns
|
||||
./db_dump.py --schema players # one table's column layout
|
||||
./db_dump.py --check # run the anchor checks, exit 1 on fail
|
||||
./db_dump.py --dump players teams -o DIR # dump named tables as JSON
|
||||
./db_dump.py --dump-all -o DIR # dump all 149
|
||||
./db_dump.py --save-mem DIR # snapshot the process (do this FIRST;
|
||||
# live memory is perishable)
|
||||
./db_dump.py --mem DIR ... # work from a snapshot, no live game
|
||||
|
||||
Requires ptrace access to FIFA17.exe (ptrace_scope=1 + same uid is enough) when
|
||||
reading live; --mem needs nothing but the snapshot directory.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import bisect
|
||||
import json
|
||||
import mmap
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
|
||||
SHARED_ANCHOR = 0x07C20760 # on every catalogue record, at record+0x18/+0x20
|
||||
TABLE_VTABLE = 0x07C31028 # descriptor[0] iff the record describes a table
|
||||
BLOCK_MARK = 0x2C020E60 # rowblock[-0x08]
|
||||
IDENT = re.compile(r'[A-Za-z][A-Za-z0-9_]*\Z')
|
||||
|
||||
# Anonymous mappings above this are the host libc arenas; the game DB is below.
|
||||
MAX_VA = 0x200000000
|
||||
|
||||
KIND_STRING, KIND_INT, KIND_DATE = 1, 2, 4
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# memory access
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def find_pid():
|
||||
for d in os.listdir('/proc'):
|
||||
if not d.isdigit():
|
||||
continue
|
||||
try:
|
||||
if open('/proc/%s/comm' % d).read().strip() == 'FIFA17.exe':
|
||||
return int(d)
|
||||
except OSError:
|
||||
pass
|
||||
raise SystemExit("FIFA17.exe is not running (use --mem DIR to work offline)")
|
||||
|
||||
|
||||
def anon_regions(pid):
|
||||
out = []
|
||||
for line in open('/proc/%d/maps' % pid):
|
||||
p = line.split()
|
||||
lo, hi = (int(x, 16) for x in p[0].split('-'))
|
||||
perms = p[1]
|
||||
path = p[5] if len(p) > 5 else ''
|
||||
if 'r' not in perms or path or lo >= MAX_VA:
|
||||
continue
|
||||
out.append((lo, hi, perms))
|
||||
return out
|
||||
|
||||
|
||||
class Image(object):
|
||||
"""Uniform read-only view over either a live process or a saved snapshot."""
|
||||
|
||||
def __init__(self, pid=None, memdir=None):
|
||||
self.regions = [] # list of dicts lo/hi
|
||||
self._buf = {}
|
||||
if memdir:
|
||||
idx = json.load(open(os.path.join(memdir, 'index.json')))
|
||||
idx.sort(key=lambda r: r['lo'])
|
||||
self.regions = idx
|
||||
self.memdir = memdir
|
||||
self.live = None
|
||||
else:
|
||||
self.live = open('/proc/%d/mem' % pid, 'rb', 0)
|
||||
self.memdir = None
|
||||
for lo, hi, perms in anon_regions(pid):
|
||||
self.regions.append({'lo': lo, 'hi': hi, 'perms': perms})
|
||||
self._los = [r['lo'] for r in self.regions]
|
||||
|
||||
# -- region buffers ----------------------------------------------------- #
|
||||
def buf(self, i):
|
||||
if i not in self._buf:
|
||||
r = self.regions[i]
|
||||
if self.memdir:
|
||||
f = open(os.path.join(self.memdir, r['file']), 'rb')
|
||||
self._buf[i] = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
|
||||
else:
|
||||
self.live.seek(r['lo'])
|
||||
self._buf[i] = self.live.read(r['hi'] - r['lo'])
|
||||
return self._buf[i]
|
||||
|
||||
def _find(self, a):
|
||||
i = bisect.bisect_right(self._los, a) - 1
|
||||
if i >= 0 and self.regions[i]['lo'] <= a < self.regions[i]['hi']:
|
||||
return i
|
||||
return -1
|
||||
|
||||
def read(self, a, n):
|
||||
if a is None or a <= 0:
|
||||
return b''
|
||||
i = self._find(a)
|
||||
if i < 0:
|
||||
return b''
|
||||
o = a - self.regions[i]['lo']
|
||||
return bytes(self.buf(i)[o:o + n])
|
||||
|
||||
def u32(self, a):
|
||||
b = self.read(a, 4)
|
||||
return struct.unpack('<I', b)[0] if len(b) == 4 else None
|
||||
|
||||
def u64(self, a):
|
||||
b = self.read(a, 8)
|
||||
return struct.unpack('<Q', b)[0] if len(b) == 8 else None
|
||||
|
||||
def cstr(self, a, maxlen=96):
|
||||
b = self.read(a, maxlen)
|
||||
j = b.find(b'\x00')
|
||||
if j <= 0:
|
||||
return None
|
||||
s = b[:j]
|
||||
return s.decode('latin1') if re.fullmatch(rb'[ -~]+', s) else None
|
||||
|
||||
def save(self, outdir):
|
||||
"""Snapshot every region to disk (do this first: live data is perishable)."""
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
idx = []
|
||||
for i, r in enumerate(self.regions):
|
||||
d = self.buf(i)
|
||||
fn = '%012x.bin' % r['lo']
|
||||
with open(os.path.join(outdir, fn), 'wb') as fh:
|
||||
fh.write(d)
|
||||
idx.append({'lo': r['lo'], 'hi': r['lo'] + len(d),
|
||||
'perms': r.get('perms', 'rw-p'), 'file': fn})
|
||||
json.dump(idx, open(os.path.join(outdir, 'index.json'), 'w'), indent=1)
|
||||
return sum(r['hi'] - r['lo'] for r in idx)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# catalogue walk
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def s32(v):
|
||||
return v - (1 << 32) if v is not None and v >= (1 << 31) else v
|
||||
|
||||
|
||||
def anchor_hits(img):
|
||||
"""Addresses of every SHARED_ANCHOR qword, 8-byte aligned."""
|
||||
pat = struct.pack('<Q', SHARED_ANCHOR)
|
||||
out = []
|
||||
for i, r in enumerate(img.regions):
|
||||
b = img.buf(i)
|
||||
st = 0
|
||||
while True:
|
||||
j = b.find(pat, st)
|
||||
if j < 0:
|
||||
break
|
||||
st = j + 1
|
||||
if j % 8 == 0:
|
||||
out.append(r['lo'] + j)
|
||||
return out
|
||||
|
||||
|
||||
def catalogue(img):
|
||||
"""name -> table dict. Tables only: the descriptor vtable is the filter."""
|
||||
tables = {}
|
||||
for anch in anchor_hits(img):
|
||||
name = img.cstr(img.u64(anch + 8) or 0, 64)
|
||||
if not name or not IDENT.match(name):
|
||||
continue
|
||||
desc = img.u64(anch - 0x18)
|
||||
head = img.read(desc or 0, 0x50)
|
||||
if len(head) < 0x50 or struct.unpack_from('<Q', head, 0)[0] != TABLE_VTABLE:
|
||||
continue
|
||||
h4, ncol = struct.unpack_from('<II', img.read(anch - 0x10, 8), 0)
|
||||
dh4, rowsz, maxbit, _ = struct.unpack_from('<IIII', head, 0x40)
|
||||
if dh4 != h4: # descriptor must self-identify
|
||||
continue
|
||||
tables[name] = {
|
||||
'name': name, 'anchor': anch, 'desc': desc, 'hash4': h4,
|
||||
'ncol': ncol, 'p2': img.u64(anch - 8),
|
||||
'rowsize': rowsz, 'maxbit': maxbit,
|
||||
'rowcount': img.u32(desc + 0x7c),
|
||||
'rowblock': img.u64(desc + 0x30),
|
||||
}
|
||||
return tables
|
||||
|
||||
|
||||
def columns(img, t):
|
||||
"""Full column list, sorted by bit offset, with names/kinds/ranges."""
|
||||
defs = {}
|
||||
for i in range(t['ncol']):
|
||||
b = img.read(t['p2'] + i * 0x30, 0x30)
|
||||
if len(b) < 0x30:
|
||||
break
|
||||
kind, h4 = struct.unpack_from('<II', b, 0)
|
||||
mn, mx = struct.unpack_from('<II', b, 8)
|
||||
nm = img.cstr(struct.unpack_from('<Q', b, 0x28)[0], 64)
|
||||
if nm:
|
||||
defs[h4] = (nm, kind, s32(mn), s32(mx))
|
||||
lay = img.read(t['desc'] + 0x88, t['ncol'] * 16)
|
||||
cols = []
|
||||
for i in range(t['ncol']):
|
||||
if (i + 1) * 16 > len(lay):
|
||||
break
|
||||
bit, h4, width, flags = struct.unpack_from('<IIII', lay, i * 16)
|
||||
nm, kind, mn, mx = defs.get(h4, ('?%08x' % h4, 0, 0, 0))
|
||||
cols.append({'name': nm, 'bit': bit, 'width': width, 'kind': kind,
|
||||
'min': mn, 'max': mx, 'flags': flags})
|
||||
cols.sort(key=lambda c: c['bit'])
|
||||
# inline vs offset strings, decided by whether the declared width fits
|
||||
for i, c in enumerate(cols):
|
||||
nxt = cols[i + 1]['bit'] if i + 1 < len(cols) else t['rowsize'] * 8
|
||||
if c['kind'] == KIND_STRING:
|
||||
if c['bit'] + c['width'] <= nxt:
|
||||
c['storage'] = 'inline-string'
|
||||
else:
|
||||
c['storage'] = 'offset-unresolved'
|
||||
c['width'] = 32
|
||||
else:
|
||||
c['storage'] = 'int'
|
||||
return cols
|
||||
|
||||
|
||||
def read_rows(img, t, cols, limit=None):
|
||||
n = t['rowcount'] or 0
|
||||
if limit is not None:
|
||||
n = min(n, limit)
|
||||
rsz = t['rowsize']
|
||||
if n == 0 or rsz == 0 or not t['rowblock']:
|
||||
return []
|
||||
blob = img.read(t['rowblock'], n * rsz)
|
||||
if len(blob) < n * rsz:
|
||||
raise RuntimeError('%s: row block short: got %d of %d bytes'
|
||||
% (t['name'], len(blob), n * rsz))
|
||||
out = []
|
||||
for i in range(n):
|
||||
row = blob[i * rsz:(i + 1) * rsz]
|
||||
raw = int.from_bytes(row, 'little')
|
||||
rec = {}
|
||||
for c in cols:
|
||||
if c['storage'] == 'inline-string':
|
||||
lo = c['bit'] // 8
|
||||
s = row[lo:lo + c['width'] // 8]
|
||||
j = s.find(b'\x00')
|
||||
rec[c['name']] = (s if j < 0 else s[:j]).decode('latin1')
|
||||
else:
|
||||
v = (raw >> c['bit']) & ((1 << c['width']) - 1)
|
||||
rec[c['name']] = v + c['min'] if c['kind'] == KIND_INT else v
|
||||
out.append(rec)
|
||||
return out
|
||||
|
||||
|
||||
def block_size(img, t):
|
||||
h = img.read((t['rowblock'] or 0) - 0x10, 16)
|
||||
if len(h) < 16:
|
||||
return None, None
|
||||
return struct.unpack_from('<I', h, 0)[0], struct.unpack_from('<Q', h, 8)[0]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# checks -- every dump must be justified against something independent
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
CHECKS = []
|
||||
|
||||
|
||||
def check(msg, cond):
|
||||
CHECKS.append((msg, bool(cond)))
|
||||
print(" [%s] %s" % ("PASS" if cond else "FAIL", msg))
|
||||
return bool(cond)
|
||||
|
||||
|
||||
def run_checks(img, tables):
|
||||
print("sanity checks (independent ground truth):")
|
||||
ok = True
|
||||
|
||||
# (1) the catalogue must not contain column names masquerading as tables
|
||||
ok &= check("catalogue holds no known column name as a table "
|
||||
"(acceleration/carddbid/assetid absent)",
|
||||
not ({'acceleration', 'carddbid', 'assetid'} & set(tables)))
|
||||
|
||||
# (2) each table's columns must match its own name
|
||||
for tn, must in (('nations', 'nationname'), ('leagues', 'leaguename'),
|
||||
('teams', 'teamname'), ('players', 'acceleration'),
|
||||
('managercards', 'carddbid'),
|
||||
('teamplayerlinks', 'jerseynumber')):
|
||||
t = tables.get(tn)
|
||||
names = {c['name'] for c in columns(img, t)} if t else set()
|
||||
ok &= check("table %-16s contains column %-14s" % (tn, must), must in names)
|
||||
|
||||
# (3) THE anchor: playerid 20801 is Cristiano Ronaldo, FIFA 17 -- 94 rated,
|
||||
# a left winger, Portuguese, 185 cm, at Real Madrid.
|
||||
t = tables['players']
|
||||
cols = columns(img, t)
|
||||
pl = {r['playerid']: r for r in read_rows(img, t, cols)}
|
||||
r = pl.get(20801)
|
||||
ok &= check("players has 20801", r is not None)
|
||||
if r:
|
||||
ok &= check(" 20801 overallrating == 94 (got %s)" % r['overallrating'],
|
||||
r['overallrating'] == 94)
|
||||
ok &= check(" 20801 preferredposition1 == 27 (LW) (got %s)"
|
||||
% r['preferredposition1'], r['preferredposition1'] == 27)
|
||||
ok &= check(" 20801 nationality == 38 (Portugal) (got %s)" % r['nationality'],
|
||||
r['nationality'] == 38)
|
||||
ok &= check(" 20801 height == 185 cm (got %s)" % r['height'], r['height'] == 185)
|
||||
ok &= check(" 20801 preferredfoot == 1 (right) (got %s)" % r['preferredfoot'],
|
||||
r['preferredfoot'] == 1)
|
||||
|
||||
# nation 38 must literally spell Portugal, from a different table
|
||||
nt = tables['nations']
|
||||
nat = {x['nationid']: x for x in read_rows(img, nt, columns(img, nt))}
|
||||
ok &= check("nations[38].nationname == 'Portugal' (got %r)"
|
||||
% (nat.get(38, {}).get('nationname')),
|
||||
nat.get(38, {}).get('nationname') == 'Portugal')
|
||||
|
||||
# and teamplayerlinks must put him at Real Madrid
|
||||
tp = tables['teamplayerlinks']
|
||||
links = [x for x in read_rows(img, tp, columns(img, tp)) if x['playerid'] == 20801]
|
||||
tt = tables['teams']
|
||||
teams = {x['teamid']: x for x in read_rows(img, tt, columns(img, tt))}
|
||||
names = sorted({teams.get(l['teamid'], {}).get('teamname') for l in links})
|
||||
ok &= check("teamplayerlinks[20801] includes Real Madrid (got %s)" % (names,),
|
||||
'Real Madrid' in names)
|
||||
|
||||
# (4) managercards' id space, measured independently in a previous session
|
||||
# (docs/managercards_ids.txt: 416 ids from 1000001 upward)
|
||||
mt = tables['managercards']
|
||||
mc = read_rows(img, mt, columns(img, mt))
|
||||
ids = sorted(x['carddbid'] for x in mc)
|
||||
ok &= check("managercards row 0 carddbid == 1000001 (got %s)" % (ids[0] if ids else None),
|
||||
ids and ids[0] == 1000001)
|
||||
ok &= check("managercards ids all in [1000001, 1002000] (got %s..%s, n=%d)"
|
||||
% (ids[0] if ids else None, ids[-1] if ids else None, len(ids)),
|
||||
ids and 1000001 <= ids[0] and ids[-1] <= 1002000)
|
||||
ok &= check("managercards assetid == carddbid on every row",
|
||||
all(x['assetid'] == x['carddbid'] for x in mc))
|
||||
|
||||
# (5) every table's row block must carry the block marker, and every byte of
|
||||
# rowcount*rowsize must actually be readable. (The u32 at block-0x10 is
|
||||
# a byte count only for the heap-resident tables; for the ~17 small
|
||||
# tables that live in the 0x07xxxxxx pool it is something else, so it is
|
||||
# reported but not required.)
|
||||
unmarked, short, oddsize = [], [], []
|
||||
for n, t in tables.items():
|
||||
if not t['rowcount'] or not t['rowblock']:
|
||||
continue
|
||||
need = t['rowcount'] * t['rowsize']
|
||||
sz, mark = block_size(img, t)
|
||||
if mark != BLOCK_MARK:
|
||||
unmarked.append(n)
|
||||
if len(img.read(t['rowblock'], need)) < need:
|
||||
short.append(n)
|
||||
if sz is None or sz < need:
|
||||
oddsize.append(n)
|
||||
ok &= check("every non-empty table's row block carries the 0x2c020e60 marker "
|
||||
"(%d without)" % len(unmarked), not unmarked)
|
||||
ok &= check("every non-empty table's rowcount*rowsize bytes are readable "
|
||||
"(%d short)" % len(short), not short)
|
||||
print(" [note] %d small pool-resident tables have a block-size field that is "
|
||||
"not a byte count: %s" % (len(oddsize), ", ".join(sorted(oddsize))))
|
||||
|
||||
return ok
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def dump_table(img, t, outdir, limit=None):
|
||||
cols = columns(img, t)
|
||||
rows = read_rows(img, t, cols, limit)
|
||||
sz, mark = block_size(img, t)
|
||||
doc = {
|
||||
'table': t['name'],
|
||||
'source': 'FIFA17.exe resident database (tools/db_dump.py)',
|
||||
'rowcount': t['rowcount'],
|
||||
'rowsize_bytes': t['rowsize'],
|
||||
'rows_emitted': len(rows),
|
||||
'rowblock': '0x%x' % (t['rowblock'] or 0),
|
||||
'rowblock_bytes': sz,
|
||||
'descriptor': '0x%x' % t['desc'],
|
||||
'schema': [{'name': c['name'], 'bit': c['bit'], 'width': c['width'],
|
||||
'kind': {1: 'string', 2: 'int', 4: 'date'}.get(c['kind'], 'unknown'),
|
||||
'min': c['min'], 'max': c['max'], 'storage': c['storage']}
|
||||
for c in cols],
|
||||
'rows': rows,
|
||||
}
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
p = os.path.join(outdir, t['name'] + '.json')
|
||||
with open(p, 'w') as fh:
|
||||
json.dump(doc, fh, ensure_ascii=False, separators=(',', ':'))
|
||||
return p, len(rows), os.path.getsize(p)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.split('\n')[0])
|
||||
ap.add_argument('--pid', type=int)
|
||||
ap.add_argument('--mem', help='work from a snapshot directory instead of a live game')
|
||||
ap.add_argument('--save-mem', help='snapshot the process to this directory and exit')
|
||||
ap.add_argument('--list', action='store_true')
|
||||
ap.add_argument('--schema', action='append', default=[])
|
||||
ap.add_argument('--dump', nargs='*')
|
||||
ap.add_argument('--dump-all', action='store_true')
|
||||
ap.add_argument('--check', action='store_true')
|
||||
ap.add_argument('--limit', type=int)
|
||||
ap.add_argument('-o', '--out', default='../data/tables')
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.mem:
|
||||
img = Image(memdir=a.mem)
|
||||
sys.stderr.write("snapshot %s: %d regions\n" % (a.mem, len(img.regions)))
|
||||
else:
|
||||
pid = a.pid or find_pid()
|
||||
img = Image(pid=pid)
|
||||
sys.stderr.write("FIFA17.exe pid %d: %d anonymous regions\n"
|
||||
% (pid, len(img.regions)))
|
||||
if a.save_mem:
|
||||
n = img.save(a.save_mem)
|
||||
sys.stderr.write("saved %.1f MB to %s\n" % (n / 1048576.0, a.save_mem))
|
||||
return 0
|
||||
|
||||
tables = catalogue(img)
|
||||
sys.stderr.write("catalogue: %d tables\n" % len(tables))
|
||||
|
||||
if a.list:
|
||||
for n in sorted(tables):
|
||||
t = tables[n]
|
||||
cols = columns(img, t)
|
||||
print("%-34s rows=%-7s rowsize=%-5d cols=%-4d | %s"
|
||||
% (n, t['rowcount'], t['rowsize'], t['ncol'],
|
||||
", ".join(c['name'] for c in cols[:6])))
|
||||
return 0
|
||||
|
||||
for n in a.schema:
|
||||
t = tables.get(n)
|
||||
if not t:
|
||||
print("%s: not in the catalogue" % n)
|
||||
continue
|
||||
print("=== %s rows=%s rowsize=%d bytes cols=%d rowblock=0x%x"
|
||||
% (n, t['rowcount'], t['rowsize'], t['ncol'], t['rowblock'] or 0))
|
||||
for c in columns(img, t):
|
||||
print(" bit %4d w %-3d %-18s %-9s min %-8s max %s"
|
||||
% (c['bit'], c['width'], c['name'], c['storage'], c['min'], c['max']))
|
||||
|
||||
rc = 0
|
||||
if a.check:
|
||||
rc = 0 if run_checks(img, tables) else 1
|
||||
|
||||
want = None
|
||||
if a.dump_all:
|
||||
want = sorted(tables)
|
||||
elif a.dump is not None:
|
||||
want = a.dump or sorted(tables)
|
||||
if want:
|
||||
total = 0
|
||||
for n in want:
|
||||
t = tables.get(n)
|
||||
if not t:
|
||||
sys.stderr.write(" %s: not in the catalogue\n" % n)
|
||||
continue
|
||||
try:
|
||||
p, nr, nb = dump_table(img, t, a.out, a.limit)
|
||||
except Exception as e: # noqa: BLE001
|
||||
sys.stderr.write(" %-30s FAILED: %s\n" % (n, e))
|
||||
rc = 1
|
||||
continue
|
||||
total += nb
|
||||
sys.stderr.write(" %-34s %7d rows %9.1f KB %s\n"
|
||||
% (n, nr, nb / 1024.0, p))
|
||||
sys.stderr.write("wrote %.1f MB into %s\n" % (total / 1048576.0, a.out))
|
||||
return rc
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -113,6 +113,27 @@ def _load(name, default):
|
||||
|
||||
ROSTER = _load("roster.json", [])
|
||||
KNOWN_POSITIONS = {int(k): v for k, v in _load("positions.json", {}).items()}
|
||||
|
||||
# data/pool.json -- the REAL thing, and it supersedes everything below it.
|
||||
#
|
||||
# Built 2026-08-05 from the game's own resident database (data/tables/*.json, dumped
|
||||
# read-only by tools/db_dump.py, then tools/build_player_facts.py). Per player it
|
||||
# carries the MEASURED position (players.preferredposition1), nationality, teamid,
|
||||
# leagueid (via leagueteamlinks) and the six card attributes.
|
||||
#
|
||||
# The six attributes are not columns: they are a weighted sum of the 29 base
|
||||
# attributes, and the weights come from the game's OWN `playerattributesmapping`
|
||||
# table rather than from published formulas. The result checks out against real FIFA
|
||||
# 17 cards: Messi 89/90/86/96/26/61 and Ibrahimovic 72/90/81/85/31/86 are exact,
|
||||
# Suarez is one off on physical, Ronaldo within two on pace and shooting.
|
||||
#
|
||||
# NOTE THE REVERSAL on nation/team/league. When those fields were unknown we sent
|
||||
# ZERO so the client would fill its own values (the merge fills them only when they
|
||||
# arrive zero). Now that we hold the game's own numbers there is nothing to gain, and
|
||||
# zeros actively HURT: our club-stats drill-downs bucket by the item's own nation and
|
||||
# leagueId, so a club full of zeros would have emptied the per-nation and per-league
|
||||
# panels that were fixed yesterday. Send the real values.
|
||||
POOL_FACTS = _load("pool.json", [])
|
||||
# Hand-checked positions, carried over from the curated pool this file replaces.
|
||||
# They are KNOWLEDGE, not measurement, which is why they rank below the codes the
|
||||
# game itself supplied. They exist because a synthetic position is unnoticeable on
|
||||
@@ -148,6 +169,21 @@ def _attrs(rating, pos):
|
||||
|
||||
|
||||
def _build():
|
||||
if POOL_FACTS:
|
||||
pool = []
|
||||
for r in POOL_FACTS:
|
||||
pid, rating = r["id"], r["rating"]
|
||||
if not pid or rating <= 0:
|
||||
# A zero id is not a harmless skip: the registrar writes
|
||||
# *(item+0x10) = 0 and the card view-model dereferences it with no
|
||||
# null check, so a zero id reaching the card UI is a crash.
|
||||
continue
|
||||
pool.append((pid, rating, r["pos"], r["nation"], r["league"], r["team"],
|
||||
list(r["attrs"])))
|
||||
return pool
|
||||
# Fallback: the rating-index roster, with synthetic positions and attributes.
|
||||
# Kept so the pool still builds if data/pool.json is missing, but everything it
|
||||
# produces below is a guess where the block above is a measurement.
|
||||
pool = []
|
||||
for r in ROSTER:
|
||||
pid, rating = r["id"], r["rating"]
|
||||
@@ -203,7 +239,12 @@ if __name__ == "__main__":
|
||||
print(" %-7s %5d ratings %d-%d" % (name, len(sel),
|
||||
min(p[1] for p in sel),
|
||||
max(p[1] for p in sel)))
|
||||
print("known positions: %d (the rest are synthetic)" % len(KNOWN_POSITIONS))
|
||||
if POOL_FACTS:
|
||||
print("source: data/pool.json -- positions, nation, club, league and all six "
|
||||
"attributes MEASURED from the game's own database")
|
||||
else:
|
||||
print("source: data/roster.json FALLBACK -- %d real positions, the rest "
|
||||
"synthetic, attributes derived from rating" % len(KNOWN_POSITIONS))
|
||||
print("\ntop 10 by rating:")
|
||||
for p in sorted(POOL, key=lambda x: -x[1])[:10]:
|
||||
print(" %-7s %-3s %-4s %-26s %s" % (p[0], p[1], p[2], name_of(p[0]), p[6]))
|
||||
|
||||
Reference in New Issue
Block a user