funman300/OpenFUT
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9faaf12dd7625109cb3afe70e3b73c7747eb0c29
OpenFUT/fifa17-recon/docker/fifa17-python/tools/db_dump.py
T
root 70a64e3709 fifa17-python: commit working FUT backend deployment (client/server split) 
Freeze the running offline FUT backend into version control as
fifa17-recon/docker/fifa17-python/ - declarative and rebuildable from a
fresh checkout:

* OPENFUT_BIND / OPENFUT_ADVERTISE client/server split in the responders
  (lsx, blaze, roster, utas, pow) + entrypoint.sh; OPENFUT_ADVERTISE is
  required for remote mode (compose and entrypoint fail without it)
* docker-compose.yml reproducing the frozen baseline container exactly
  (env, ports incl. the 8085->8080 POW-content remap, /state bind, restart)
* .env.example / .env for site config - the LAN IP is never hardcoded in source
* tools/ + data/ staged from openfut-fut-backend:python-baseline-2026-08-10,
  verified byte-identical to the running container at freeze time
* client_arm.sh (the 105 client-side arming counterpart)
* Dockerfile bakes /app/SHA256SUMS.txt so any image is self-identifying
* docs/BASELINE-python-2026-08-10.md: frozen image/container/hash record,
  restore instructions and rebuild-equivalence procedure

Secrets (redir key/cert, .env) and runtime state (docker/state) stay gitignored.
The live container is untouched pending the .105 launcher audit.
2026-08-10 23:54:04 +00:00

570 lines
23 KiB
Python
Raw Blame History

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Dump FIFA 17's LOADED relational database (schema + every row) out of a live
FIFA17.exe, or out of a saved memory image.
READ-ONLY. /proc/PID/mem is opened 'rb' and only ever seek()/read(). There is
no write path in this file.
WHY THIS EXISTS
---------------
dbdata.dll on disk is packed (see dbdata_extract.py's docstring), so the game's
tables cannot be read off disk. They ARE fully resident in the running process,
in a self-describing form: the DB carries its own catalogue, its own column
names, and its own bit-level row layout. This tool walks that catalogue and
decodes the rows.
dbschema_probe.py got as far as the catalogue but had two defects, both fixed
here and both worth writing down because they are easy to re-introduce:
1. THE NAME IS AT anchor+0x08, NOT AT anchor-0x20. Every catalogue record --
table records and column records alike -- carries the constant qword
0x07c20760 (SHARED_ANCHOR). Reading the name from the start of the
32-byte-aligned slot picks up the PREVIOUS record's name, which silently
mislabels every table by one slot: what the old tool called "gkcoachcards"
is really `players`, what it called "teams" is really `teamplayerlinks`,
and so on. The mislabelling is invisible because both names are real.
The check that catches it: a table's column set must match its name
(`nations` must contain nationname, `players` must contain acceleration).
2. --list MIXED COLUMNS INTO THE TABLE LIST. Column records and table
records share the same anchor, so an anchor scan alone yields both. The
discriminator used here is the TABLE VTABLE: a record is a table if and
only if its descriptor's first qword is 0x07c31028. That takes 4126
anchor hits down to exactly 149 tables, with zero column names among them.
THE STRUCTURES (resolved live 2026-08-04, pid 52703, game in the FUT club UI)
------------------------------------------------------------------------------
CATALOGUE RECORD (table). Found by scanning for SHARED_ANCHOR; the record
starts 0x18 before it:
+0x00 void* descriptor
+0x08 u32 table name hash4 (repeated at descriptor+0x40)
+0x0c u32 column count
+0x10 void* column-definition array ("p2")
+0x18 u64 SHARED_ANCHOR 0x07c20760
+0x20 char* table name
(stride 0x30)
COLUMN-DEFINITION RECORD (0x30 bytes, p2[i]) -- the human-readable half:
+0x00 u32 kind: 1 = string, 2 = integer, 4 = date
+0x04 u32 name hash4
+0x08 u32 min value (as u32; may be negative, e.g. -1 for a position)
+0x0c u32 max value
+0x20 u64 SHARED_ANCHOR
+0x28 char* column name
TABLE DESCRIPTOR (at the record's descriptor pointer):
+0x00 u64 0x07c31028 (the table vtable -- the discriminator)
+0x30 void* ROW BLOCK
+0x40 u32 table name hash4 (self-identification)
+0x44 u32 row size in BYTES
+0x48 u32 max bit index (== rowsize*8 - 1)
+0x7c u32 ROW COUNT
+0x82 u16 column count
+0x88 ... column layout array, `ncol` records of 16 bytes:
u32 bit_offset, u32 name hash4, u32 bit_width, u32 flags
(sorted by hash4, so it needs sorting by bit_offset to read)
ROW BLOCK (pointed at by descriptor+0x30) is preceded by a 16-byte header:
-0x10 u32 byte size of the block
-0x08 u64 0x2c020e60 (a second constant, a useful check)
Rows are `rowsize` bytes, densely packed, no gaps.
FIELD DECODING
--------------
value = (int.from_bytes(row, 'little') >> bit_offset) & ((1 << width) - 1)
Integer columns then add `min`, so a column declared min=-1 max=32 stores 0..33
and reads back -1..32. This was confirmed on managercards.carddbid (bit 64,
width 24) reading 1000001 on row 0 -- the exact value docs/managercards_ids.txt
recorded from a completely independent live sweep.
STRINGS come in two forms and the tool decides per column:
* INLINE when bit_offset + width <= the next column's bit_offset: the field
is a fixed-size NUL-terminated char array inside the row. This is how
nations.nationname, leagues.leaguename and teams.teamname are stored, and
all three decode to real names.
* OFFSET otherwise: the field is a 32-bit offset into a string pool that is
NOT resident as a flat blob (searched for; not found). Those columns are
emitted as raw integers and flagged `"storage": "offset-unresolved"` in the
schema block. managercards.firstname/lastname and playernames.name are of
this kind. Player names are already available from data/roster.json via
dbdata_extract.py, so nothing depends on resolving them.
USAGE
-----
./db_dump.py --list # every table, row count, columns
./db_dump.py --schema players # one table's column layout
./db_dump.py --check # run the anchor checks, exit 1 on fail
./db_dump.py --dump players teams -o DIR # dump named tables as JSON
./db_dump.py --dump-all -o DIR # dump all 149
./db_dump.py --save-mem DIR # snapshot the process (do this FIRST;
# live memory is perishable)
./db_dump.py --mem DIR ... # work from a snapshot, no live game
Requires ptrace access to FIFA17.exe (ptrace_scope=1 + same uid is enough) when
reading live; --mem needs nothing but the snapshot directory.
"""
import argparse
import bisect
import json
import mmap
import os
import re
import struct
import sys
SHARED_ANCHOR = 0x07C20760 # on every catalogue record, at record+0x18/+0x20
TABLE_VTABLE = 0x07C31028 # descriptor[0] iff the record describes a table
BLOCK_MARK = 0x2C020E60 # rowblock[-0x08]
IDENT = re.compile(r'[A-Za-z][A-Za-z0-9_]*\Z')
# Anonymous mappings above this are the host libc arenas; the game DB is below.
MAX_VA = 0x200000000
KIND_STRING, KIND_INT, KIND_DATE = 1, 2, 4
# --------------------------------------------------------------------------- #
# memory access
# --------------------------------------------------------------------------- #
def find_pid():
for d in os.listdir('/proc'):
if not d.isdigit():
continue
try:
if open('/proc/%s/comm' % d).read().strip() == 'FIFA17.exe':
return int(d)
except OSError:
pass
raise SystemExit("FIFA17.exe is not running (use --mem DIR to work offline)")
def anon_regions(pid):
out = []
for line in open('/proc/%d/maps' % pid):
p = line.split()
lo, hi = (int(x, 16) for x in p[0].split('-'))
perms = p[1]
path = p[5] if len(p) > 5 else ''
if 'r' not in perms or path or lo >= MAX_VA:
continue
out.append((lo, hi, perms))
return out
class Image(object):
"""Uniform read-only view over either a live process or a saved snapshot."""
def __init__(self, pid=None, memdir=None):
self.regions = [] # list of dicts lo/hi
self._buf = {}
if memdir:
idx = json.load(open(os.path.join(memdir, 'index.json')))
idx.sort(key=lambda r: r['lo'])
self.regions = idx
self.memdir = memdir
self.live = None
else:
self.live = open('/proc/%d/mem' % pid, 'rb', 0)
self.memdir = None
for lo, hi, perms in anon_regions(pid):
self.regions.append({'lo': lo, 'hi': hi, 'perms': perms})
self._los = [r['lo'] for r in self.regions]
# -- region buffers ----------------------------------------------------- #
def buf(self, i):
if i not in self._buf:
r = self.regions[i]
if self.memdir:
f = open(os.path.join(self.memdir, r['file']), 'rb')
self._buf[i] = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
else:
self.live.seek(r['lo'])
self._buf[i] = self.live.read(r['hi'] - r['lo'])
return self._buf[i]
def _find(self, a):
i = bisect.bisect_right(self._los, a) - 1
if i >= 0 and self.regions[i]['lo'] <= a < self.regions[i]['hi']:
return i
return -1
def read(self, a, n):
if a is None or a <= 0:
return b''
i = self._find(a)
if i < 0:
return b''
o = a - self.regions[i]['lo']
return bytes(self.buf(i)[o:o + n])
def u32(self, a):
b = self.read(a, 4)
return struct.unpack('<I', b)[0] if len(b) == 4 else None
def u64(self, a):
b = self.read(a, 8)
return struct.unpack('<Q', b)[0] if len(b) == 8 else None
def cstr(self, a, maxlen=96):
b = self.read(a, maxlen)
j = b.find(b'\x00')
if j <= 0:
return None
s = b[:j]
return s.decode('latin1') if re.fullmatch(rb'[ -~]+', s) else None
def save(self, outdir):
"""Snapshot every region to disk (do this first: live data is perishable)."""
os.makedirs(outdir, exist_ok=True)
idx = []
for i, r in enumerate(self.regions):
d = self.buf(i)
fn = '%012x.bin' % r['lo']
with open(os.path.join(outdir, fn), 'wb') as fh:
fh.write(d)
idx.append({'lo': r['lo'], 'hi': r['lo'] + len(d),
'perms': r.get('perms', 'rw-p'), 'file': fn})
json.dump(idx, open(os.path.join(outdir, 'index.json'), 'w'), indent=1)
return sum(r['hi'] - r['lo'] for r in idx)
# --------------------------------------------------------------------------- #
# catalogue walk
# --------------------------------------------------------------------------- #
def s32(v):
return v - (1 << 32) if v is not None and v >= (1 << 31) else v
def anchor_hits(img):
"""Addresses of every SHARED_ANCHOR qword, 8-byte aligned."""
pat = struct.pack('<Q', SHARED_ANCHOR)
out = []
for i, r in enumerate(img.regions):
b = img.buf(i)
st = 0
while True:
j = b.find(pat, st)
if j < 0:
break
st = j + 1
if j % 8 == 0:
out.append(r['lo'] + j)
return out
def catalogue(img):
"""name -> table dict. Tables only: the descriptor vtable is the filter."""
tables = {}
for anch in anchor_hits(img):
name = img.cstr(img.u64(anch + 8) or 0, 64)
if not name or not IDENT.match(name):
continue
desc = img.u64(anch - 0x18)
head = img.read(desc or 0, 0x50)
if len(head) < 0x50 or struct.unpack_from('<Q', head, 0)[0] != TABLE_VTABLE:
continue
h4, ncol = struct.unpack_from('<II', img.read(anch - 0x10, 8), 0)
dh4, rowsz, maxbit, _ = struct.unpack_from('<IIII', head, 0x40)
if dh4 != h4: # descriptor must self-identify
continue
tables[name] = {
'name': name, 'anchor': anch, 'desc': desc, 'hash4': h4,
'ncol': ncol, 'p2': img.u64(anch - 8),
'rowsize': rowsz, 'maxbit': maxbit,
'rowcount': img.u32(desc + 0x7c),
'rowblock': img.u64(desc + 0x30),
}
return tables
def columns(img, t):
"""Full column list, sorted by bit offset, with names/kinds/ranges."""
defs = {}
for i in range(t['ncol']):
b = img.read(t['p2'] + i * 0x30, 0x30)
if len(b) < 0x30:
break
kind, h4 = struct.unpack_from('<II', b, 0)
mn, mx = struct.unpack_from('<II', b, 8)
nm = img.cstr(struct.unpack_from('<Q', b, 0x28)[0], 64)
if nm:
defs[h4] = (nm, kind, s32(mn), s32(mx))
lay = img.read(t['desc'] + 0x88, t['ncol'] * 16)
cols = []
for i in range(t['ncol']):
if (i + 1) * 16 > len(lay):
break
bit, h4, width, flags = struct.unpack_from('<IIII', lay, i * 16)
nm, kind, mn, mx = defs.get(h4, ('?%08x' % h4, 0, 0, 0))
cols.append({'name': nm, 'bit': bit, 'width': width, 'kind': kind,
'min': mn, 'max': mx, 'flags': flags})
cols.sort(key=lambda c: c['bit'])
# inline vs offset strings, decided by whether the declared width fits
for i, c in enumerate(cols):
nxt = cols[i + 1]['bit'] if i + 1 < len(cols) else t['rowsize'] * 8
if c['kind'] == KIND_STRING:
if c['bit'] + c['width'] <= nxt:
c['storage'] = 'inline-string'
else:
c['storage'] = 'offset-unresolved'
c['width'] = 32
else:
c['storage'] = 'int'
return cols
def read_rows(img, t, cols, limit=None):
n = t['rowcount'] or 0
if limit is not None:
n = min(n, limit)
rsz = t['rowsize']
if n == 0 or rsz == 0 or not t['rowblock']:
return []
blob = img.read(t['rowblock'], n * rsz)
if len(blob) < n * rsz:
raise RuntimeError('%s: row block short: got %d of %d bytes'
% (t['name'], len(blob), n * rsz))
out = []
for i in range(n):
row = blob[i * rsz:(i + 1) * rsz]
raw = int.from_bytes(row, 'little')
rec = {}
for c in cols:
if c['storage'] == 'inline-string':
lo = c['bit'] // 8
s = row[lo:lo + c['width'] // 8]
j = s.find(b'\x00')
rec[c['name']] = (s if j < 0 else s[:j]).decode('latin1')
else:
v = (raw >> c['bit']) & ((1 << c['width']) - 1)
rec[c['name']] = v + c['min'] if c['kind'] == KIND_INT else v
out.append(rec)
return out
def block_size(img, t):
h = img.read((t['rowblock'] or 0) - 0x10, 16)
if len(h) < 16:
return None, None
return struct.unpack_from('<I', h, 0)[0], struct.unpack_from('<Q', h, 8)[0]
# --------------------------------------------------------------------------- #
# checks -- every dump must be justified against something independent
# --------------------------------------------------------------------------- #
CHECKS = []
def check(msg, cond):
CHECKS.append((msg, bool(cond)))
print(" [%s] %s" % ("PASS" if cond else "FAIL", msg))
return bool(cond)
def run_checks(img, tables):
print("sanity checks (independent ground truth):")
ok = True
# (1) the catalogue must not contain column names masquerading as tables
ok &= check("catalogue holds no known column name as a table "
"(acceleration/carddbid/assetid absent)",
not ({'acceleration', 'carddbid', 'assetid'} & set(tables)))
# (2) each table's columns must match its own name
for tn, must in (('nations', 'nationname'), ('leagues', 'leaguename'),
('teams', 'teamname'), ('players', 'acceleration'),
('managercards', 'carddbid'),
('teamplayerlinks', 'jerseynumber')):
t = tables.get(tn)
names = {c['name'] for c in columns(img, t)} if t else set()
ok &= check("table %-16s contains column %-14s" % (tn, must), must in names)
# (3) THE anchor: playerid 20801 is Cristiano Ronaldo, FIFA 17 -- 94 rated,
# a left winger, Portuguese, 185 cm, at Real Madrid.
t = tables['players']
cols = columns(img, t)
pl = {r['playerid']: r for r in read_rows(img, t, cols)}
r = pl.get(20801)
ok &= check("players has 20801", r is not None)
if r:
ok &= check(" 20801 overallrating == 94 (got %s)" % r['overallrating'],
r['overallrating'] == 94)
ok &= check(" 20801 preferredposition1 == 27 (LW) (got %s)"
% r['preferredposition1'], r['preferredposition1'] == 27)
ok &= check(" 20801 nationality == 38 (Portugal) (got %s)" % r['nationality'],
r['nationality'] == 38)
ok &= check(" 20801 height == 185 cm (got %s)" % r['height'], r['height'] == 185)
ok &= check(" 20801 preferredfoot == 1 (right) (got %s)" % r['preferredfoot'],
r['preferredfoot'] == 1)
# nation 38 must literally spell Portugal, from a different table
nt = tables['nations']
nat = {x['nationid']: x for x in read_rows(img, nt, columns(img, nt))}
ok &= check("nations[38].nationname == 'Portugal' (got %r)"
% (nat.get(38, {}).get('nationname')),
nat.get(38, {}).get('nationname') == 'Portugal')
# and teamplayerlinks must put him at Real Madrid
tp = tables['teamplayerlinks']
links = [x for x in read_rows(img, tp, columns(img, tp)) if x['playerid'] == 20801]
tt = tables['teams']
teams = {x['teamid']: x for x in read_rows(img, tt, columns(img, tt))}
names = sorted({teams.get(l['teamid'], {}).get('teamname') for l in links})
ok &= check("teamplayerlinks[20801] includes Real Madrid (got %s)" % (names,),
'Real Madrid' in names)
# (4) managercards' id space, measured independently in a previous session
# (docs/managercards_ids.txt: 416 ids from 1000001 upward)
mt = tables['managercards']
mc = read_rows(img, mt, columns(img, mt))
ids = sorted(x['carddbid'] for x in mc)
ok &= check("managercards row 0 carddbid == 1000001 (got %s)" % (ids[0] if ids else None),
ids and ids[0] == 1000001)
ok &= check("managercards ids all in [1000001, 1002000] (got %s..%s, n=%d)"
% (ids[0] if ids else None, ids[-1] if ids else None, len(ids)),
ids and 1000001 <= ids[0] and ids[-1] <= 1002000)
ok &= check("managercards assetid == carddbid on every row",
all(x['assetid'] == x['carddbid'] for x in mc))
# (5) every table's row block must carry the block marker, and every byte of
# rowcount*rowsize must actually be readable. (The u32 at block-0x10 is
# a byte count only for the heap-resident tables; for the ~17 small
# tables that live in the 0x07xxxxxx pool it is something else, so it is
# reported but not required.)
unmarked, short, oddsize = [], [], []
for n, t in tables.items():
if not t['rowcount'] or not t['rowblock']:
continue
need = t['rowcount'] * t['rowsize']
sz, mark = block_size(img, t)
if mark != BLOCK_MARK:
unmarked.append(n)
if len(img.read(t['rowblock'], need)) < need:
short.append(n)
if sz is None or sz < need:
oddsize.append(n)
ok &= check("every non-empty table's row block carries the 0x2c020e60 marker "
"(%d without)" % len(unmarked), not unmarked)
ok &= check("every non-empty table's rowcount*rowsize bytes are readable "
"(%d short)" % len(short), not short)
print(" [note] %d small pool-resident tables have a block-size field that is "
"not a byte count: %s" % (len(oddsize), ", ".join(sorted(oddsize))))
return ok
# --------------------------------------------------------------------------- #
def dump_table(img, t, outdir, limit=None):
cols = columns(img, t)
rows = read_rows(img, t, cols, limit)
sz, mark = block_size(img, t)
doc = {
'table': t['name'],
'source': 'FIFA17.exe resident database (tools/db_dump.py)',
'rowcount': t['rowcount'],
'rowsize_bytes': t['rowsize'],
'rows_emitted': len(rows),
'rowblock': '0x%x' % (t['rowblock'] or 0),
'rowblock_bytes': sz,
'descriptor': '0x%x' % t['desc'],
'schema': [{'name': c['name'], 'bit': c['bit'], 'width': c['width'],
'kind': {1: 'string', 2: 'int', 4: 'date'}.get(c['kind'], 'unknown'),
'min': c['min'], 'max': c['max'], 'storage': c['storage']}
for c in cols],
'rows': rows,
}
os.makedirs(outdir, exist_ok=True)
p = os.path.join(outdir, t['name'] + '.json')
with open(p, 'w') as fh:
json.dump(doc, fh, ensure_ascii=False, separators=(',', ':'))
return p, len(rows), os.path.getsize(p)
def main():
ap = argparse.ArgumentParser(description=__doc__.split('\n')[0])
ap.add_argument('--pid', type=int)
ap.add_argument('--mem', help='work from a snapshot directory instead of a live game')
ap.add_argument('--save-mem', help='snapshot the process to this directory and exit')
ap.add_argument('--list', action='store_true')
ap.add_argument('--schema', action='append', default=[])
ap.add_argument('--dump', nargs='*')
ap.add_argument('--dump-all', action='store_true')
ap.add_argument('--check', action='store_true')
ap.add_argument('--limit', type=int)
ap.add_argument('-o', '--out', default='../data/tables')
a = ap.parse_args()
if a.mem:
img = Image(memdir=a.mem)
sys.stderr.write("snapshot %s: %d regions\n" % (a.mem, len(img.regions)))
else:
pid = a.pid or find_pid()
img = Image(pid=pid)
sys.stderr.write("FIFA17.exe pid %d: %d anonymous regions\n"
% (pid, len(img.regions)))
if a.save_mem:
n = img.save(a.save_mem)
sys.stderr.write("saved %.1f MB to %s\n" % (n / 1048576.0, a.save_mem))
return 0
tables = catalogue(img)
sys.stderr.write("catalogue: %d tables\n" % len(tables))
if a.list:
for n in sorted(tables):
t = tables[n]
cols = columns(img, t)
print("%-34s rows=%-7s rowsize=%-5d cols=%-4d | %s"
% (n, t['rowcount'], t['rowsize'], t['ncol'],
", ".join(c['name'] for c in cols[:6])))
return 0
for n in a.schema:
t = tables.get(n)
if not t:
print("%s: not in the catalogue" % n)
continue
print("=== %s rows=%s rowsize=%d bytes cols=%d rowblock=0x%x"
% (n, t['rowcount'], t['rowsize'], t['ncol'], t['rowblock'] or 0))
for c in columns(img, t):
print(" bit %4d w %-3d %-18s %-9s min %-8s max %s"
% (c['bit'], c['width'], c['name'], c['storage'], c['min'], c['max']))
rc = 0
if a.check:
rc = 0 if run_checks(img, tables) else 1
want = None
if a.dump_all:
want = sorted(tables)
elif a.dump is not None:
want = a.dump or sorted(tables)
if want:
total = 0
for n in want:
t = tables.get(n)
if not t:
sys.stderr.write(" %s: not in the catalogue\n" % n)
continue
try:
p, nr, nb = dump_table(img, t, a.out, a.limit)
except Exception as e: # noqa: BLE001
sys.stderr.write(" %-30s FAILED: %s\n" % (n, e))
rc = 1
continue
total += nb
sys.stderr.write(" %-34s %7d rows %9.1f KB %s\n"
% (n, nr, nb / 1024.0, p))
sys.stderr.write("wrote %.1f MB into %s\n" % (total / 1048576.0, a.out))
return rc
if __name__ == '__main__':
sys.exit(main())
Reference in New Issue View Git Blame Copy Permalink