OpenFUT/fifa17-recon/tools/db_dump.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Dump FIFA 17's LOADED relational database (schema + every row) out of a live
FIFA17.exe, or out of a saved memory image.

READ-ONLY.  /proc/PID/mem is opened 'rb' and only ever seek()/read().  There is
no write path in this file.

WHY THIS EXISTS
---------------
dbdata.dll on disk is packed (see dbdata_extract.py's docstring), so the game's
tables cannot be read off disk.  They ARE fully resident in the running process,
in a self-describing form: the DB carries its own catalogue, its own column
names, and its own bit-level row layout.  This tool walks that catalogue and
decodes the rows.

dbschema_probe.py got as far as the catalogue but had two defects, both fixed
here and both worth writing down because they are easy to re-introduce:

  1. THE NAME IS AT anchor+0x08, NOT AT anchor-0x20.  Every catalogue record --
     table records and column records alike -- carries the constant qword
     0x07c20760 (SHARED_ANCHOR).  Reading the name from the start of the
     32-byte-aligned slot picks up the PREVIOUS record's name, which silently
     mislabels every table by one slot: what the old tool called "gkcoachcards"
     is really `players`, what it called "teams" is really `teamplayerlinks`,
     and so on.  The mislabelling is invisible because both names are real.
     The check that catches it: a table's column set must match its name
     (`nations` must contain nationname, `players` must contain acceleration).

  2. --list MIXED COLUMNS INTO THE TABLE LIST.  Column records and table
     records share the same anchor, so an anchor scan alone yields both.  The
     discriminator used here is the TABLE VTABLE: a record is a table if and
     only if its descriptor's first qword is 0x07c31028.  That takes 4126
     anchor hits down to exactly 149 tables, with zero column names among them.

THE STRUCTURES  (resolved live 2026-08-04, pid 52703, game in the FUT club UI)
------------------------------------------------------------------------------
CATALOGUE RECORD (table).  Found by scanning for SHARED_ANCHOR; the record
starts 0x18 before it:
    +0x00 void*  descriptor
    +0x08 u32    table name hash4   (repeated at descriptor+0x40)
    +0x0c u32    column count
    +0x10 void*  column-definition array  ("p2")
    +0x18 u64    SHARED_ANCHOR 0x07c20760
    +0x20 char*  table name
(stride 0x30)

COLUMN-DEFINITION RECORD (0x30 bytes, p2[i]) -- the human-readable half:
    +0x00 u32    kind: 1 = string, 2 = integer, 4 = date
    +0x04 u32    name hash4
    +0x08 u32    min value (as u32; may be negative, e.g. -1 for a position)
    +0x0c u32    max value
    +0x20 u64    SHARED_ANCHOR
    +0x28 char*  column name

TABLE DESCRIPTOR (at the record's descriptor pointer):
    +0x00 u64    0x07c31028              (the table vtable -- the discriminator)
    +0x30 void*  ROW BLOCK
    +0x40 u32    table name hash4        (self-identification)
    +0x44 u32    row size in BYTES
    +0x48 u32    max bit index           (== rowsize*8 - 1)
    +0x7c u32    ROW COUNT
    +0x82 u16    column count
    +0x88 ...    column layout array, `ncol` records of 16 bytes:
                     u32 bit_offset, u32 name hash4, u32 bit_width, u32 flags
                 (sorted by hash4, so it needs sorting by bit_offset to read)

ROW BLOCK (pointed at by descriptor+0x30) is preceded by a 16-byte header:
    -0x10 u32    byte size of the block
    -0x08 u64    0x2c020e60             (a second constant, a useful check)
Rows are `rowsize` bytes, densely packed, no gaps.

FIELD DECODING
--------------
    value = (int.from_bytes(row, 'little') >> bit_offset) & ((1 << width) - 1)
Integer columns then add `min`, so a column declared min=-1 max=32 stores 0..33
and reads back -1..32.  This was confirmed on managercards.carddbid (bit 64,
width 24) reading 1000001 on row 0 -- the exact value docs/managercards_ids.txt
recorded from a completely independent live sweep.

STRINGS come in two forms and the tool decides per column:
  * INLINE  when bit_offset + width <= the next column's bit_offset: the field
    is a fixed-size NUL-terminated char array inside the row.  This is how
    nations.nationname, leagues.leaguename and teams.teamname are stored, and
    all three decode to real names.
  * OFFSET  otherwise: the field is a 32-bit offset into a string pool that is
    NOT resident as a flat blob (searched for; not found).  Those columns are
    emitted as raw integers and flagged `"storage": "offset-unresolved"` in the
    schema block.  managercards.firstname/lastname and playernames.name are of
    this kind.  Player names are already available from data/roster.json via
    dbdata_extract.py, so nothing depends on resolving them.

USAGE
-----
    ./db_dump.py --list                      # every table, row count, columns
    ./db_dump.py --schema players            # one table's column layout
    ./db_dump.py --check                     # run the anchor checks, exit 1 on fail
    ./db_dump.py --dump players teams -o DIR # dump named tables as JSON
    ./db_dump.py --dump-all -o DIR           # dump all 149
    ./db_dump.py --save-mem DIR              # snapshot the process (do this FIRST;
                                             #   live memory is perishable)
    ./db_dump.py --mem DIR ...               # work from a snapshot, no live game

Requires ptrace access to FIFA17.exe (ptrace_scope=1 + same uid is enough) when
reading live; --mem needs nothing but the snapshot directory.
"""

import argparse
import bisect
import json
import mmap
import os
import re
import struct
import sys

SHARED_ANCHOR = 0x07C20760      # on every catalogue record, at record+0x18/+0x20
TABLE_VTABLE  = 0x07C31028      # descriptor[0] iff the record describes a table
BLOCK_MARK    = 0x2C020E60      # rowblock[-0x08]
IDENT = re.compile(r'[A-Za-z][A-Za-z0-9_]*\Z')

# Anonymous mappings above this are the host libc arenas; the game DB is below.
MAX_VA = 0x200000000

KIND_STRING, KIND_INT, KIND_DATE = 1, 2, 4


# --------------------------------------------------------------------------- #
# memory access
# --------------------------------------------------------------------------- #

def find_pid():
    for d in os.listdir('/proc'):
        if not d.isdigit():
            continue
        try:
            if open('/proc/%s/comm' % d).read().strip() == 'FIFA17.exe':
                return int(d)
        except OSError:
            pass
    raise SystemExit("FIFA17.exe is not running (use --mem DIR to work offline)")


def anon_regions(pid):
    out = []
    for line in open('/proc/%d/maps' % pid):
        p = line.split()
        lo, hi = (int(x, 16) for x in p[0].split('-'))
        perms = p[1]
        path = p[5] if len(p) > 5 else ''
        if 'r' not in perms or path or lo >= MAX_VA:
            continue
        out.append((lo, hi, perms))
    return out


class Image(object):
    """Uniform read-only view over either a live process or a saved snapshot."""

    def __init__(self, pid=None, memdir=None):
        self.regions = []           # list of dicts lo/hi
        self._buf = {}
        if memdir:
            idx = json.load(open(os.path.join(memdir, 'index.json')))
            idx.sort(key=lambda r: r['lo'])
            self.regions = idx
            self.memdir = memdir
            self.live = None
        else:
            self.live = open('/proc/%d/mem' % pid, 'rb', 0)
            self.memdir = None
            for lo, hi, perms in anon_regions(pid):
                self.regions.append({'lo': lo, 'hi': hi, 'perms': perms})
        self._los = [r['lo'] for r in self.regions]

    # -- region buffers ----------------------------------------------------- #
    def buf(self, i):
        if i not in self._buf:
            r = self.regions[i]
            if self.memdir:
                f = open(os.path.join(self.memdir, r['file']), 'rb')
                self._buf[i] = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
            else:
                self.live.seek(r['lo'])
                self._buf[i] = self.live.read(r['hi'] - r['lo'])
        return self._buf[i]

    def _find(self, a):
        i = bisect.bisect_right(self._los, a) - 1
        if i >= 0 and self.regions[i]['lo'] <= a < self.regions[i]['hi']:
            return i
        return -1

    def read(self, a, n):
        if a is None or a <= 0:
            return b''
        i = self._find(a)
        if i < 0:
            return b''
        o = a - self.regions[i]['lo']
        return bytes(self.buf(i)[o:o + n])

    def u32(self, a):
        b = self.read(a, 4)
        return struct.unpack('<I', b)[0] if len(b) == 4 else None

    def u64(self, a):
        b = self.read(a, 8)
        return struct.unpack('<Q', b)[0] if len(b) == 8 else None

    def cstr(self, a, maxlen=96):
        b = self.read(a, maxlen)
        j = b.find(b'\x00')
        if j <= 0:
            return None
        s = b[:j]
        return s.decode('latin1') if re.fullmatch(rb'[ -~]+', s) else None

    def save(self, outdir):
        """Snapshot every region to disk (do this first: live data is perishable)."""
        os.makedirs(outdir, exist_ok=True)
        idx = []
        for i, r in enumerate(self.regions):
            d = self.buf(i)
            fn = '%012x.bin' % r['lo']
            with open(os.path.join(outdir, fn), 'wb') as fh:
                fh.write(d)
            idx.append({'lo': r['lo'], 'hi': r['lo'] + len(d),
                        'perms': r.get('perms', 'rw-p'), 'file': fn})
        json.dump(idx, open(os.path.join(outdir, 'index.json'), 'w'), indent=1)
        return sum(r['hi'] - r['lo'] for r in idx)


# --------------------------------------------------------------------------- #
# catalogue walk
# --------------------------------------------------------------------------- #

def s32(v):
    return v - (1 << 32) if v is not None and v >= (1 << 31) else v


def anchor_hits(img):
    """Addresses of every SHARED_ANCHOR qword, 8-byte aligned."""
    pat = struct.pack('<Q', SHARED_ANCHOR)
    out = []
    for i, r in enumerate(img.regions):
        b = img.buf(i)
        st = 0
        while True:
            j = b.find(pat, st)
            if j < 0:
                break
            st = j + 1
            if j % 8 == 0:
                out.append(r['lo'] + j)
    return out


def catalogue(img):
    """name -> table dict.  Tables only: the descriptor vtable is the filter."""
    tables = {}
    for anch in anchor_hits(img):
        name = img.cstr(img.u64(anch + 8) or 0, 64)
        if not name or not IDENT.match(name):
            continue
        desc = img.u64(anch - 0x18)
        head = img.read(desc or 0, 0x50)
        if len(head) < 0x50 or struct.unpack_from('<Q', head, 0)[0] != TABLE_VTABLE:
            continue
        h4, ncol = struct.unpack_from('<II', img.read(anch - 0x10, 8), 0)
        dh4, rowsz, maxbit, _ = struct.unpack_from('<IIII', head, 0x40)
        if dh4 != h4:                      # descriptor must self-identify
            continue
        tables[name] = {
            'name': name, 'anchor': anch, 'desc': desc, 'hash4': h4,
            'ncol': ncol, 'p2': img.u64(anch - 8),
            'rowsize': rowsz, 'maxbit': maxbit,
            'rowcount': img.u32(desc + 0x7c),
            'rowblock': img.u64(desc + 0x30),
        }
    return tables


def columns(img, t):
    """Full column list, sorted by bit offset, with names/kinds/ranges."""
    defs = {}
    for i in range(t['ncol']):
        b = img.read(t['p2'] + i * 0x30, 0x30)
        if len(b) < 0x30:
            break
        kind, h4 = struct.unpack_from('<II', b, 0)
        mn, mx = struct.unpack_from('<II', b, 8)
        nm = img.cstr(struct.unpack_from('<Q', b, 0x28)[0], 64)
        if nm:
            defs[h4] = (nm, kind, s32(mn), s32(mx))
    lay = img.read(t['desc'] + 0x88, t['ncol'] * 16)
    cols = []
    for i in range(t['ncol']):
        if (i + 1) * 16 > len(lay):
            break
        bit, h4, width, flags = struct.unpack_from('<IIII', lay, i * 16)
        nm, kind, mn, mx = defs.get(h4, ('?%08x' % h4, 0, 0, 0))
        cols.append({'name': nm, 'bit': bit, 'width': width, 'kind': kind,
                     'min': mn, 'max': mx, 'flags': flags})
    cols.sort(key=lambda c: c['bit'])
    # inline vs offset strings, decided by whether the declared width fits
    for i, c in enumerate(cols):
        nxt = cols[i + 1]['bit'] if i + 1 < len(cols) else t['rowsize'] * 8
        if c['kind'] == KIND_STRING:
            if c['bit'] + c['width'] <= nxt:
                c['storage'] = 'inline-string'
            else:
                c['storage'] = 'offset-unresolved'
                c['width'] = 32
        else:
            c['storage'] = 'int'
    return cols


def read_rows(img, t, cols, limit=None):
    n = t['rowcount'] or 0
    if limit is not None:
        n = min(n, limit)
    rsz = t['rowsize']
    if n == 0 or rsz == 0 or not t['rowblock']:
        return []
    blob = img.read(t['rowblock'], n * rsz)
    if len(blob) < n * rsz:
        raise RuntimeError('%s: row block short: got %d of %d bytes'
                           % (t['name'], len(blob), n * rsz))
    out = []
    for i in range(n):
        row = blob[i * rsz:(i + 1) * rsz]
        raw = int.from_bytes(row, 'little')
        rec = {}
        for c in cols:
            if c['storage'] == 'inline-string':
                lo = c['bit'] // 8
                s = row[lo:lo + c['width'] // 8]
                j = s.find(b'\x00')
                rec[c['name']] = (s if j < 0 else s[:j]).decode('latin1')
            else:
                v = (raw >> c['bit']) & ((1 << c['width']) - 1)
                rec[c['name']] = v + c['min'] if c['kind'] == KIND_INT else v
        out.append(rec)
    return out


def block_size(img, t):
    h = img.read((t['rowblock'] or 0) - 0x10, 16)
    if len(h) < 16:
        return None, None
    return struct.unpack_from('<I', h, 0)[0], struct.unpack_from('<Q', h, 8)[0]


# --------------------------------------------------------------------------- #
# checks -- every dump must be justified against something independent
# --------------------------------------------------------------------------- #

CHECKS = []


def check(msg, cond):
    CHECKS.append((msg, bool(cond)))
    print("  [%s] %s" % ("PASS" if cond else "FAIL", msg))
    return bool(cond)


def run_checks(img, tables):
    print("sanity checks (independent ground truth):")
    ok = True

    # (1) the catalogue must not contain column names masquerading as tables
    ok &= check("catalogue holds no known column name as a table "
                "(acceleration/carddbid/assetid absent)",
                not ({'acceleration', 'carddbid', 'assetid'} & set(tables)))

    # (2) each table's columns must match its own name
    for tn, must in (('nations', 'nationname'), ('leagues', 'leaguename'),
                     ('teams', 'teamname'), ('players', 'acceleration'),
                     ('managercards', 'carddbid'),
                     ('teamplayerlinks', 'jerseynumber')):
        t = tables.get(tn)
        names = {c['name'] for c in columns(img, t)} if t else set()
        ok &= check("table %-16s contains column %-14s" % (tn, must), must in names)

    # (3) THE anchor: playerid 20801 is Cristiano Ronaldo, FIFA 17 -- 94 rated,
    #     a left winger, Portuguese, 185 cm, at Real Madrid.
    t = tables['players']
    cols = columns(img, t)
    pl = {r['playerid']: r for r in read_rows(img, t, cols)}
    r = pl.get(20801)
    ok &= check("players has 20801", r is not None)
    if r:
        ok &= check("  20801 overallrating == 94 (got %s)" % r['overallrating'],
                    r['overallrating'] == 94)
        ok &= check("  20801 preferredposition1 == 27 (LW) (got %s)"
                    % r['preferredposition1'], r['preferredposition1'] == 27)
        ok &= check("  20801 nationality == 38 (Portugal) (got %s)" % r['nationality'],
                    r['nationality'] == 38)
        ok &= check("  20801 height == 185 cm (got %s)" % r['height'], r['height'] == 185)
        ok &= check("  20801 preferredfoot == 1 (right) (got %s)" % r['preferredfoot'],
                    r['preferredfoot'] == 1)

    # nation 38 must literally spell Portugal, from a different table
    nt = tables['nations']
    nat = {x['nationid']: x for x in read_rows(img, nt, columns(img, nt))}
    ok &= check("nations[38].nationname == 'Portugal' (got %r)"
                % (nat.get(38, {}).get('nationname')),
                nat.get(38, {}).get('nationname') == 'Portugal')

    # and teamplayerlinks must put him at Real Madrid
    tp = tables['teamplayerlinks']
    links = [x for x in read_rows(img, tp, columns(img, tp)) if x['playerid'] == 20801]
    tt = tables['teams']
    teams = {x['teamid']: x for x in read_rows(img, tt, columns(img, tt))}
    names = sorted({teams.get(l['teamid'], {}).get('teamname') for l in links})
    ok &= check("teamplayerlinks[20801] includes Real Madrid (got %s)" % (names,),
                'Real Madrid' in names)

    # (4) managercards' id space, measured independently in a previous session
    #     (docs/managercards_ids.txt: 416 ids from 1000001 upward)
    mt = tables['managercards']
    mc = read_rows(img, mt, columns(img, mt))
    ids = sorted(x['carddbid'] for x in mc)
    ok &= check("managercards row 0 carddbid == 1000001 (got %s)" % (ids[0] if ids else None),
                ids and ids[0] == 1000001)
    ok &= check("managercards ids all in [1000001, 1002000] (got %s..%s, n=%d)"
                % (ids[0] if ids else None, ids[-1] if ids else None, len(ids)),
                ids and 1000001 <= ids[0] and ids[-1] <= 1002000)
    ok &= check("managercards assetid == carddbid on every row",
                all(x['assetid'] == x['carddbid'] for x in mc))

    # (5) every table's row block must carry the block marker, and every byte of
    #     rowcount*rowsize must actually be readable.  (The u32 at block-0x10 is
    #     a byte count only for the heap-resident tables; for the ~17 small
    #     tables that live in the 0x07xxxxxx pool it is something else, so it is
    #     reported but not required.)
    unmarked, short, oddsize = [], [], []
    for n, t in tables.items():
        if not t['rowcount'] or not t['rowblock']:
            continue
        need = t['rowcount'] * t['rowsize']
        sz, mark = block_size(img, t)
        if mark != BLOCK_MARK:
            unmarked.append(n)
        if len(img.read(t['rowblock'], need)) < need:
            short.append(n)
        if sz is None or sz < need:
            oddsize.append(n)
    ok &= check("every non-empty table's row block carries the 0x2c020e60 marker "
                "(%d without)" % len(unmarked), not unmarked)
    ok &= check("every non-empty table's rowcount*rowsize bytes are readable "
                "(%d short)" % len(short), not short)
    print("  [note] %d small pool-resident tables have a block-size field that is "
          "not a byte count: %s" % (len(oddsize), ", ".join(sorted(oddsize))))

    return ok


# --------------------------------------------------------------------------- #

def dump_table(img, t, outdir, limit=None):
    cols = columns(img, t)
    rows = read_rows(img, t, cols, limit)
    sz, mark = block_size(img, t)
    doc = {
        'table': t['name'],
        'source': 'FIFA17.exe resident database (tools/db_dump.py)',
        'rowcount': t['rowcount'],
        'rowsize_bytes': t['rowsize'],
        'rows_emitted': len(rows),
        'rowblock': '0x%x' % (t['rowblock'] or 0),
        'rowblock_bytes': sz,
        'descriptor': '0x%x' % t['desc'],
        'schema': [{'name': c['name'], 'bit': c['bit'], 'width': c['width'],
                    'kind': {1: 'string', 2: 'int', 4: 'date'}.get(c['kind'], 'unknown'),
                    'min': c['min'], 'max': c['max'], 'storage': c['storage']}
                   for c in cols],
        'rows': rows,
    }
    os.makedirs(outdir, exist_ok=True)
    p = os.path.join(outdir, t['name'] + '.json')
    with open(p, 'w') as fh:
        json.dump(doc, fh, ensure_ascii=False, separators=(',', ':'))
    return p, len(rows), os.path.getsize(p)


def main():
    ap = argparse.ArgumentParser(description=__doc__.split('\n')[0])
    ap.add_argument('--pid', type=int)
    ap.add_argument('--mem', help='work from a snapshot directory instead of a live game')
    ap.add_argument('--save-mem', help='snapshot the process to this directory and exit')
    ap.add_argument('--list', action='store_true')
    ap.add_argument('--schema', action='append', default=[])
    ap.add_argument('--dump', nargs='*')
    ap.add_argument('--dump-all', action='store_true')
    ap.add_argument('--check', action='store_true')
    ap.add_argument('--limit', type=int)
    ap.add_argument('-o', '--out', default='../data/tables')
    a = ap.parse_args()

    if a.mem:
        img = Image(memdir=a.mem)
        sys.stderr.write("snapshot %s: %d regions\n" % (a.mem, len(img.regions)))
    else:
        pid = a.pid or find_pid()
        img = Image(pid=pid)
        sys.stderr.write("FIFA17.exe pid %d: %d anonymous regions\n"
                         % (pid, len(img.regions)))
        if a.save_mem:
            n = img.save(a.save_mem)
            sys.stderr.write("saved %.1f MB to %s\n" % (n / 1048576.0, a.save_mem))
            return 0

    tables = catalogue(img)
    sys.stderr.write("catalogue: %d tables\n" % len(tables))

    if a.list:
        for n in sorted(tables):
            t = tables[n]
            cols = columns(img, t)
            print("%-34s rows=%-7s rowsize=%-5d cols=%-4d | %s"
                  % (n, t['rowcount'], t['rowsize'], t['ncol'],
                     ", ".join(c['name'] for c in cols[:6])))
        return 0

    for n in a.schema:
        t = tables.get(n)
        if not t:
            print("%s: not in the catalogue" % n)
            continue
        print("=== %s  rows=%s  rowsize=%d bytes  cols=%d  rowblock=0x%x"
              % (n, t['rowcount'], t['rowsize'], t['ncol'], t['rowblock'] or 0))
        for c in columns(img, t):
            print("   bit %4d  w %-3d  %-18s %-9s min %-8s max %s"
                  % (c['bit'], c['width'], c['name'], c['storage'], c['min'], c['max']))

    rc = 0
    if a.check:
        rc = 0 if run_checks(img, tables) else 1

    want = None
    if a.dump_all:
        want = sorted(tables)
    elif a.dump is not None:
        want = a.dump or sorted(tables)
    if want:
        total = 0
        for n in want:
            t = tables.get(n)
            if not t:
                sys.stderr.write("  %s: not in the catalogue\n" % n)
                continue
            try:
                p, nr, nb = dump_table(img, t, a.out, a.limit)
            except Exception as e:                      # noqa: BLE001
                sys.stderr.write("  %-30s FAILED: %s\n" % (n, e))
                rc = 1
                continue
            total += nb
            sys.stderr.write("  %-34s %7d rows  %9.1f KB  %s\n"
                             % (n, nr, nb / 1024.0, p))
        sys.stderr.write("wrote %.1f MB into %s\n" % (total / 1048576.0, a.out))
    return rc


if __name__ == '__main__':
    sys.exit(main())