Skip to content

dat: generate native readers for typed HSD archive loading - #3658

Merged
itsgrimetime merged 5 commits into
doldecomp:masterfrom
itsgrimetime:codex/dat-generated-readers
Oct 11, 2026
Merged

itsgrimetime merged 5 commits into
doldecomp:masterfrom
itsgrimetime:codex/dat-generated-readers

Conversation

@itsgrimetime

@itsgrimetime itsgrimetime commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Generate C readers for the known 32-bit big-endian layout and dispatch HSD_ArchiveGetPublicAs(type, archive, name) through a generated type table. Rust compiles annotations into field readers and branching code; native C supplies the host layout.

Readers recurse directly with ordinary archive-owned allocations and identity/reference caches for cycles and shared arrays. Remove the private arena, task/copy queues and runtime annotation/operator tables. Keep binding scopes in local arrays or constants, and tracing/readback/verifiers in test support. Fix first-field alias moves that could copy an enclosing record into undersized pointer storage, with independent fixtures for both traversal orders.

Keep existing HSD_Archive declarations and allocations. A TARGET_PC field owns native reader state; the remaining src changes provide typed dispatch and normal/preload cleanup. Remove DAT_NATIVE_READERS; MELEE_DAT_READERS only controls building the adapter locally.

Untyped/script data stays serialized; specialized lbArchiveRelocate consumers still need native integration. Caller pointers cannot follow moves from subsequently loaded roots. #3653 remains separate until this lands.

Validation: 58 Rust tests and Clippy; six native tests including parity/readback across 7,050 archives; ASan/UBSan HSD dispatch and typed access for 27 fighters. Reader-enabled game call sites compile, and the GALE01 MWCC rebuild matches its expected hash. 605/605 samples match. Formatting and EditorConfig pass. All eight CI build/check jobs pass on 724d002. Generated outputs are uncommitted.

@itsgrimetime itsgrimetime added ai-assisted Utilizes a LLM to do the heavy lifting portability Improves non-matching builds labels Oct 10, 2026

@ribbanya ribbanya left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm missing something but why does it need any of this stuff still? The task is to read an archive (meaning understand the header) then using some kind of type dispatch expand it to its in-memory form and return it. So GetPublicAs specifies the type and then the only job of the C code here is to read a symbol back as that type.

So I don't see why, for example, a growable vector or a private arena would be necessary.

@ribbanya

ribbanya commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

The intended concept is very simple: Rust generates "dumb" C code that simply reads a 32-bit BE archive and translates it at runtime to the same struct laid out appropriately for the running platform. So each generated set of C files only understands one architecture and polymorphism is abstracted through simple tables and callbacks (a dispatched reader). It doesn't perform any complex reasoning, it just takes the type and symbol name requested from the HSD_ArchiveGetPublicAs and translates it.

Again, maybe I am missing something that makes this unviable but I can't think of anything. There should be enough information in the dwarf for the Rust code to generate this kind of C code.

@ribbanya

Copy link
Copy Markdown
Collaborator

So basically what we need is HSD_ArchiveGetPublicAs to be some kind of switch statement or indexed callback table based on the type argument. Then for each type, a generated translation unit is responsible for only reading that type from the serialized form to memory. Then the dispatcher returns the expanded type.

@ribbanya

Copy link
Copy Markdown
Collaborator

We can create a new #if branch definition for HSD_ArchiveGetPublicAs to facilitate this, under TARGET_PC maybe.

@itsgrimetime itsgrimetime changed the title dat: generate native type readers in Rust dat: generate native readers for typed HSD archive loading Oct 10, 2026

@ribbanya ribbanya left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is much better, thank you. Will need some tweaking for a while but it's a good foundation.

Comment thread src/sysdolphin/baselib/archive.h Outdated
Comment thread src/sysdolphin/baselib/archive.h Outdated
Comment thread src/sysdolphin/baselib/archive.h Outdated

@ribbanya ribbanya left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said, needs more iteration, but this looks good for now. Thanks!

@itsgrimetime
itsgrimetime enabled auto-merge (squash) October 11, 2026 18:50
@itsgrimetime
itsgrimetime force-pushed the codex/dat-generated-readers branch from 7e5c71d to f8fecfd Compare October 11, 2026 18:58
@itsgrimetime
itsgrimetime merged commit b2b201b into doldecomp:master Oct 11, 2026
10 checks passed
YoshiGamer123 added a commit to ProxysMirrors/meleeweb that referenced this pull request Oct 14, 2026
…s, particle fixes

18 upstream commits (through af59f8c). web-single builds clean.

Conflicts + integration resolved keeping the port paths:
- archive.c: upstream gates the body under #ifndef TARGET_PC (native builds use
  the generated reader in tools/dat-cli). The wasm port loads archives through
  this file with its in-place endian swap, so kept the port include block
  (hsd_endian/archive_swap.h) and HSD_ArchiveParse on TARGET_PC; dropped the
  dangling EOF #endif.
- granime.c grAnime_801C6F50: kept the port TARGET_PC wasm-signature-safe
  dispatch switch; took upstream grAnimeCallback typed-union code for the #else
  (GameCube) branch.
- archive.h / lbarchive.c / lbdvd.c: upstream doldecomp#3658 moved archive loading onto a
  native DatArchive reader under TARGET_PC (native_data field, growing
  HSD_Archive to 0x48 and breaking ASSERT_SIZE 0x44; plus HSD_ArchiveReleaseNative
  / HSD_ArchiveGetPublicTyped, which are undefined for the port -- only in
  tools/dat-cli). The port uses its own endian-swap + DAT_ROOTS schema, so
  reverted those TARGET_PC additions: dropped native_data (restores 0x44),
  restored the DAT_ROOTS_ENABLED HSD_ArchiveGetPublicAs, removed the
  HSD_ArchiveReleaseNative calls in lbarchive.c and lbdvd.c.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01S9oD2L2sgETDKcnJp2UdYy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted Utilizes a LLM to do the heavy lifting portability Improves non-matching builds

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants