feat!: schema v2 — CanNode CPG, L3 dataflow, TS-prefixed graph vocabulary (1.0.0, graph schema 2.0.0)#78
Merged
Merged
Conversation
…icing (-a 3) Implements the dataflow half of #2: whole-program dependence graphs built in-process from the ts-morph AST, emitted as a schema-versioned program_graphs section of analysis.json, gated by -a 3 / --graphs. - src/dataflow/cfg.ts: exceptional statement-level CFG per callable (ENTRY/param/statement/EXIT nodes in source-span order; true/false, loop_back, switch_case, exception, await_resume, yield edge kinds; region-spliced try/catch/finally; synthetic loop-exit edge keeps EXIT the post-dominance root). - src/dataflow/dominance.ts: CHK iterative post-dominators + Ferrante–Ottenstein–Warren control dependence (CDG). - src/dataflow/defuse.ts: k-limited access paths (declaration-keyed bases: local/param/this/captured/module), copy-alias union-find MVP, forward reaching definitions, DDG extraction, capture-at-declaration for closures, EXIT-as-formal-out routing. - src/dataflow/summaries.ts: Tarjan SCC condensation of the provenance-merged call graph; bottom-up relational summaries (param→return, transitive global reads/writes) co-defined to a monotone fixpoint inside SCCs; persisted with dependency edges to graphs_summaries.json for later incrementality. - src/dataflow/sdg.ts: HRB stitching — CALL, PARAM_IN (args by position, globals as extra params), PARAM_OUT, and SUMMARY edges, all keyed by (signature, node_id) with no dangling endpoints. - src/dataflow/slice.ts: two-phase context-sensitive backward slicing as an SDG query. - CLI: -a 3, --graphs cfg,dfg,pdg,sdg, --graph-field-depth (strictly validated); -a 1/-a 2 are untouched (level 3 is fully flag-gated). - test/fixtures/dataflow-app + test/dataflow.test.ts: every contract gate with exact hand-computed expected sets (CFG reachability, CDG sets, loop-carried/shadowing/aliasing DDG, intraprocedural and interprocedural slices, SUMMARY for the a→b→c chain, global flow, mutual-recursion fixpoint, byte-identical determinism). Follow-ups staged in #2: taint models-as-data (PR E), Jelly points-to aliasing (PR F), CPG Neo4j projection (PR G), incremental re-analysis (PR H).
Implements the contract's parallel model on top of the sequential oracle: - Stage split: fact extraction (AST-bound, once per callable) is hoisted out of the summary fixpoint; the reaching-defs solve (defuse.solveDefUse) is now pure data and re-runs without touching the AST. CallableGraphData is the serializable per-callable projection that crosses the worker boundary. - Stages 1-4 (CFG, dominance, def-use facts, PDG) fan out per callable over a Bun worker pool, partitioned by file; each worker materializes its own whole-program ts-morph project (ASTs cannot be structured-cloned) and returns plain data. - The call-graph solve overlaps extraction: at -a 3, core.ts posts the extraction to the pool BEFORE the provider (tsc resolver + Jelly subprocess) runs on the main thread, and joins before summaries. - Stages 6-7 run as a Kahn-style ready-queue wavefront over the Tarjan SCC condensation DAG (per-SCC dependency counters; the SCC and its internal fixpoint are the atomic unit, one worker each). - Determinism: --jobs N output is byte-identical to --jobs 1 (span-ordered ids, collect-then-sort emission, sccFixpoint pure), enforced by a differential test. --jobs 1 is the sequential debug mode. - Failure discipline: a dying worker is retired and its queue never strands (a stranded queue previously let the process exit 0 without emitting output); extraction failure closes the pool so the wavefront degrades sequentially too — a warning, never wrong or missing output. - Compiled binary: the worker is a second bun build --compile entrypoint, embedded as /$bunfs/root/dataflow/worker.js; pool resolves the URL per runtime. Verified: dist/cants -a 3 -j 2 runs workers with byte-identical output. - Default is sequential: measurement (self-analysis: 36 files, 211 callables) shows per-worker project load dominates the parallelizable graph math at small/mid scale (2.5x slower at -j 14), so -j N is an explicit opt-in for large codebases. Also fixes a latent bug: TSCallable.path is absolute, so the summary cache's symbol_table[c.path] content-hash lookup always missed; now keyed by the project-relative file key.
Runs analyze() fresh at -a 1/2/3/4 on dataflow-app and asserts each level's symbol-table id / body-node key / edge key sets are a strict superset of the level below, plus a full-depth Neo4j projection vs analysis.json count-parity check. Supersedes the narrower L3->L4-only spot-check with an end-to-end gate across all four levels.
GraphNode carried only line/column, so spanOf() in schema/v2/dataflow.ts hardcoded bytes:[0,0] for every L3 entry/exit/param/statement node — module.source.slice(...) on any L3 body node returned "". Add start_offset/end_offset (UTF-16 char offsets, matching the L1 convention) to GraphNode, populate them at the single construction site (dataflow/extract.ts's emitNode, from the owning ts-morph node's getStart()/getEnd() — the whole callable for entry/exit, the param decl for param nodes), and have spanOf() use them.
Consumers (the CLDK SDK, tooling correlating multiple analysis.json
artifacts) need to know which analyzer produced a given output and at
what version, independent of schema_version. Add a V2Analyzer{name,
version} type and a required analyzer field on V2Application, populated
from the literal package name and the existing ANALYZER_VERSION constant
(src/utils/version.ts) — the same version stamped into the cache, not a
new hardcoded literal.
…json
ANALYZER_VERSION was "0.1.0" while package.json was "0.5.0", so the new
analyzer{name,version} manifest advertised a false version. Bump it to
match, and assert analyzer.version equals package.json's version (imported,
not hardcoded) so future drift fails the schema-v2 gate.
…discarded The v2 emitter only reads external_symbols/synthesized_callables and populates call_graph at level >= 2 (homeExternals/homeSynthesized in src/schema/v2/emit.ts), so running the full provider solve — including the heavier Jelly leg — at -a 1 computed a result that was thrown away. Gate provider.build to analysisLevel >= 2; levels 3/4 already require >= 2 for callee resolution, so this is safe.
…ecisions + neo4j lockstep) Strengthens the code comment at src/schema/v2/dataflow.ts to record that "reaching-defs" is a deliberate, sanctioned additive DDG prov token — a documented deviation from the canonical "ssa" tag — accepted across JSON, Neo4j (open string[] property, no enum), and the SDK. Full rationale recorded in .claude/SCHEMA_DECISIONS.md (gitignored, local- only). No emitter behavior change: the token stays "reaching-defs" and test/schema-v2.test.ts's assertion is untouched.
…hips Resolves each type's heritage signatures (base_classes/implements_types) through idBySig to can:// ids in the v2 emitter, dropping unresolved external/library supertypes rather than nulling them. Exposes the result as extends_ids/implements_ids on the V2 type node (JSON props unchanged) and projects them as EXTENDS/IMPLEMENTS relationships in Neo4j via the previously-dead RowBuilder.edgeToSymbol deferred gate, which was also re-pointed at the correct CanNode/id addressing (it had been hardcoded to a Symbol/signature pair no node in the v2 schema actually carries). Adds a first-party class hierarchy fixture to dataflow-app and extends both the schema conformance test and the issue #27 Neo4j<->JSON parity gate (the latter now also has an exhaustive total-edge accounting test).
The JSON envelope advertises analyzer{name,version} (#29), but the
Neo4j :Application node had neither — so the two co-primary
projections diverged on analyzer identity. project() now sources
name/version from the same V2Application.analyzer the envelope uses
(populated from ANALYZER_VERSION in src/utils/version.ts), declared
in the schema catalog and regenerated into schema.neo4j.json.
…sion
Bare name/version on :Application collided with project()'s _appName param
and every other CanNode's bare `name`. Namespace analyzer identity the same
way the JSON envelope does (analyzer{name,version}), unambiguous from the
node's own identity fields.
src/build/neo4j/rows.ts and src/schema/v2/dataflow.ts used raw 0x00 bytes as template-literal key separators (Map keys / sort-comparator keys), which made git treat both files as binary and hid contract- adjacent edits from review. Replace each raw NUL with the `\0` escape sequence — at runtime "\0" is still U+0000, so every key built from it is byte-identical; only the on-disk representation changes, from binary to plain text.
…lices (#45) Adds a first-party `ColoredShape extends Shape` interface to the dataflow-app hierarchy fixture and asserts the resulting Interface→Interface EXTENDS edge by node identity in the neo4j schema test — the one heritage path #33 handles but nothing previously exercised. Also asserts that L3 `@entry`/`@exit` body nodes byte-slice to the callable's whole span (per extract.ts's emitNode), alongside the existing statement-node slice test. Confirms `param` is contracted out before emission (folded onto `@entry`/`@formal_in`) and is never present as a raw body node, so no slice test is fabricated for it.
The container-gated bolt test still queried the v1-era :Symbol label, which no longer exists in schema v2 (every project-owned node carries :CanNode plus a specific kind label). Since :CanNode spans ALL kinds (not just the signature-keyed declaration types v1's :Symbol covered), the exhaustiveness check now enumerates all 11 specific labels that merge under :CanNode per schema.neo4j.json: Module, Class, Interface, Enum, TypeAlias, Namespace, Callable, Field, BodyNode, External, AnonymousCallable.
…#70) A plain endpoint-pair MERGE collapses legitimately-distinct edges of one type between the same two nodes: the DDG carries one edge per variable (plus the prov split), and a conditional's true/false CFG_NEXT pair shares its endpoints — the live graph silently materialized fewer dependence edges than the projection produced, with var/prov/kind props last-write-wins. Found in codeanalyzer-python during its v2 stage-5 gates (bolt count test 106 != 109) and fixed there the same way. - EdgeRow gains an optional `key` discriminant; RowBuilder.edge passes it. - bolt/cypher writers MERGE keyed families on `(a)-[r:TYPE {_k: row.k}]->(b)` and group keyed vs unkeyed rows separately. - project(): DDG keyed per `(var, prov)`, CFG_NEXT per `kind`; CDG/SUMMARY/ PARAM_* stay plain (their endpoint pairs are unique by construction). - schema catalog declares `_k` on DDG/CFG_NEXT; schema.neo4j.json regenerated. - test/neo4j-edge-identity.test.ts covers survival of per-var DDG rows, the keyed MERGE shape, and that undiscriminated families keep the plain MERGE.
#70) The issue #27 parity gate compares rows to JSON before the writers run, so a missing or non-unique _k would still collapse edges in the live graph without failing any test (the bolt container suite pushes at -a 1, where the discriminated families are empty). Assert on the L4 dataflow projection that every DDG/CFG_NEXT row carries a key, that (from, to, _k) tuples are fully distinct, and — non-vacuously — that both families really produce parallel edges between one endpoint pair on this fixture.
…; scope + index bolt lookups
A 2.0.0 push onto a 1.1.0 DB left the whole 1.x subgraph orphaned and
poisonous: old nodes carry twin TS labels so they match v2 queries, two
:Application nodes make the version read nondeterministic, and the
vanished-decl sweep never removed them (their id is null → three-valued
logic dropped the row).
- Legacy-generation wipe when the version gate forces a full upsert:
detect-and-delete 1.x _module carriers, shared External/Package/Decorator
nodes, and the id-less :Application in one idempotent step before the
per-module loop. Unanchored on purpose (legacy nodes lack :CanNode).
- Null-guard the vanished-decl sweep (x.id IS NULL OR NOT x.id IN keys).
- Scope the schema-version read to this app's :Application {id} so a shared
DB can't surface a foreign analyzer's version.
- Anchor the per-module edge-delete and decl sweep on (:CanNode {_module})
and add the cannode_module index to back them; regenerate schema.neo4j.json.
…d 1.1.0→2.0.0 migration scenario Replace the stale v1 magic numbers (constraints ≥ 8, indexes ≥ 11) with counts derived from the exported CONSTRAINTS/INDEXES catalog. Add a migration scenario: seed a minimal schema-1.1.0 graph (twin labels, an id-less :Application), run a full 2.0.0 bolt push against the same DB, and assert exactly one :Application survives (id set, version 2.0.0), no _module-carrying non-:CanNode residue remains, and TSModule count matches the fixture (no duplicates).
Add a minimal CI workflow (pull_request to main + push to main): a single ubuntu-latest job that checks out, sets up Bun, installs with a frozen lockfile, runs typecheck and the default test suite, then runs the container-gated Neo4j bolt suite with RUN_CONTAINER_TESTS=1 (ubuntu runners ship Docker, so testcontainers works out of the box).
- src/schema/schema.ts header: the spine field is external_symbols, not the nonexistent 'entrypoints'. - delete the orphaned TSSymbol interface (zero references). - rows.ts NodeRef.keyProp comment: only 'id' exists at schema v2. - project.ts idOf doc: it IS the identity (pass-through), not an @Local strip.
…ia the checker (#53)
…, project references, scoped installs (closes #56)
…uite (v2 envelope) project() takes the V2Application envelope (app.application.id, etc.), not the raw TSApplication analyze() returns — neo4j-schema.test.ts already routes through toV2Detailed()/toV2() before calling project(), but the container-only bolt suite (never run outside Docker CI) predates the v2 envelope reshape and passed analyze()'s result straight through, crashing on `root.id` being undefined. Also bumps the suite's default analysisLevel from 1 to 2: 6078c7e gated the call-graph (incl. jelly) solve behind level >= 2 since the v2 emitter discards it at level 1 anyway, so the level-1 default silently stopped producing the TS_CALLS edges this suite's "known resolved call edge" assertion checks for.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #46. Closes #27. Closes #28. Closes #29. Closes #31. Closes #32. Closes #33. Closes #43. Closes #44. Closes #45. Closes #53. Closes #55. Closes #56. Closes #60. Closes #66. Closes #68. Closes #70. Closes #9. Closes #10. Part of #64.
Highlights: multi-tsconfig program construction (nested Angular-style frontends now resolve; juice-shop external-call resolution 0.287 → 0.857);
_kedge-identity discriminant.Breaking: graph schema 2.0.0 (CanNode/id keys, TS-prefixed labels, TS_ rel types); analysis.json is the schema-v2 envelope. Migration: re-analyze; update queries Callable→TSCallable, CALLS→TS_CALLS. Migrating a shared 1.x database: the first 2.0.0 bolt push removes ALL pre-2.0.0 TS graph data (legacy-generation wipe).