diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..2d9d084 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,30 @@ +name: CI + +on: + pull_request: + branches: [main] + push: + branches: [main] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - name: Check out code + uses: actions/checkout@v4 + + - name: Set up Bun + uses: oven-sh/setup-bun@v2 + + - name: Install dependencies + run: bun install --frozen-lockfile + + - name: Typecheck + run: bun run typecheck + + - name: Unit + conformance tests + run: bun test + + # Container suite is opt-in; ubuntu-latest ships Docker, so testcontainers works out of the box. + - name: Neo4j bolt container tests + run: RUN_CONTAINER_TESTS=1 bun test test/neo4j-bolt.test.ts diff --git a/CLAUDE.md b/CLAUDE.md index c27c626..ce21910 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,12 +6,46 @@ Agent guidance for `codellm-devkit/codeanalyzer-typescript` (`cants`). `cants` is a TypeScript/JavaScript static analyzer built on the TypeScript compiler (via [ts-morph](https://ts-morph.com/)). It is the CLDK TypeScript backend: it emits -the canonical CLDK `analysis.json` — a **symbol table** plus a **resolver-based call -graph** — and can project that same analysis into a **Neo4j** property graph. It -mirrors its [Python](https://github.com/codellm-devkit/codeanalyzer-python) and +the **canonical schema v2** — one additive Code Property Graph — in **two projections**, +`analysis.json` and a **Neo4j** property graph. It mirrors its +[Python](https://github.com/codellm-devkit/codeanalyzer-python) and [Java](https://github.com/codellm-devkit/codeanalyzer-java) sibling analyzers, so output-shape parity with them is a first-class concern. +## Schema v2 — the additive CPG (read this before touching output) + +The output is **one scale-free structure**: a containment tree of nodes (id / kind / +`span` / children) with **typed edge overlays** (a CPG). Every classic artifact — symbol +table, call graph, CFG, PDG, SDG — is a *projection* of that one structure, and analysis +**levels** are how deeply it is populated (each level only ever *adds*, never rewrites): + +- **L1** (`-a 1`): the tree to callable depth — `application → symbol_table{module} → + types{}/functions{}/fields{} → callables{}` — plus `call` nodes in each callable's + `body{}` (with `callee` unresolved). `source` is stored once per module; every node's + text slices off it via `span.bytes`. +- **L2** (`-a 2`): the `call_graph` edge list (callable→callable) at the application scope, + and the `callee` slot on each call node refined `null → id` (the one sanctioned mutation). +- **L3** (`-a 3`): the rest of `body{}` (statements + `@entry`/`@exit`) and the intra-callable + edge lists `cfg`/`cdg`/`ddg` (reaching-definitions, `prov:["reaching-defs"]`) hung on each callable. +- **L4** (`-a 4`): the synthetic `@formal_in:N`/`@formal_out`/`/actual_in:N`/`/actual_out` + vertices, the intra-caller `summary` edges, and the application-scope `param_in`/`param_out` + lists (the interprocedural SDG). + +**Identity is two-tier**: durable `can://////` ids at callable +depth and above; ordinal `@:` (or `@`) below. Intra-callable +edge lists use **bare local ids**; cross-callable lists use **fully-qualified `can://…@local`** +ids. `L1 ⊆ L2 ⊆ L3 ⊆ L4` is a CI-checkable monotonicity gate (`test/schema-v2.test.ts`). The +model + every decision live in `.claude/SCHEMA_DECISIONS.md` (§ "Schema v2 migration") and the +skillset's `canonical-schema.md`. + +**Provider/client boundary:** the analyzer is a *pure graph provider* — it emits the graph +substrate (CFG/PDG/SDG + `summary` edges) and stops. Slicing and taint are reachability +*queries* over it and belong to the frontend SDK; never add a `taint_flows` section here. + +The v1→v2 emitter is a pure transform in **`src/schema/v2/`** (`emit.ts` reshapes the v1 +in-memory model, `dataflow.ts` maps `program_graphs` into the tree) — the parse/resolve/ +dataflow *compute* is untouched; only serialization is v2. + The call graph defaults to the **union** of two backends: the TS compiler's resolver and the embedded [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer (which recovers higher-order/callback edges the resolver misses). Merged edges keep a @@ -29,14 +63,23 @@ it first; everything else is a stage it calls, in order: JSDoc, with precise source spans. 3. **call graph** (`src/semantic_analysis`) — `selectProvider()` picks tsc / jelly / union; each provider returns edges + external (phantom) symbols. -4. **cache** (`src/utils/cache.ts`) — content-hash cache under `.codeanalyzer/`, so - re-analysis only touches what changed. -5. **output** (`src/build`, `src/build/neo4j`) — `analysis.json`, a self-contained - `graph.cypher` snapshot, or an incremental Bolt push to a live database. - -The shape of everything is the **schema** in `src/schema` (`TSApplication` is the top -type). The Neo4j schema is versioned and enforced by a conformance test — treat it as -a contract. +4. **program graphs** (`src/dataflow`) — levels 3–4 (`-a 3`/`-a 4`): CFG → post-dominance/CDG → + access-path def-use → PDG → SCC-condensed bottom-up summaries → SDG. This is the *compute*; + it produces the internal `program_graphs` model, which `src/schema/v2/dataflow.ts` then maps + **into the v2 tree** (`body{}` + `cfg`/`cdg`/`ddg`/`summary` per callable + `param_in`/ + `param_out`). Decisions: `.claude/SCHEMA_DECISIONS.md`; contract + staged follow-ups: issue #2. +5. **cache** (`src/utils/cache.ts`) — content-hash cache under `.codeanalyzer/`, so + re-analysis only touches what changed (levels 3–4 also record summaries + + dependency edges in `graphs_summaries.json`). +6. **output** (`src/schema/v2`, `src/build/neo4j`) — `src/schema/v2/emit.ts` reshapes the v1 + compute model into the schema-v2 `analysis.json`; `src/build/neo4j` projects the *same* v2 + tree into a `graph.cypher` snapshot or an incremental Bolt push. `--emit neo4j` is always + **full-depth** (levels gate the JSON path only; combining `-a`/`--graphs` with it is an error). + +The **output** shape is schema v2 (`src/schema/v2/model.ts`, `V2Application` the top type); the +types in `src/schema` (`TSApplication`) are the *internal compute model* the emitter transforms. +The Neo4j schema (`src/build/neo4j/schema.ts`, v2.0.0) is versioned and enforced by a conformance +test — treat both as contracts and keep them in lockstep with the JSON. ## Directory map @@ -47,10 +90,12 @@ a contract. | `src/options` | Parsed CLI options / `AnalysisOptions` | | `src/syntactic_analysis` | Symbol table (ts-morph traversal) | | `src/semantic_analysis` | Call-graph providers (tsc, jelly, union), phantoms | -| `src/schema` | `TSApplication` types + signatures (the output contract) | -| `src/build` | Dep materialization + output; `build/neo4j` = graph projection | -| `src/utils` | fs, caching, logging, serialization, version | -| `test` | Bun tests + `fixtures/sample-app` | +| `src/dataflow` | L3/L4 program-graph **compute**: CFG, dominance/CDG, def-use, summaries, SDG | +| `src/schema` | `TSApplication` — the internal compute model + `signatureOf` + `program_graphs` | +| `src/schema/v2` | **the schema-v2 emitter**: `model.ts` (target shape) + `emit.ts` (tree/L1/L2) + `dataflow.ts` (L3/L4) | +| `src/build` | Dep materialization; `build/neo4j` = the v2 graph projection (project/rows/cypher/bolt/schema) | +| `src/utils` | fs, caching, logging, serialization (`serialize.ts` → `toV2`), version | +| `test` | Bun tests + `fixtures/sample-app` + `fixtures/dataflow-app`; `schema-v2.test.ts` = the L1–L4 gates | ## Commands diff --git a/README.md b/README.md index 5646c55..55d1a92 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ # codeanalyzer-typescript (`cants`) -**A TypeScript/JavaScript static-analysis toolkit — the CLDK backend that emits a canonical symbol table and call graph, as `analysis.json` or a Neo4j property graph.** +**A TypeScript/JavaScript static-analysis toolkit — the CLDK backend that emits the canonical schema-v2 Code Property Graph (symbol table → call graph → intraprocedural dataflow → interprocedural SDG), as `analysis.json` or a Neo4j property graph.** [![PyPI](https://img.shields.io/pypi/v/codeanalyzer-typescript?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/codeanalyzer-typescript/) [![Python](https://img.shields.io/pypi/pyversions/codeanalyzer-typescript?style=for-the-badge&logo=python&logoColor=white)](https://pypi.org/project/codeanalyzer-typescript/) @@ -16,9 +16,10 @@ --- `cants` is a static analyzer for TypeScript/JavaScript built on the TypeScript compiler (via -[ts-morph](https://ts-morph.com/)). It produces the canonical CodeLLM-DevKit (CLDK) -`analysis.json` — a symbol table plus a resolver-based call graph — and can project that same -analysis into a **Neo4j property graph**. It is the TypeScript backend behind +[ts-morph](https://ts-morph.com/)). It produces the canonical CodeLLM-DevKit (CLDK) **schema v2** +— one additive Code Property Graph, built up level by level (symbol table → call graph → +intraprocedural dataflow → interprocedural SDG) — as `analysis.json` and can project that same +structure into a **Neo4j property graph**. It is the TypeScript backend behind [CLDK](https://github.com/codellm-devkit/python-sdk), mirroring its [Python](https://github.com/codellm-devkit/codeanalyzer-python) and [Java](https://github.com/codellm-devkit/codeanalyzer-java) siblings. @@ -135,8 +136,8 @@ written to `analysis.json` (or `graph.cypher` for `--emit neo4j`) in that direct ```text Usage: cants [options] -CLDK TypeScript analyzer — emits the canonical analysis.json (symbol table + -resolver call graph), or a Neo4j graph. +CLDK TypeScript analyzer — emits the canonical schema-v2 CPG (symbol table → +call graph → dataflow → SDG) as analysis.json, or a Neo4j graph. Options: -i, --input project root to analyze (not required for @@ -157,9 +158,19 @@ Options: visible in shell history / process list) (default: "neo4j", env: NEO4J_PASSWORD) --neo4j-database Neo4j database name (env: NEO4J_DATABASE) - -a, --analysis-level analysis depth: 1 = symbol table + tsc resolver - call graph + RTA (default); 2 = call graph - (default: "1") + -a, --analysis-level analysis depth: 1 = symbol table (default); 2 = + + resolver call graph; 3 = + intraprocedural + dataflow (cfg/cdg/ddg); 4 = + interprocedural + SDG (param_in/param_out/summary) (default: "1") + --graphs dataflow sections to emit, comma-separated: cfg + | dfg | pdg (require -a 3) | sdg (requires -a + 4); default: all rungs at or below the level + --graph-field-depth access-path depth bound (k-limit) for level-3 + dataflow (default: "3") + -j, --jobs worker parallelism for level-3 graphs (default: + sequential; opt in with N ≥ 2 on large projects + — each worker loads its own copy of the + program) -t, --target-files restrict analysis to specific files (incremental) --skip-tests skip test trees (default) @@ -213,32 +224,96 @@ Options: cants --input ./my-ts-project --eager --cache-dir /path/to/custom-cache ``` +6. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:** + ```sh + cants --input ./my-ts-project -a 3 # full program_graphs section + cants --input ./my-ts-project -a 3 --graphs cfg,pdg # scope the emitted graphs + ``` + ## Output targets `cants` builds one analysis in memory and can emit it three ways (`--emit`): ### `analysis.json` (default) -A `TSApplication` document — the canonical CLDK contract the Python SDK parses: +The **canonical schema v2** — one additive Code Property Graph: a containment tree of nodes +(`id` / `kind` / `span` / children) with typed edge overlays. Analysis **levels** populate it more +deeply; each level only ever *adds*. ```jsonc { - "symbol_table": { /* file path → module (classes, interfaces, enums, - type aliases, functions, namespaces, variables, …) */ }, - "call_graph": [ /* CALL_DEP edges: { source, target, type, weight, - provenance, tags } keyed by callable signature */ ], - "external_symbols": { /* phantom stubs for call targets outside the project */ } + "schema_version": "2.0.0", "language": "typescript", "max_level": 4, "k_limit": 3, + "application": { + "id": "can://typescript/", "kind": "application", + "symbol_table": { // L1: the tree, keyed by file path + "": { "kind": "module", "source": "…", + "types": { /* class | interface | enum | type_alias | namespace nodes */ }, + "functions": { /* callable nodes: { id, kind, span, body{}, cfg[], cdg[], ddg[], summary[] } */ }, + "fields": { /* module-level bindings */ } } }, + "call_graph": [ /* L2: { src, dst, prov, weight } — callable → callable, can:// ids */ ], + "param_in": [ /* L4: actual_in → formal_in, fully-qualified can://…@local ids */ ], + "param_out": [ /* L4: formal_out → actual_out */ ] + } } ``` -Caller- and callee-side identifiers come from a single signature canonicalizer, so call-graph -`source`/`target` values byte-match the corresponding `symbol_table` / `external_symbols` keys. +Each callable's `body{}` is keyed by local id (`line:col`, or `@entry`/`@formal_in:N`/… for +synthetic vertices); intra-callable edge lists (`cfg`/`cdg`/`ddg`/`summary`) use those bare local +ids, cross-callable lists use fully-qualified `can://…@local` ids. A single signature canonicalizer +underlies every `can://` id, so call edges, dataflow edges, and tree nodes all join. The full model +is `.claude/SCHEMA_DECISIONS.md` (§ "Schema v2 migration") and the CLDK `canonical-schema.md`. + +### Dataflow (`-a 3` intraprocedural, `-a 4` interprocedural) + +Native dependence graphs, built in-process from the same ts-morph AST (no external engine), grown +**into the tree** (not a separate section): + +- **`-a 3`** completes each callable's `body{}` with statement nodes and hangs the intra-callable + edge lists `cfg` (exceptional control flow), `cdg` (control dependence), and `ddg` (data + dependence via reaching-definitions, `prov:["reaching-defs"]`) on the callable. +- **`-a 4`** adds the synthetic `@formal_in:N` / `@formal_out` / `/actual_in:N` / `/actual_out` + vertices, the intra-caller `summary` edges, and the application-scope `param_in` / `param_out` + lists — the whole-program System Dependence Graph. + +`-a 3` implies `-a 2`; `-a 4` implies `-a 3`. `--graphs cfg,dfg,pdg,sdg` scopes which rungs emit +(`cfg`/`dfg`/`pdg` require `-a 3`, `sdg` requires `-a 4`). `L1 ⊆ L2 ⊆ L3 ⊆ L4` is a monotonicity +gate. Every node is addressed by its `can://…@local` id, so dataflow edges, call edges, and tree +nodes all join. + +**Substrate (locked in [issue #2](https://github.com/codellm-devkit/codeanalyzer-typescript/issues/2)):** +the CFG and reaching-definitions are hand-built from the ts-morph AST; the call-graph oracle is +the existing provenance-merged tsc ∪ Jelly graph; aliasing is a flow-insensitive copy-alias MVP +(Jelly points-to-backed propagation is a staged upgrade). Function summaries are composed +bottom-up over the SCC condensation of the call graph, with k-limited access paths; module +globals ride the SDG as extra parameters. The analysis is deliberately sound-leaning and +over-approximate; known unsoundness (dynamic `eval`, reflection/monkey-patching, npm-internal +effects) is recorded in `.claude/SCHEMA_DECISIONS.md`. The analyzer is a **pure graph provider**: +it emits the dependence-graph substrate (CFG/PDG/SDG + `summary` edges) and stops — backward +slicing and taint are reachability *queries* over the SDG that live in the frontend SDK, not here. + +**Parallelism (`-j/--jobs`).** The pipeline implements the level-3 parallel execution model: +stage-1–4 extraction fans out per callable over a Bun worker pool (partitioned by file) and is +posted *before* the call-graph solve so the two overlap; summary composition runs as a +Kahn-style ready-queue wavefront over the SCC condensation DAG (the SCC is the atomic unit). +`--jobs N` output is **byte-identical** to `--jobs 1` (node ids are span-ordered, all edge lists +are collect-then-sorted, and the SCC fixpoint is a pure function of its inputs) — enforced by a +differential test. It is off by default and worth opting into only on large codebases: ts-morph +ASTs cannot cross the worker boundary, so each extraction worker loads its own copy of the +program, which dominates the parallelizable graph math on small/mid repos (self-analysis runs +2.5× slower at `-j 14`). Worker failure at any stage degrades to the sequential path with a +warning — never to wrong or missing output. + +Levels 1/2 are unaffected: nothing in level 3 runs unless `-a 3` is requested. ### Neo4j graph -`--emit neo4j` projects the same analysis into a labeled property graph (declarations keyed by -their signature under a shared `:Symbol` label; calls, imports, inheritance, decorators, and call -sites as relationships): +`--emit neo4j` projects the **same v2 tree** into a labeled property graph: every node keyed by its +`can://` id under a shared `:CanNode` merge label (+ a `TS`-prefixed specific kind label, e.g. +`:TSModule`, `:TSCallable`), containment as `TS_HAS_MODULE`/`TS_DECLARES`/`TS_HAS_METHOD`/ +`TS_HAS_FIELD`/`TS_HAS_BODY_NODE` edges, and the overlays (`TS_CALLS`, `TS_CFG_NEXT`, `TS_CDG`, +`TS_DDG`, `TS_SUMMARY`, `TS_PARAM_IN`, `TS_PARAM_OUT`) as typed relationships. The graph is +**always full-depth** — analysis levels gate the JSON path only, so combining `-a`/`--graphs` with +`--emit neo4j` is an error: - **Without `--neo4j-uri`** — writes a self-contained `graph.cypher` (constraints + indexes, a scoped wipe, then batched `MERGE`s). Load it with `cypher-shell < graph.cypher`. diff --git a/package.json b/package.json index 6c13526..02c960b 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "codeanalyzer-typescript", - "version": "0.5.0", - "description": "CLDK TypeScript analyzer — emits the canonical CLDK analysis.json (symbol table + resolver-based call graph) via ts-morph.", + "version": "1.0.0", + "description": "CLDK TypeScript analyzer — emits the canonical schema v2 (additive CPG: symbol table → call graph → intraprocedural dataflow → interprocedural SDG) as analysis.json and Neo4j, via ts-morph.", "type": "module", "module": "src/index.ts", "bin": { @@ -9,7 +9,7 @@ }, "scripts": { "start": "bun run src/index.ts", - "build": "bun build ./src/main.ts --compile --external @babel/preset-typescript --outfile dist/cants", + "build": "bun build ./src/main.ts ./src/dataflow/worker.ts --compile --external @babel/preset-typescript --outfile dist/cants", "gen:schema": "bun run src/index.ts --emit schema > schema.neo4j.json", "gen:readme": "bun run scripts/update-readme.ts", "test:container": "RUN_CONTAINER_TESTS=1 bun test test/neo4j-bolt.test.ts", diff --git a/schema.neo4j.json b/schema.neo4j.json index c10af05..dc97524 100644 --- a/schema.neo4j.json +++ b/schema.neo4j.json @@ -1,478 +1,401 @@ { - "schema_version": "1.1.0", + "schema_version": "2.0.0", "generator": "codeanalyzer-typescript", - "marker_labels": [ - "Entrypoint" - ], + "marker_labels": [], "node_labels": [ { - "label": "Application", + "label": "TSApplication", "mergeLabel": "Application", - "key": "name", + "key": "id", "properties": { - "name": "string", - "schema_version": "string" + "id": "string", + "schema_version": "string", + "language": "string", + "max_level": "integer", + "k_limit": "integer", + "analyzer_name": "string", + "analyzer_version": "string" } }, { - "label": "Module", - "mergeLabel": "Module", - "key": "file_key", + "label": "TSModule", + "mergeLabel": "CanNode", + "key": "id", "properties": { - "file_key": "string", - "module_name": "string", + "id": "string", + "kind": "string", + "_module": "string", + "name": "string", "is_tsx": "boolean", "is_declaration_file": "boolean", "content_hash": "string", - "last_modified": "integer", - "file_size": "integer", - "_module": "string" + "start_line": "integer", + "end_line": "integer" } }, { - "label": "Class", - "mergeLabel": "Symbol", - "key": "signature", + "label": "TSClass", + "mergeLabel": "CanNode", + "key": "id", "properties": { + "id": "string", + "kind": "string", + "_module": "string", "signature": "string", "name": "string", - "code": "string", "base_classes": "string[]", "implements_types": "string[]", - "type_parameter_names": "string[]", - "docstring": "string", "is_abstract": "boolean", "is_exported": "boolean", "is_ambient": "boolean", "start_line": "integer", - "end_line": "integer", - "framework": "string", - "detection_source": "string", - "route_path": "string", - "http_methods": "string[]", - "entrypoint_count": "integer", - "_module": "string" + "end_line": "integer" } }, { - "label": "Interface", - "mergeLabel": "Symbol", - "key": "signature", + "label": "TSInterface", + "mergeLabel": "CanNode", + "key": "id", "properties": { + "id": "string", + "kind": "string", + "_module": "string", "signature": "string", "name": "string", - "code": "string", "base_classes": "string[]", - "type_parameter_names": "string[]", - "call_signatures": "string[]", - "index_signatures": "string[]", - "docstring": "string", "is_exported": "boolean", "is_ambient": "boolean", "start_line": "integer", - "end_line": "integer", - "_module": "string" + "end_line": "integer" } }, { - "label": "Enum", - "mergeLabel": "Symbol", - "key": "signature", + "label": "TSEnum", + "mergeLabel": "CanNode", + "key": "id", "properties": { + "id": "string", + "kind": "string", + "_module": "string", "signature": "string", "name": "string", - "code": "string", - "member_names": "string[]", - "member_values": "string[]", - "docstring": "string", "is_const": "boolean", "is_exported": "boolean", "is_ambient": "boolean", "start_line": "integer", - "end_line": "integer", - "_module": "string" + "end_line": "integer" } }, { - "label": "TypeAlias", - "mergeLabel": "Symbol", - "key": "signature", + "label": "TSTypeAlias", + "mergeLabel": "CanNode", + "key": "id", "properties": { + "id": "string", + "kind": "string", + "_module": "string", "signature": "string", "name": "string", - "code": "string", "aliased_type": "string", - "type_parameter_names": "string[]", - "docstring": "string", "is_exported": "boolean", "is_ambient": "boolean", "start_line": "integer", - "end_line": "integer", - "_module": "string" + "end_line": "integer" } }, { - "label": "Namespace", - "mergeLabel": "Symbol", - "key": "signature", + "label": "TSNamespace", + "mergeLabel": "CanNode", + "key": "id", "properties": { + "id": "string", + "kind": "string", + "_module": "string", "signature": "string", "name": "string", - "docstring": "string", "is_exported": "boolean", "is_ambient": "boolean", "start_line": "integer", - "end_line": "integer", - "_module": "string" + "end_line": "integer" } }, { - "label": "Callable", - "mergeLabel": "Symbol", - "key": "signature", + "label": "TSCallable", + "mergeLabel": "CanNode", + "key": "id", "properties": { + "id": "string", + "kind": "string", + "_module": "string", "signature": "string", "name": "string", - "path": "string", - "kind": "string", "return_type": "string", "cyclomatic_complexity": "integer", - "code": "string", - "code_start_line": "integer", - "start_line": "integer", - "end_line": "integer", "accessibility": "string", "accessor_kind": "string", - "docstring": "string", - "type_parameter_names": "string[]", - "parameters_json": "string", - "accessed_symbols_json": "string", "is_static": "boolean", "is_abstract": "boolean", "is_async": "boolean", "is_generator": "boolean", - "is_optional": "boolean", - "is_readonly": "boolean", "is_exported": "boolean", "is_ambient": "boolean", "is_implicit": "boolean", - "framework": "string", - "detection_source": "string", - "route_path": "string", - "http_methods": "string[]", - "entrypoint_count": "integer", - "_module": "string" - } - }, - { - "label": "External", - "mergeLabel": "Symbol", - "key": "signature", - "properties": { - "signature": "string", - "name": "string", - "module": "string" + "start_line": "integer", + "end_line": "integer" } }, { - "label": "AnonymousCallable", - "mergeLabel": "Symbol", - "key": "signature", + "label": "TSField", + "mergeLabel": "CanNode", + "key": "id", "properties": { - "signature": "string", + "id": "string", + "kind": "string", + "_module": "string", "name": "string", - "path": "string", + "type": "string", "start_line": "integer", - "start_column": "integer", - "_module": "string" - } - }, - { - "label": "Package", - "mergeLabel": "Package", - "key": "name", - "properties": { - "name": "string" - } - }, - { - "label": "Decorator", - "mergeLabel": "Decorator", - "key": "qualified_name", - "properties": { - "qualified_name": "string", - "name": "string" + "end_line": "integer" } }, { - "label": "CallSite", - "mergeLabel": "CallSite", + "label": "TSBodyNode", + "mergeLabel": "CanNode", "key": "id", "properties": { "id": "string", - "method_name": "string", - "receiver_expr": "string", - "receiver_type": "string", - "argument_types": "string[]", - "type_arguments": "string[]", - "return_type": "string", - "callee_signature": "string", - "is_constructor_call": "boolean", - "is_optional_chain": "boolean", + "kind": "string", + "_module": "string", + "of": "string", + "parent": "string", + "callee": "string", "start_line": "integer", - "start_column": "integer", - "end_line": "integer", - "end_column": "integer", - "_module": "string" + "end_line": "integer" } }, { - "label": "Attribute", - "mergeLabel": "Attribute", + "label": "TSExternal", + "mergeLabel": "CanNode", "key": "id", "properties": { "id": "string", + "kind": "string", + "_module": "string", "name": "string", - "type": "string", - "initializer": "string", - "accessibility": "string", - "docstring": "string", - "is_static": "boolean", - "is_readonly": "boolean", - "is_optional": "boolean", - "is_abstract": "boolean", - "start_line": "integer", - "end_line": "integer", - "_module": "string" + "module": "string" } }, { - "label": "Variable", - "mergeLabel": "Variable", + "label": "TSAnonymousCallable", + "mergeLabel": "CanNode", "key": "id", "properties": { "id": "string", + "kind": "string", + "_module": "string", "name": "string", - "type": "string", - "initializer": "string", - "scope": "string", - "declaration_kind": "string", - "is_readonly": "boolean", - "is_exported": "boolean", + "path": "string", "start_line": "integer", - "end_line": "integer", - "_module": "string" + "start_column": "integer" } } ], "relationship_types": [ { - "type": "HAS_MODULE", + "type": "TS_HAS_MODULE", "from": [ - "Application" + "TSApplication" ], "to": [ - "Module" + "TSModule" ], "properties": {} }, { - "type": "DECLARES", + "type": "TS_DECLARES", "from": [ - "Module", - "Namespace", - "Class", - "Callable" + "TSModule", + "TSNamespace", + "TSCallable" ], "to": [ - "Class", - "Interface", - "Enum", - "TypeAlias", - "Namespace", - "Callable" + "TSClass", + "TSInterface", + "TSEnum", + "TSTypeAlias", + "TSNamespace", + "TSCallable" ], "properties": {} }, { - "type": "HAS_METHOD", + "type": "TS_HAS_METHOD", "from": [ - "Class", - "Interface" + "TSClass", + "TSInterface" ], "to": [ - "Callable" + "TSCallable" ], "properties": {} }, { - "type": "HAS_ATTRIBUTE", + "type": "TS_HAS_FIELD", "from": [ - "Class", - "Interface" + "TSModule", + "TSClass", + "TSInterface", + "TSEnum", + "TSNamespace" ], "to": [ - "Attribute" + "TSField" ], "properties": {} }, { - "type": "DECLARES_VAR", + "type": "TS_HAS_BODY_NODE", "from": [ - "Module", - "Namespace", - "Callable" + "TSCallable", + "TSAnonymousCallable" ], "to": [ - "Variable" + "TSBodyNode" ], "properties": {} }, { - "type": "HAS_CALLSITE", + "type": "TS_RESOLVES_TO", "from": [ - "Callable" + "TSBodyNode" ], "to": [ - "CallSite" + "TSCallable", + "TSExternal", + "TSAnonymousCallable" ], "properties": {} }, { - "type": "RESOLVES_TO", + "type": "TS_CALLS", "from": [ - "CallSite" + "TSCallable", + "TSAnonymousCallable" ], "to": [ - "Callable", - "External" + "TSCallable", + "TSExternal", + "TSAnonymousCallable" ], - "properties": {} + "properties": { + "weight": "integer", + "prov": "string[]" + } }, { - "type": "CALLS", + "type": "TS_CFG_NEXT", "from": [ - "Callable" + "TSBodyNode" ], "to": [ - "Callable", - "External" + "TSBodyNode" ], "properties": { - "weight": "integer", - "provenance": "string[]", - "dispatch": "string", - "external": "boolean", - "module": "string" + "kind": "string", + "_k": "string" } }, { - "type": "EXTENDS", + "type": "TS_CDG", "from": [ - "Class", - "Interface" + "TSBodyNode" ], "to": [ - "Class", - "Interface" + "TSBodyNode" ], "properties": {} }, { - "type": "IMPLEMENTS", + "type": "TS_DDG", "from": [ - "Class" + "TSBodyNode" ], "to": [ - "Interface" + "TSBodyNode" ], - "properties": {} + "properties": { + "var": "string", + "prov": "string[]", + "_k": "string" + } }, { - "type": "IMPORTS", + "type": "TS_SUMMARY", "from": [ - "Module" + "TSBodyNode" ], "to": [ - "Module", - "Package" + "TSBodyNode" ], "properties": { - "imported_names": "string[]", - "import_kinds": "string[]", - "is_type_only": "boolean" + "var": "string" } }, { - "type": "RE_EXPORTS", + "type": "TS_PARAM_IN", "from": [ - "Module" + "TSBodyNode" ], "to": [ - "Module", - "Package" + "TSBodyNode" ], - "properties": {} + "properties": { + "var": "string" + } }, { - "type": "MEMBER_OF", + "type": "TS_PARAM_OUT", "from": [ - "External" + "TSBodyNode" ], "to": [ - "Package" + "TSBodyNode" + ], + "properties": { + "var": "string" + } + }, + { + "type": "TS_EXTENDS", + "from": [ + "TSClass", + "TSInterface" + ], + "to": [ + "TSClass", + "TSInterface" ], "properties": {} }, { - "type": "DECORATED_BY", + "type": "TS_IMPLEMENTS", "from": [ - "Class", - "Callable", - "Attribute" + "TSClass" ], "to": [ - "Decorator" + "TSInterface", + "TSClass" ], - "properties": { - "positional_arguments": "string[]", - "keyword_arguments_json": "string", - "start_line": "integer", - "end_line": "integer" - } + "properties": {} } ], "constraints": [ - "CREATE CONSTRAINT application_name IF NOT EXISTS FOR (x:Application) REQUIRE x.name IS UNIQUE", - "CREATE CONSTRAINT module_file_key IF NOT EXISTS FOR (x:Module) REQUIRE x.file_key IS UNIQUE", - "CREATE CONSTRAINT symbol_signature IF NOT EXISTS FOR (x:Symbol) REQUIRE x.signature IS UNIQUE", - "CREATE CONSTRAINT package_name IF NOT EXISTS FOR (x:Package) REQUIRE x.name IS UNIQUE", - "CREATE CONSTRAINT decorator_qualified_name IF NOT EXISTS FOR (x:Decorator) REQUIRE x.qualified_name IS UNIQUE", - "CREATE CONSTRAINT callsite_id IF NOT EXISTS FOR (x:CallSite) REQUIRE x.id IS UNIQUE", - "CREATE CONSTRAINT attribute_id IF NOT EXISTS FOR (x:Attribute) REQUIRE x.id IS UNIQUE", - "CREATE CONSTRAINT variable_id IF NOT EXISTS FOR (x:Variable) REQUIRE x.id IS UNIQUE" + "CREATE CONSTRAINT application_id IF NOT EXISTS FOR (x:Application) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT cannode_id IF NOT EXISTS FOR (x:CanNode) REQUIRE x.id IS UNIQUE" ], "indexes": [ - "CREATE INDEX callable_name IF NOT EXISTS FOR (c:Callable) ON (c.name)", - "CREATE INDEX decorator_name IF NOT EXISTS FOR (d:Decorator) ON (d.name)", - "CREATE FULLTEXT INDEX code_fts IF NOT EXISTS FOR (c:Callable) ON EACH [c.code, c.docstring]" - ], - "label_twins": { - "Application": "TSApplication", - "Module": "TSModule", - "Class": "TSClass", - "Interface": "TSInterface", - "Enum": "TSEnum", - "TypeAlias": "TSTypeAlias", - "Namespace": "TSNamespace", - "Callable": "TSCallable", - "External": "TSExternal", - "AnonymousCallable": "TSAnonymousCallable", - "Package": "TSPackage", - "Decorator": "TSDecorator", - "CallSite": "TSCallSite", - "Attribute": "TSAttribute", - "Variable": "TSVariable", - "Entrypoint": "TSEntrypoint" - } + "CREATE INDEX callable_name IF NOT EXISTS FOR (c:TSCallable) ON (c.name)", + "CREATE INDEX cannode_kind IF NOT EXISTS FOR (n:CanNode) ON (n.kind)", + "CREATE INDEX cannode_module IF NOT EXISTS FOR (n:CanNode) ON (n._module)" + ] } diff --git a/src/build/materialize.ts b/src/build/materialize.ts index 7181f5a..135496e 100644 --- a/src/build/materialize.ts +++ b/src/build/materialize.ts @@ -1,11 +1,28 @@ import { execFileSync } from "node:child_process"; import * as fs from "node:fs"; import * as path from "node:path"; +import { ts } from "ts-morph"; import type { AnalysisOptions } from "../options"; +import { SKIP_DIRS } from "../syntactic_analysis/discovery"; import type { Logger } from "../utils"; +/** + * One ts-morph program to construct. `configPath` is the tsconfig the Project is built from (or + * `null` for default compiler options); `scopeDir` is the subtree this program CLAIMS — every + * discovered source file is assigned to the DEEPEST program whose `scopeDir` is an ancestor of it. + * For a leaf config reached through a solution-style config, `scopeDir` is the SOLUTION config's + * directory (so its sibling files aren't stranded), which is why scope is tracked separately. + */ +export interface ProgramSpec { + configPath: string | null; + scopeDir: string; +} + export interface Materialization { + // The ROOT program's tsconfig — kept for single-program consumers (dataflow, backward compat). tsConfigFilePath: string | null; + // Every program to build, DEEPEST-scope programs first and the root program LAST. + programs: ProgramSpec[]; degraded: boolean; notes: string[]; } @@ -31,37 +48,65 @@ export function materialize(opts: AnalysisOptions, log: Logger): Materialization const notes: string[] = []; let degraded = false; - const tsconfig = findProjectConfig(root); + const programs = discoverPrograms(root, opts.skipTests); + const rootProgram = programs[programs.length - 1]!; // root is always last (see discoverPrograms) + const tsconfig = rootProgram.configPath; notes.push(tsconfig ? `config: ${path.relative(root, tsconfig) || path.basename(tsconfig)}` : "no tsconfig/jsconfig — using default compiler options"); + for (const p of programs) { + if (p === rootProgram) continue; + notes.push(`nested program: ${path.relative(root, p.configPath ?? p.scopeDir)} (scope ${path.relative(root, p.scopeDir) || "."})`); + } + + // Materialize deps at the root, then at each nested program scope that has its own package.json + // (Angular monorepos install the frontend separately — its node_modules never lives at the root). + // Dedup by resolved scope dir so a solution config with several referenced leaves installs once. + const installedScopes = new Set([path.resolve(root)]); + if (installDeps(root, opts, log, notes)) degraded = true; + for (const p of programs) { + const dir = path.resolve(p.scopeDir); + if (installedScopes.has(dir)) continue; + installedScopes.add(dir); + if (!fs.existsSync(path.join(dir, "package.json"))) continue; + if (installDeps(dir, opts, log, notes)) degraded = true; + } - const hasPkg = fs.existsSync(path.join(root, "package.json")); - const hasNodeModules = fs.existsSync(path.join(root, "node_modules")); + return { tsConfigFilePath: tsconfig, programs, degraded, notes }; +} + +/** + * Install dependencies in `dir` (the root, or a nested program scope). Same policy everywhere: + * `--no-build` / no package.json / present-node_modules (unless `--eager`) all skip; failures + * degrade rather than crash. Returns whether the install DEGRADED (failed). + */ +function installDeps(dir: string, opts: AnalysisOptions, log: Logger, notes: string[]): boolean { + const rel = path.relative(opts.input, dir) || "."; + const hasPkg = fs.existsSync(path.join(dir, "package.json")); + const hasNodeModules = fs.existsSync(path.join(dir, "node_modules")); if (opts.noBuild) { - notes.push("--no-build: skipping dependency materialization"); - } else if (!hasPkg) { - notes.push("no package.json — nothing to materialize"); - } else if (hasNodeModules && !opts.eager) { - notes.push("node_modules present — reused (pass --eager to reinstall)"); - } else { - const inst = resolveInstaller(root); - try { - log.info(`materializing dependencies: ${inst.label}`); - execFileSync(inst.bin, inst.args, { - cwd: root, - stdio: ["ignore", "ignore", "inherit"], - timeout: 600_000, - }); - notes.push(`ran ${inst.label}`); - } catch (e) { - degraded = true; - const msg = String((e as Error).message ?? e).slice(0, 160); - notes.push(`dependency install failed — continuing with partial types (${msg})`); - log.warn("dependency materialization failed; continuing with partial types"); - } + notes.push(`--no-build: skipping dependency materialization (${rel})`); + return false; + } + if (!hasPkg) { + notes.push(`no package.json — nothing to materialize (${rel})`); + return false; + } + if (hasNodeModules && !opts.eager) { + notes.push(`node_modules present — reused (${rel}; pass --eager to reinstall)`); + return false; + } + const inst = resolveInstaller(dir); + try { + log.info(`materializing dependencies (${rel}): ${inst.label}`); + execFileSync(inst.bin, inst.args, { cwd: dir, stdio: ["ignore", "ignore", "inherit"], timeout: 600_000 }); + notes.push(`ran ${inst.label} (${rel})`); + return false; + } catch (e) { + const msg = String((e as Error).message ?? e).slice(0, 160); + notes.push(`dependency install failed — continuing with partial types (${rel}: ${msg})`); + log.warn(`dependency materialization failed (${rel}); continuing with partial types`); + return true; } - - return { tsConfigFilePath: tsconfig, degraded, notes }; } interface Installer { @@ -101,6 +146,119 @@ function binAvailable(bin: string): boolean { } } +// ============================================================================================== +// Program discovery — find every ts-morph program the monorepo needs, not just the root one. +// ============================================================================================== + +/** + * Discover the programs to build. The ROOT program keeps the historical `findProjectConfig` + * behavior (root tsconfig or default options) and is returned LAST. Nested programs come from + * every `tsconfig.json` found under the tree (outside `node_modules`), resolved to LEAF configs: + * a solution-style config (`files: []`/absent + non-empty `references`) contributes its referenced + * configs instead of itself, scoped to the solution config's own directory. Deeper-scope programs + * are returned before shallower ones so a first-match-wins assignment picks the deepest owner. + */ +export function discoverPrograms(root: string, skipTests: boolean): ProgramSpec[] { + const rootResolved = path.resolve(root); + const nested: ProgramSpec[] = []; + const seenLeaf = new Set(); + + for (const cfg of walkTsconfigs(root)) { + const dir = path.dirname(cfg); + if (path.resolve(dir) === rootResolved) continue; // the root config is the root program's job + for (const leaf of resolveLeafConfigs(cfg, skipTests, new Set())) { + const key = `${path.resolve(leaf)}|${path.resolve(dir)}`; + if (seenLeaf.has(key)) continue; + seenLeaf.add(key); + nested.push({ configPath: leaf, scopeDir: dir }); + } + } + + // Deepest scope first — file assignment takes the first program whose scope contains the file. + nested.sort((a, b) => b.scopeDir.length - a.scopeDir.length); + nested.push({ configPath: findProjectConfig(root), scopeDir: root }); // root program, always last + return nested; +} + +/** Recursively collect files literally named `tsconfig.json`, skipping vendored/build trees. */ +function walkTsconfigs(root: string): string[] { + const out: string[] = []; + const walk = (dir: string): void => { + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const e of entries) { + const abs = path.join(dir, e.name); + if (e.isDirectory()) { + if (SKIP_DIRS.has(e.name)) continue; + walk(abs); + } else if (e.isFile() && e.name === "tsconfig.json") { + out.push(abs); + } + } + }; + walk(root); + return out; +} + +/** + * Resolve one config to the LEAF configs a program should be built from. A solution-style config + * (no `files`/empty + non-empty `references`) contributes the configs it references rather than + * itself; the follow is defensive-recursive (a referenced config can itself be solution-style) + * guarded by a visited set. Test/spec configs are dropped when `skipTests`. + */ +function resolveLeafConfigs(cfg: string, skipTests: boolean, visited: Set): string[] { + const abs = path.resolve(cfg); + if (visited.has(abs) || !fs.existsSync(abs)) return []; + visited.add(abs); + + const json = readTsconfig(abs); + const refs: Array<{ path?: string }> = Array.isArray(json?.references) ? json.references : []; + const files: unknown = json?.files; + const isSolution = (!Array.isArray(files) || files.length === 0) && refs.length > 0; + + if (!isSolution) { + if (skipTests && isTestConfig(abs)) return []; + return [abs]; + } + + const out: string[] = []; + for (const ref of refs) { + if (!ref?.path) continue; + const refPath = resolveReferencePath(path.dirname(abs), ref.path); + if (!refPath) continue; + if (skipTests && isTestConfig(refPath)) continue; + out.push(...resolveLeafConfigs(refPath, skipTests, visited)); + } + return out; +} + +/** A project reference `path` may point at a config file or a directory containing `tsconfig.json`. */ +function resolveReferencePath(fromDir: string, refPath: string): string | null { + const abs = path.resolve(fromDir, refPath); + if (abs.endsWith(".json")) return fs.existsSync(abs) ? abs : null; + const asDir = path.join(abs, "tsconfig.json"); + return fs.existsSync(asDir) ? asDir : null; +} + +/** Read a (possibly jsonc, trailing-comma) tsconfig via the TS parser; `{}` on any error. */ +function readTsconfig(abs: string): { references?: unknown; files?: unknown } { + try { + const r = ts.readConfigFile(abs, (p) => fs.readFileSync(p, "utf8")); + return (r.config as { references?: unknown; files?: unknown }) ?? {}; + } catch { + return {}; + } +} + +/** A config is a test/spec config when its filename carries a `spec`/`test` marker. */ +function isTestConfig(cfg: string): boolean { + return /\b(spec|test)\b/i.test(path.basename(cfg)); +} + /** tsconfig.json → any tsconfig*.json → jsconfig.json (JS projects). */ function findProjectConfig(root: string): string | null { const direct = path.join(root, "tsconfig.json"); diff --git a/src/build/neo4j/bolt.ts b/src/build/neo4j/bolt.ts index c8c981c..2e1246b 100644 --- a/src/build/neo4j/bolt.ts +++ b/src/build/neo4j/bolt.ts @@ -11,8 +11,8 @@ * 5. on a FULL run only, prune modules whose source file vanished. * * Nodes are MERGE-upserted, never blindly deleted, so a declaration another (unchanged) module - * still references survives and its incoming edges stay valid. `:External`/`:Package`/`:Decorator` - * are shared (no `_module`) and are MERGE-only. + * still references survives and its incoming edges stay valid. `:TSExternal` nodes are shared (no + * `_module`) and are MERGE-only. * * `neo4j-driver` is imported dynamically so it stays off the hot path and out of the default * (json) output entirely. @@ -21,7 +21,7 @@ import type { Logger } from "../../utils"; import type { EdgeRow, GraphRows, NodeRow, Prop } from "./rows"; import { chunk } from "./rows"; -import { CONSTRAINTS, INDEXES } from "./schema"; +import { CONSTRAINTS, INDEXES, SCHEMA_VERSION } from "./schema"; export interface BoltConfig { uri: string; @@ -30,9 +30,37 @@ export interface BoltConfig { database: string | null; } -const DESCENDANTS = "[:DECLARES|HAS_METHOD|HAS_ATTRIBUTE|DECLARES_VAR|HAS_CALLSITE*1..]"; +const DESCENDANTS = "[:TS_DECLARES|TS_HAS_METHOD|TS_HAS_FIELD|TS_HAS_BODY_NODE*1..]"; const BATCH = 1000; +/** #68: a DB written by a different schema version must be fully re-upserted, not hash-diffed — + * otherwise unchanged modules keep the old label vocabulary while :Application advertises the new. */ +export function shouldForceFullUpsert(dbVersion: string | null, producerVersion: string): boolean { + return dbVersion !== producerVersion; +} + +/** + * #46/#68 migration: wipe the pre-2.0.0 (schema 1.x) residue when the version gate forces a full + * upsert. Pushing 2.0.0 onto a 1.1.0 DB otherwise orphans the whole 1.x subgraph AND poisons v2 + * queries — 1.x nodes carry twin TS labels (`:Module:TSModule`, `:Symbol:TSCallable`) so they match + * v2 patterns, and a second `:Application` (keyed on name, no `id`) makes the version read + * nondeterministic. These run ONCE, before the per-module loop; they are ordered, idempotent, and + * no-op on a fresh DB. + * + * They are intentionally UNANCHORED (no `:CanNode` guard on the MATCH) — that is the whole point: + * they must match the *legacy* nodes, which never carry `:CanNode`. Every v2 project-owned node DOES + * carry `:CanNode`, so the `AND NOT n:CanNode` predicate spares everything current. RETURN count so + * the caller can log what each statement removed. + */ +export const LEGACY_WIPE_STATEMENTS: readonly string[] = [ + // 1.x project-owned nodes carried `_module` under twin labels (:Module:TSModule, …) but not :CanNode. + "MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode DETACH DELETE n RETURN count(n) AS wiped", + // 1.x shared nodes (externals / packages / decorators) had no `_module`; keyed on name, not id. + "MATCH (n) WHERE (n:External OR n:Package OR n:Decorator) AND NOT n:CanNode DETACH DELETE n RETURN count(n) AS wiped", + // The 1.x :Application node was keyed on `name`, so it has no `id`. + "MATCH (a:Application) WHERE a.id IS NULL DETACH DELETE a RETURN count(a) AS wiped", +]; + export async function boltWriter( rows: GraphRows, cfg: BoltConfig, @@ -65,34 +93,75 @@ export async function boltWriter( } } - // 2. diff content_hash. + // 2. read schema version and decide if we need to force a full upsert. Scope the read to THIS + // app's :Application (by id) — an unscoped `MATCH (a:Application)` could read a foreign analyzer's + // node in a shared database and misjudge the version. Absent id → null → forces (safe default). + const appId = rows.nodes.find((n) => n.labels[0] === "Application")?.value ?? null; + let dbSchemaVersion: string | null = null; + if (appId !== null) { + await withSession(session, async (s) => { + const res = await s.run( + "MATCH (a:Application {id: $appId}) RETURN a.schema_version AS v LIMIT 1", + { appId }, + ); + dbSchemaVersion = res.records[0]?.get("v") ?? null; + }); + } + const forceAll = shouldForceFullUpsert(dbSchemaVersion, SCHEMA_VERSION); + if (forceAll) { + log.info( + `neo4j(bolt): schema ${dbSchemaVersion ?? "(none)"} → ${SCHEMA_VERSION}, full upsert forced`, + ); + // Detect-and-wipe the pre-2.0.0 subgraph in one step, BEFORE any new write, so stale twin-label + // nodes can't survive to poison v2 queries or the content-hash diff. Idempotent on fresh DBs. + const counts: number[] = []; + await withSession(session, async (s) => { + for (const stmt of LEGACY_WIPE_STATEMENTS) { + const res = await s.run(stmt); + const c = res.records[0]?.get("wiped"); + counts.push(typeof c?.toNumber === "function" ? c.toNumber() : Number(c ?? 0)); + } + }); + const wiped = counts.reduce((a, b) => a + b, 0); + if (wiped > 0) { + log.info( + `neo4j(bolt): wiped ${wiped} legacy (pre-2.0.0) nodes ` + + `(module=${counts[0]}, shared=${counts[1]}, app=${counts[2]})`, + ); + } + } + + // 3. diff content_hash. const dbHash = new Map(); await withSession(session, async (s) => { - const res = await s.run("MATCH (m:Module) RETURN m.file_key AS k, m.content_hash AS h"); + const res = await s.run("MATCH (m:TSModule) RETURN m._module AS k, m.content_hash AS h"); for (const rec of res.records) dbHash.set(rec.get("k"), rec.get("h")); }); const changed = new Set(); for (const [m, nodes] of byModule) { const rowHash = hashOf(nodes, m); - if (!dbHash.has(m) || rowHash === undefined || rowHash !== dbHash.get(m)) changed.add(m); + if (forceAll || !dbHash.has(m) || rowHash === undefined || rowHash !== dbHash.get(m)) changed.add(m); } log.info( `neo4j(bolt): ${byModule.size} modules (${changed.size} changed), ${shared.length} shared nodes, ` + `${rows.edges.length} edges`, ); - // 3. shared nodes are always upserted (MERGE-only). + // 4. shared nodes are always upserted (MERGE-only). await upsertNodes(session, neo4j, shared); - // 4. per changed module: purge owned edges + vanished decls, then upsert its nodes. + // 5. per changed module: purge owned edges + vanished decls, then upsert its nodes. for (const m of changed) { const nodes = byModule.get(m)!; const keys = nodes.map((n) => n.value); await withSession(session, async (s) => { await s.executeWrite(async (tx: any) => { - await tx.run(`MATCH (x {_module: $m})-[r]->() DELETE r`, { m }); + // Anchor on :CanNode so these seek the module's index slice instead of scanning the whole + // store (and never touch non-CanNode nodes). The `x.id IS NULL` guard defends the sweep + // against a null key — three-valued logic would otherwise drop the row from `NOT x.id IN`. + await tx.run(`MATCH (x:CanNode {_module: $m})-[r]->() DELETE r`, { m }); await tx.run( - `MATCH (x {_module: $m}) WHERE NOT coalesce(x.signature, x.id, x.file_key) IN $keys DETACH DELETE x`, + `MATCH (x:CanNode {_module: $m}) WHERE x.id IS NULL OR NOT x.id IN $keys DETACH DELETE x`, { m, keys }, ); }); @@ -100,19 +169,19 @@ export async function boltWriter( await upsertNodes(session, neo4j, nodes); } - // 5. upsert edges owned by a changed module (owner = source node's module) or shared. + // 6. upsert edges owned by a changed module (owner = source node's module) or shared. const edges = rows.edges.filter((e) => { const owner = moduleOf.get(e.from.value); return owner === undefined || changed.has(owner); }); await upsertEdges(session, neo4j, edges); - // 6. orphan prune — only safe on a full run (a targeted run can't tell deleted from untargeted). + // 7. orphan prune — only safe on a full run (a targeted run can't tell deleted from untargeted). if (fullRun) { const present = [...byModule.keys()]; await withSession(session, async (s) => { const res = await s.run( - `MATCH (m:Module) WHERE NOT m.file_key IN $present ` + + `MATCH (m:TSModule) WHERE NOT m._module IN $present ` + `OPTIONAL MATCH (m)-${DESCENDANTS}->(x) DETACH DELETE x, m RETURN count(m) AS pruned`, { present }, ); @@ -150,18 +219,21 @@ async function upsertNodes(session: () => any, neo4j: any, nodes: NodeRow[]): Pr async function upsertEdges(session: () => any, neo4j: any, edges: EdgeRow[]): Promise { const groups = new Map(); for (const e of edges) { - bucket(groups, `${e.type}|${e.from.label}.${e.from.keyProp}|${e.to.label}.${e.to.keyProp}`).push(e); + bucket(groups, `${e.type}|${e.from.label}.${e.from.keyProp}|${e.to.label}.${e.to.keyProp}|${e.key !== undefined}`).push(e); } for (const group of groups.values()) { const { type, from, to } = group[0]; + // Discriminated relationships MERGE on `{_k}` so several distinct edges of one type coexist + // between the same endpoint pair (per-var DDG, the true/false CFG_NEXT pair). See EdgeRow.key. + const relKey = group[0].key !== undefined ? " {_k: row.k}" : ""; const cypher = `UNWIND $rows AS row ` + `MATCH (a:${from.label} {${from.keyProp}: row.f}) ` + `MATCH (b:${to.label} {${to.keyProp}: row.t}) ` + - `MERGE (a)-[r:${type}]->(b) SET r += row.p`; + `MERGE (a)-[r:${type}${relKey}]->(b) SET r += row.p`; for (const batch of chunk(group, BATCH)) { - const payload = batch.map((e) => ({ f: e.from.value, t: e.to.value, p: toParams(e.props, neo4j) })); + const payload = batch.map((e) => ({ f: e.from.value, t: e.to.value, k: e.key ?? null, p: toParams(e.props, neo4j) })); await withSession(session, (s) => s.run(cypher, { rows: payload })); } } @@ -189,8 +261,9 @@ function bucket(map: Map, key: K): V[] { return arr; } -function hashOf(nodes: NodeRow[], fileKey: string): string | undefined { - const mod = nodes.find((n) => n.labels[0] === "Module" && n.value === fileKey); +function hashOf(nodes: NodeRow[], _fileKey: string): string | undefined { + // Every node in `nodes` shares the same _module; the Module row (labels include "TSModule") carries the hash. + const mod = nodes.find((n) => n.labels.includes("TSModule")); const h = mod?.props.content_hash; return typeof h === "string" ? h : undefined; } diff --git a/src/build/neo4j/cypher.ts b/src/build/neo4j/cypher.ts index aefe6f1..986d7b0 100644 --- a/src/build/neo4j/cypher.ts +++ b/src/build/neo4j/cypher.ts @@ -13,15 +13,15 @@ import { CONSTRAINTS, INDEXES } from "./schema"; const BATCH = 500; -export function renderCypher(rows: GraphRows, appName: string): string { +export function renderCypher(rows: GraphRows, appId: string): string { const out: string[] = []; out.push("// ── constraints & indexes ──"); for (const stmt of CONSTRAINTS) out.push(`${stmt};`); for (const stmt of INDEXES) out.push(`${stmt};`); - out.push("", "// ── wipe this project's prior subgraph (externals/packages/decorators are shared) ──"); - out.push(wipe(appName)); + out.push("", "// ── wipe this project's prior subgraph (external targets are shared) ──"); + out.push(wipe(appId)); out.push("", "// ── nodes ──"); for (const block of nodeStatements(rows.nodes)) out.push(block); @@ -33,12 +33,12 @@ export function renderCypher(rows: GraphRows, appName: string): string { return out.join("\n"); } -function wipe(appName: string): string { - const name = cypherValue(appName); +function wipe(appId: string): string { + const id = cypherValue(appId); return [ - `MATCH (a:Application {name: ${name}})`, - "OPTIONAL MATCH (a)-[:HAS_MODULE]->(m:Module)", - "OPTIONAL MATCH (m)-[:DECLARES|HAS_METHOD|HAS_ATTRIBUTE|DECLARES_VAR|HAS_CALLSITE*1..]->(x)", + `MATCH (a:Application {id: ${id}})`, + "OPTIONAL MATCH (a)-[:TS_HAS_MODULE]->(m:TSModule)", + "OPTIONAL MATCH (m)-[:TS_DECLARES|TS_HAS_METHOD|TS_HAS_FIELD|TS_HAS_BODY_NODE*1..]->(x)", "DETACH DELETE x, m, a;", ].join("\n"); } @@ -81,22 +81,24 @@ function nodeStatements(nodes: NodeRow[]): string[] { function edgeStatements(edges: EdgeRow[]): string[] { const groups = new Map(); for (const e of edges) { - const k = `${e.type}|${e.from.label}.${e.from.keyProp}|${e.to.label}.${e.to.keyProp}`; + const k = `${e.type}|${e.from.label}.${e.from.keyProp}|${e.to.label}.${e.to.keyProp}|${e.key !== undefined}`; (groups.get(k) ?? groups.set(k, []).get(k)!).push(e); } const blocks: string[] = []; for (const group of groups.values()) { const { type, from, to } = group[0]; + // Discriminated relationships MERGE on `{_k}` — see EdgeRow.key (issue #70). + const keyed = group[0].key !== undefined; for (const batch of chunk(group, BATCH)) { const list = batch - .map((e) => ` {f: ${cypherValue(e.from.value)}, t: ${cypherValue(e.to.value)}, p: ${cypherMap(e.props)}}`) + .map((e) => ` {f: ${cypherValue(e.from.value)}, t: ${cypherValue(e.to.value)}, ${keyed ? `k: ${cypherValue(e.key!)}, ` : ""}p: ${cypherMap(e.props)}}`) .join(",\n"); blocks.push( `UNWIND [\n${list}\n] AS row\n` + `MATCH (a:${from.label} {${from.keyProp}: row.f})\n` + `MATCH (b:${to.label} {${to.keyProp}: row.t})\n` + - `MERGE (a)-[r:${type}]->(b)\n` + + `MERGE (a)-[r:${type}${keyed ? " {_k: row.k}" : ""}]->(b)\n` + `SET r += row.p;`, ); } diff --git a/src/build/neo4j/index.ts b/src/build/neo4j/index.ts index be460ec..e94820d 100644 --- a/src/build/neo4j/index.ts +++ b/src/build/neo4j/index.ts @@ -3,6 +3,6 @@ export { project } from "./project"; export { renderCypher } from "./cypher"; export { boltWriter, type BoltConfig } from "./bolt"; -export { SCHEMA_VERSION, TS_PREFIX, twinOf, withTwins, buildSchemaDocument, NODE_LABELS, REL_TYPES, MARKER_LABELS } from "./schema"; +export { SCHEMA_VERSION, TS_PREFIX, buildSchemaDocument, NODE_LABELS, REL_TYPES, MARKER_LABELS, CONSTRAINTS, INDEXES } from "./schema"; export type { SchemaDocument } from "./schema"; export type { GraphRows, NodeRow, EdgeRow } from "./rows"; diff --git a/src/build/neo4j/project.ts b/src/build/neo4j/project.ts index d112d3d..5120636 100644 Binary files a/src/build/neo4j/project.ts and b/src/build/neo4j/project.ts differ diff --git a/src/build/neo4j/rows.ts b/src/build/neo4j/rows.ts index 73d085a..35d2107 100644 Binary files a/src/build/neo4j/rows.ts and b/src/build/neo4j/rows.ts differ diff --git a/src/build/neo4j/schema.ts b/src/build/neo4j/schema.ts index 76a5ce5..a25165b 100644 --- a/src/build/neo4j/schema.ts +++ b/src/build/neo4j/schema.ts @@ -1,26 +1,32 @@ /** - * The declarative Neo4j schema — the single in-repo source of truth for the graph contract: node - * labels with their keys and typed properties, relationship types and their endpoints, and the - * Cypher DDL (uniqueness constraints + indexes). The constraints are DERIVED from the node labels - * (one per distinct mergeLabel/key) so a new label brings its own constraint — there is no second - * list to keep in sync. `--emit schema` serializes all of this to a machine-readable schema.json, - * and the conformance test (test/neo4j-schema.test.ts) asserts the real emitter never produces a - * label / relationship / property that isn't declared here — so this file cannot silently drift - * from project.ts. + * The declarative Neo4j schema (schema v2) — the single in-repo source of truth for the graph + * contract: node labels with their keys and typed properties, relationship types and their + * endpoints, and the Cypher DDL (uniqueness constraints + indexes). Constraints are DERIVED from + * the node labels (one per distinct mergeLabel/key). `--emit schema` serializes all of this to + * schema.neo4j.json, and the conformance test (test/neo4j-schema.test.ts) asserts the emitter + * never produces an undeclared label / relationship / property. * - * SCHEMA_VERSION is the contract version: bump MAJOR on a breaking change (renamed/removed label, - * relationship or key), MINOR on an additive change (new label/rel/property). It is stamped onto - * the :Application node of every emitted graph so any consumer can detect a producer/consumer - * mismatch at runtime. + * Schema v2 mirrors the additive-CPG JSON (canonical-schema.md): every tree/body node is a graph + * node keyed on its `can://` id under the shared MERGE label `CanNode` (one uniqueness constraint, + * uniform edge endpoints), carrying a specific kind label; containment renders as HAS_x / DECLARES + * edges and every typed overlay (call_graph, cfg/cdg/ddg/summary, param_in/param_out) as a typed + * relationship. At 2.0.0 (issue #66) the vocabulary is namespaced for TypeScript: every specific + * node label is `TS`-prefixed (`TSModule`, `TSCallable`, ...; the shared MERGE labels `CanNode` and + * `Application` stay bare) and every relationship type is `TS_`-prefixed (`TS_CALLS`, ...) — this + * lets a shared multi-language database attribute TS-origin graph elements (epic #64) without a + * transitional dual-labeling step. + * + * SCHEMA_VERSION: MAJOR on a breaking change (renamed/removed label, relationship or key), MINOR + * on additive. v2 is a MAJOR bump from v1 (keys moved signature→can:// id; labels reshaped). */ -export const SCHEMA_VERSION = "1.1.0"; +export const SCHEMA_VERSION = "2.0.0"; export type PropType = "string" | "integer" | "float" | "boolean" | "string[]" | "integer[]"; export interface NodeLabel { /** The specific label (also the key in NODE_LABELS). */ label: string; - /** The label the uniqueness constraint / MERGE is on (`Symbol` for signature-keyed nodes). */ + /** The label the uniqueness constraint / MERGE is on (`CanNode` for can://-id-keyed nodes). */ mergeLabel: string; key: string; properties: Record; @@ -34,312 +40,136 @@ export interface RelType { } /** Labels layered onto a node in addition to its primary/specific label. */ -export const MARKER_LABELS = ["Entrypoint"] as const; +export const MARKER_LABELS = [] as const; -/** - * Language-namespace twins (schema 1.1.0, additive): every specific and marker label is also - * stamped as `TS