Merge/evidence v2 into main

docs/pipelines/association.md

@@ -342,3 +342,12 @@ The point of the publication stage is not just file creation. It is the point wh
 ## Current additive metadata path
 
 The association export manifest framework adds reserved `_datahub` metadata blocks to published outputs. These blocks are additive and allow the project to carry forward provenance and future coverage fields without breaking existing payload consumers.
+
+## Association evidence v2 sidecar
+
+The release-candidate v2 model separates exact provider rows, source association
+observations, canonical variants, consequence annotations, clinical assertions,
+population observations, and variant-phenotype summaries. It replaces the
+legacy representative-row annotation contract only when the backend is explicitly
+configured for v2. See [Association Evidence v2](../schemas/association_evidence_v2.md)
+for counting, source completeness, build, migration, and rollback semantics.

docs/reference/scripts.md

@@ -620,3 +620,39 @@ DuckDB before the current points-table ingest/publish stages run.
 ## Script philosophy
 
 The scripts directory is intentionally operational. Business/scientific logic lives in `src/datahub/` when it can. Scripts compose that logic and add environment/runtime concerns such as CLI parsing, checkpoints, and scheduler integration.
+
+## Association evidence v2
+
+### `scripts/dataset_specific_scripts/unified/build_association_evidence_v2.py`
+
+Builds the resumable non-composite association sidecar from archived CVD/trait
+provider rows and registers retained variant-index artifacts for gene-scoped runtime
+access when provider detail is unavailable. It writes only to explicit
+output, checkpoint, manifest, temporary, and log paths. Use `--dry-run` to inspect
+the plan and `--checksum-inputs` only when a second full read of source files is
+acceptable. `--provider-chunk-rows 0` uses direct inserts for inputs up to 2 GB
+and bounded one-million-row chunks for oversized inputs; set a positive value to
+force bounded chunks for every provider file.
+
+### `scripts/dataset_specific_scripts/unified/build_association_evidence_v2_serving.py`
+
+Publishes the checksum-linked, bounded Parquet serving package from a verified normalized sidecar. The production release candidate uses 4,096 variant buckets for exact variant evidence and gene-keyed projections for summary requests.
+
+### `scripts/dataset_specific_scripts/unified/build_association_evidence_v2_source_summary_index.py`
+
+Streams every registered MVP compact artifact into a resumable gene-keyed Parquet association index. It emits deterministic association-record IDs, record_kind=source_summary_association, evidence_granularity=source_summary, exact phenotype paths, reported p-values, retained compact fields, and immutable artifact provenance. It never fabricates provider or study records.
+
+### build_association_evidence_v2_source_summary_rollup.py
+
+Builds resumable gene-keyed default-summary and exact-phenotype projections from
+the completed source-summary index. It records exact phenotype variant-ID sets so
+provider-level legacy and source-summary MVP evidence can be unioned without duplicate
+inflation. It advances the serving manifest to schema **2.10.0-rc2**; it does not
+replace the source-summary index or create inferred provider records.
+
+### `scripts/dataset_specific_scripts/unified/audit_association_evidence_v2_release.py`
+
+Reproduces portal-wide v1/v2 count and category comparisons using the normalized sidecar plus distinct compact-source membership, with a resumable runtime checkpoint.
+
+See the [v2 schema](../schemas/association_evidence_v2.md) for the full
+command and scientific contract.

docs/runbooks/association-evidence-v2-release.md

@@ -0,0 +1,82 @@
+# Association Evidence v2 Release Candidate
+
+Use this runbook only after the staged sidecar, backend, and frontend have passed
+scientific, reconciliation, performance, and browser gates.
+
+## Pre-deployment gates
+
+- DataHub manifest and output SHA-256 verify.
+- Every DataHub QA check passes.
+- The manifest records the exact `--provider-chunk-rows` policy used for oversized sources.
+- Source-completeness statuses match the archived inputs.
+- The complete gene-keyed MVP source-summary association index, file manifest, and checksums verify; compact rows are first-class association evidence and no provider or study rows are fabricated.
+- The schema 2.10 default-summary rollups, exact phenotype variant-ID sets, file manifest, and checksums verify against the complete source-summary index.
+- Five bounded multiplicity fixtures reconcile provider rows, source
+  observations, annotations, assertions, summaries, and exports.
+- All v1/v2 category differences are classified and reviewed.
+- The full comparison audit reuses the verified schema 2.10 distinct-membership rollup; it does not rescan compact source artifacts.
+- Summary, detail, selected-rsID drill-down, and ZIP export use the same filter scope.
+- Default search and explicit p-value searches pass performance limits.
+- Browser tests pass for HMGCR and at least one high-volume gene.
+- A timestamped environment/configuration backup and exact v1 rollback command exist.
+
+## Deployment shape
+
+Evidence v2 is an immutable sidecar selected by environment:
+
+```text
+HBP_ASSOC_EVIDENCE_MODEL=v2
+HBP_ASSOC_EVIDENCE_V2_PATH=/data/DataHub/releases/association-evidence-v2/RELEASE_ID/association_evidence_v2.duckdb
+HBP_ASSOC_EVIDENCE_V2_SERVING_MANIFEST=/data/DataHub/releases/association-evidence-v2/RELEASE_ID/serving/serving-manifest.json
+HBP_ASSOC_ARTIFACT_ROOT=/data/DataHub/analyzed_data/association_new/final
+```
+
+Do not overwrite `association_serving_slim.duckdb`. Keep the prior backend
+environment and v1 artifacts available until post-deployment validation completes.
+The backend opens the normalized sidecar and serving package read-only. It resolves first-class MVP source-summary associations from the staged gene-keyed index and retains `HBP_ASSOC_ARTIFACT_ROOT` only as a pre-release fallback when that index is absent. Production v2 validation requires the complete index; request-time parsing of large compact JSON artifacts is not an accepted deployment state. The backend retains an explicit v1 fallback. MVP records use evidence_granularity=source_summary and provider_detail_status=not_applicable; provider and study rows are never fabricated.
+
+Build the release-candidate serving tree with
+`--variant-bucket-characters 3` and a checksum-verified
+`--coarse-serving-root`. Provider and association records then use
+4,096 bounded SHA-256 prefix buckets; the table-specific width is recorded in
+the serving manifest and read by the backend. This changes only physical query
+partitioning, not normalized evidence, counts, filters, or provenance.
+
+After the complete source-summary index verifies, run **build_association_evidence_v2_source_summary_rollup.py**. Production validation requires **schema_version=2.10.0-rc2**. The backend may use these rollups only for an unfiltered default summary and exact CVD phenotype counting. Any phenotype-path or explicit p-value filter, drill-down, export, or provenance request continues to use the full indexed rows.
+
+## Approved migration sequence
+
+1. Verify the staged SHA-256 manifest and DataHub QA report without opening the file
+   for writing.
+2. Copy the immutable sidecar, complete serving partition tree, and serving manifest
+   into a new release-ID directory; never reuse an existing release path. Preserve or
+   dereference hard links deliberately during the copy and verify every copied
+   checksum.
+3. Save a timestamped copy of the backend service environment and current v1/v2
+   selector values. Record the deployed backend/frontend commits and build checksum.
+4. Set the two environment variables above and restart only the backend during an
+   approved maintenance window. No DuckDB import or schema migration is performed.
+5. Require exact HMGCR, PCSK9, LDLR, TTN, and ANK2 summary/detail/export
+   reconciliation, source-completeness status, API latency, browser, CAPTCHA export,
+   and log-health gates before declaring the release live.
+6. Preserve the staged release, copied release, v1 serving DB, previous environment,
+   and all validation reports for the rollback window.
+
+## Rollback
+
+Restore the timestamped prior environment or set:
+
+```text
+HBP_ASSOC_EVIDENCE_MODEL=v1
+```
+
+Then restart only the backend in an approved window and repeat the recorded v1
+health/search/download checks. The v2 sidecar can remain on disk because v1 mode does
+not open it. No DataHub artifact deletion, reverse transformation, or database
+conversion is required. If rollback validation fails, keep the service stopped and
+restore the complete timestamped service environment rather than editing data.
+
+The application default remains v1. A deployment enters v2 only when the selector,
+normalized sidecar, and serving manifest are all explicitly configured. Missing or
+invalid serving partitions fail the v2 validation gate; they are not replaced by a
+request-time global scan.