Skip to content

fix: schema migrations for titles/loi_date + person resolver version bump (patches PR #189)#190

Closed
sroussey wants to merge 4 commits into
harnessfrom
claude/wonderful-hypatia-8v3snc
Closed

fix: schema migrations for titles/loi_date + person resolver version bump (patches PR #189)#190
sroussey wants to merge 4 commits into
harnessfrom
claude/wonderful-hypatia-8v3snc

Conversation

@sroussey

Copy link
Copy Markdown
Contributor

Patches three high-severity issues that landed with PR #189. Stacked on harness.

H1 — titles column migration for person_observations

PersonObservationSchema renamed title (string | null) → titles (string[] | null), but SqliteTabularStorage.setupDatabase() uses CREATE TABLE IF NOT EXISTS — a no-op on existing tables. Any existing edgar.db crashes on the first observation write with table has no column named titles.

  • New src/storage/observation/PersonObservationTitlesMigration.ts — dual-backend guarded ALTER TABLE ... ADD COLUMN titles + backfill: titles = json_array(title) (SQLite) / jsonb_build_array(title) (Postgres) when title is non-null and non-empty; null/empty preserved as null.
  • Idempotent — probes for titles column presence and no-ops when already migrated; no-op on a fresh DB with no table; in-memory (test) backend silently no-ops.
  • Legacy title column is left in place for backward compat.
  • Wired in setupAllDatabases.ts before PersonObservationRepo.setupDatabase().

H2 — loi_date columns migration for SPAC tables

Same CREATE TABLE IF NOT EXISTS problem — PR #189 added loi_date to spac, spac_deal, spac_history without an ALTER TABLE migrator. Existing SPAC-tracking DBs crash on the first SpacReportWriter.rebuild after upgrade.

  • New src/storage/spac/SpacLoiColumnsMigration.ts — dual-backend guarded ADD COLUMN loi_date for all three tables in a single transaction.
  • spac.status = "loi" needs no DDLTypeStringEnum generates plain TEXT with no CHECK constraint (confirmed in src/util/TypeBoxUtil.ts); the enum is validated at the schema layer, not the DB.
  • Wired in setupAllDatabases.ts before the three SPAC setupDatabase() calls.

H3 — Person resolver version bump (1.0.0 → 2.0.0)

PersonNormalization.stripNamePartPunctuation was extended to strip periods/commas and fold typographic apostrophes to ASCII. That changes the (normalized_first, normalized_middle, normalized_last, normalized_suffix) tuple PersonResolver.personKey looks up by exact match, scoped by resolver_version. bootstrapComponentVersions kept person at 1.0.0, so post-fold observations under the same resolver_version silently miss pre-fold canonicals and mint duplicates — silent identity duplication.

The code fix is the missing ceremony plumbing so the operator can bump:

  • src/cli/groups/version.tssnapshotTargetCount now supports kind === "resolver" (person → PersonObservationRepo.count(), company → CompanyObservationRepo.count(); family-tier kinds are refused, matching computeResolverCoverage).
  • src/storage/versioning/ceremonies.tspromote's major-bump coverage gate dispatches by kind: extractor uses the existing ExtractorRunRepo.countSuccessfulAtVersion gate; resolver uses computeResolverCoverage(...) and rejects when fraction < 1.0 unless --force.

Operator ceremony (must run on existing DBs)

Aliases must be re-applied manually against the fresh 2.0.0 canonical UUIDs — dump before, re-issue after.

sec canonical person alias-list                 # dump for re-apply
sec version start-dev resolver person 2.0.0 --bump major \
  --notes "PersonNormalization punctuation/typographic fold changes key tuple"
sec resolve --kind person --resolver-version 2.0.0 --all
sec version coverage resolver person
sec version promote resolver person
sec version drop-previous resolver person
# now re-issue `sec canonical person alias ...` for each entry from the dump

Documented in CHANGELOG.md (new Unreleased section) and CLAUDE.md (new Resolver version bumps subsection under PR4 CLI additions).

Test plan

  • bun test src/storage/observation/PersonObservationTitlesMigration.test.ts — 4 pass (backfill, idempotency, fresh-DB no-op, repo end-to-end with titles: ["CEO"])
  • bun test src/storage/spac/SpacLoiColumnsMigration.test.ts — 4 pass (adds column to all three tables, idempotent, fresh-DB no-op, SpacRepo.saveSpac({status: "loi", loi_date: "2026-01-15", ...}) round-trips)
  • bun test src/config/ — 13 pass
  • bun test src/storage/person/PersonNormalization.test.ts — 24 pass (incl. new it.each golden pin on the 2.0.0 tuple)
  • bun test src/resolver/ — 68 pass (incl. new cross-version isolation + 2.0.0 idempotency tests on PersonResolver)
  • bun test src/storage/versioning/ — 109 pass (incl. new bootstrap → startDev(resolver:person, 2.0.0, major) → promote(force) rotation test, and three new resolver-coverage-gate cases in ceremonies)
  • bun test src/cli/groups/version.test.ts — 12 pass (incl. new start-dev resolver person 2.0.0 --bump major completes and snapshots observation count as denominator)

Total: 232 tests across the touched surface, all passing. No pre-existing failures encountered.

References

🤖 Generated with Claude Code

https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF


Generated by Claude Code

claude added 4 commits July 15, 2026 08:34
PersonObservationSchema renamed title (string|null) → titles (string[]|null)
in PR #189, but SqliteTabularStorage.setupDatabase uses CREATE TABLE IF NOT
EXISTS — a no-op on existing tables. Any existing edgar.db crashes on the
first observation write with 'table has no column named titles'.

Add a dual-backend ALTER TABLE + backfill migration following the
Form8KEventLegacyMigration pattern. Backfills existing title values as
json_array(title); null and empty preserved as null. Legacy title column
is left in place (no DROP COLUMN) for backward compat.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF
PR #189 added loi_date to spac, spac_deal, spac_history without an ALTER
TABLE migrator. Existing SPAC-tracking sec.db files crash on first
SpacReportWriter.rebuild after upgrade.

Add dual-backend guarded ADD COLUMN for all three tables. spac.status
gained enum value 'loi' but TypeStringEnum generates plain TEXT with no
CHECK constraint, so no additional DDL is required for the status column.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF
… change

PersonNormalization.stripNamePartPunctuation was extended to strip
periods/commas and fold typographic apostrophes to ASCII. That changes
the tuple written to normalized_first/middle/last/suffix, which
PersonResolver.personKey looks up by exact match scoped by
resolver_version. bootstrapComponentVersions kept person at 1.0.0, so
post-fold observations miss pre-fold canonicals and mint duplicates
under the same resolver_version — silent identity duplication.

Enable the ceremony:
- version.ts snapshotTargetCount now supports kind='resolver' (person →
  PersonObservationRepo.count(), company → CompanyObservationRepo.count();
  family-tier refused).
- ceremonies.ts promote coverage gate now dispatches by kind (extractor
  → existing gate; resolver → computeResolverCoverage; --force
  bypasses).

Operator ceremony to migrate existing DBs is documented in CHANGELOG
and CLAUDE.md. Aliases must be re-applied manually against the fresh
2.0.0 canonical UUIDs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF
CHANGELOG entry + CLAUDE.md worked example.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF

Copy link
Copy Markdown
Contributor Author

Closing — sroussey clarified that there's no production sec data yet, so:

  • The titletitles and loi_date ALTER TABLE migrations aren't needed (fresh CREATE TABLE will produce the new schema; no title-columned rows exist to backfill).
  • The person resolver 1.0.0 → 2.0.0 bump isn't needed (no pre-fold canonical rows to preserve; the punctuation-fold change lands under 1.0.0 with no duplication risk).

The snapshotTargetCount / promote coverage-gate enablers for kind === "resolver" are still useful groundwork but not urgent — happy to re-open as a smaller PR if wanted. Deferred medium findings from the original review (S-1 / D extractor 1.0.0 output-shape drift, backfill descriptor version-gating, ReadOnlyTabularStorage.updateWhere no-op) remain in the log for future tracking.


Generated by Claude Code

@sroussey sroussey closed this Jul 15, 2026
@sroussey
sroussey deleted the claude/wonderful-hypatia-8v3snc branch July 15, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants