fix: schema migrations for titles/loi_date + person resolver version bump (patches PR #189)#190
Closed
sroussey wants to merge 4 commits into
Closed
fix: schema migrations for titles/loi_date + person resolver version bump (patches PR #189)#190sroussey wants to merge 4 commits into
sroussey wants to merge 4 commits into
Conversation
PersonObservationSchema renamed title (string|null) → titles (string[]|null) in PR #189, but SqliteTabularStorage.setupDatabase uses CREATE TABLE IF NOT EXISTS — a no-op on existing tables. Any existing edgar.db crashes on the first observation write with 'table has no column named titles'. Add a dual-backend ALTER TABLE + backfill migration following the Form8KEventLegacyMigration pattern. Backfills existing title values as json_array(title); null and empty preserved as null. Legacy title column is left in place (no DROP COLUMN) for backward compat. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF
PR #189 added loi_date to spac, spac_deal, spac_history without an ALTER TABLE migrator. Existing SPAC-tracking sec.db files crash on first SpacReportWriter.rebuild after upgrade. Add dual-backend guarded ADD COLUMN for all three tables. spac.status gained enum value 'loi' but TypeStringEnum generates plain TEXT with no CHECK constraint, so no additional DDL is required for the status column. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF
… change PersonNormalization.stripNamePartPunctuation was extended to strip periods/commas and fold typographic apostrophes to ASCII. That changes the tuple written to normalized_first/middle/last/suffix, which PersonResolver.personKey looks up by exact match scoped by resolver_version. bootstrapComponentVersions kept person at 1.0.0, so post-fold observations miss pre-fold canonicals and mint duplicates under the same resolver_version — silent identity duplication. Enable the ceremony: - version.ts snapshotTargetCount now supports kind='resolver' (person → PersonObservationRepo.count(), company → CompanyObservationRepo.count(); family-tier refused). - ceremonies.ts promote coverage gate now dispatches by kind (extractor → existing gate; resolver → computeResolverCoverage; --force bypasses). Operator ceremony to migrate existing DBs is documented in CHANGELOG and CLAUDE.md. Aliases must be re-applied manually against the fresh 2.0.0 canonical UUIDs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF
CHANGELOG entry + CLAUDE.md worked example. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF
Contributor
Author
|
Closing — sroussey clarified that there's no production sec data yet, so:
The Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Patches three high-severity issues that landed with PR #189. Stacked on
harness.H1 —
titlescolumn migration forperson_observationsPersonObservationSchemarenamedtitle(string | null) →titles(string[] | null), butSqliteTabularStorage.setupDatabase()usesCREATE TABLE IF NOT EXISTS— a no-op on existing tables. Any existingedgar.dbcrashes on the first observation write withtable has no column named titles.src/storage/observation/PersonObservationTitlesMigration.ts— dual-backend guardedALTER TABLE ... ADD COLUMN titles+ backfill:titles = json_array(title)(SQLite) /jsonb_build_array(title)(Postgres) whentitleis non-null and non-empty; null/empty preserved as null.titlescolumn presence and no-ops when already migrated; no-op on a fresh DB with no table; in-memory (test) backend silently no-ops.titlecolumn is left in place for backward compat.setupAllDatabases.tsbeforePersonObservationRepo.setupDatabase().H2 —
loi_datecolumns migration for SPAC tablesSame CREATE TABLE IF NOT EXISTS problem — PR #189 added
loi_datetospac,spac_deal,spac_historywithout an ALTER TABLE migrator. Existing SPAC-tracking DBs crash on the firstSpacReportWriter.rebuildafter upgrade.src/storage/spac/SpacLoiColumnsMigration.ts— dual-backend guardedADD COLUMN loi_datefor all three tables in a single transaction.spac.status = "loi"needs no DDL —TypeStringEnumgenerates plain TEXT with no CHECK constraint (confirmed insrc/util/TypeBoxUtil.ts); the enum is validated at the schema layer, not the DB.setupAllDatabases.tsbefore the three SPACsetupDatabase()calls.H3 — Person resolver version bump (1.0.0 → 2.0.0)
PersonNormalization.stripNamePartPunctuationwas extended to strip periods/commas and fold typographic apostrophes to ASCII. That changes the(normalized_first, normalized_middle, normalized_last, normalized_suffix)tuplePersonResolver.personKeylooks up by exact match, scoped byresolver_version.bootstrapComponentVersionskept person at 1.0.0, so post-fold observations under the sameresolver_versionsilently miss pre-fold canonicals and mint duplicates — silent identity duplication.The code fix is the missing ceremony plumbing so the operator can bump:
src/cli/groups/version.ts—snapshotTargetCountnow supportskind === "resolver"(person →PersonObservationRepo.count(), company →CompanyObservationRepo.count(); family-tier kinds are refused, matchingcomputeResolverCoverage).src/storage/versioning/ceremonies.ts—promote's major-bump coverage gate dispatches bykind: extractor uses the existingExtractorRunRepo.countSuccessfulAtVersiongate; resolver usescomputeResolverCoverage(...)and rejects whenfraction < 1.0unless--force.Operator ceremony (must run on existing DBs)
Aliases must be re-applied manually against the fresh 2.0.0 canonical UUIDs — dump before, re-issue after.
Documented in
CHANGELOG.md(new Unreleased section) andCLAUDE.md(new Resolver version bumps subsection under PR4 CLI additions).Test plan
bun test src/storage/observation/PersonObservationTitlesMigration.test.ts— 4 pass (backfill, idempotency, fresh-DB no-op, repo end-to-end withtitles: ["CEO"])bun test src/storage/spac/SpacLoiColumnsMigration.test.ts— 4 pass (adds column to all three tables, idempotent, fresh-DB no-op,SpacRepo.saveSpac({status: "loi", loi_date: "2026-01-15", ...})round-trips)bun test src/config/— 13 passbun test src/storage/person/PersonNormalization.test.ts— 24 pass (incl. newit.eachgolden pin on the 2.0.0 tuple)bun test src/resolver/— 68 pass (incl. new cross-version isolation + 2.0.0 idempotency tests on PersonResolver)bun test src/storage/versioning/— 109 pass (incl. new bootstrap → startDev(resolver:person, 2.0.0, major) → promote(force) rotation test, and three new resolver-coverage-gate cases in ceremonies)bun test src/cli/groups/version.test.ts— 12 pass (incl. newstart-dev resolver person 2.0.0 --bump majorcompletes and snapshots observation count as denominator)Total: 232 tests across the touched surface, all passing. No pre-existing failures encountered.
References
src/storage/form-8k-event/Form8KEventLegacyMigration.ts🤖 Generated with Claude Code
https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF
Generated by Claude Code