Add accredited-investor portal attribution for Form D filings#197
Merged
Conversation
sroussey
force-pushed
the
claude/sec-accredited-investor-portals-vtsw65
branch
from
July 16, 2026 02:47
8dc1b84 to
4d27a2b
Compare
…ribution Accredited-investor portals (AngelList, Forge, EquityZen, ...) never register with the SEC, so a curated tier bootstraps from an embedded copy of the embarc portals-accredited list: accredited_portal (slug-keyed), plus accredited_portal_signal fingerprints (normalized entity names, phone international numbers, address hash ids) and derived form_d_portal_attribution rows. processFormD harvests issuer / related-person / sales-compensation candidates and PortalAttributor matches them exactly against the signal table (address > phone > name); 'sec accredited-portal attribute' recomputes attributions from stored observations with clear-then-recompute semantics. New 'sec accredited-portal' CLI group: import, list, signal add/list/remove, attribute, filings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS
Correctness: unscoped attribution now clears the filing's prior rows before recomputing (re-ingest replays self-heal after signal changes); the backfill filters bad-data placeholder tokens out of person name parts exactly like the ingest path (shared pushAttributionCandidates builder ends the divergence); the matches audit trail keeps every corroborating filing role and the strongest-match tie-break is deterministic. Efficiency: signal lookups batch through getBulk; clearPortal/clearAccession use deleteSearch instead of row-by-row deletes; the backfill streams observations with keyset pagination instead of materializing both tables. Cleanup: dead signal-type const objects removed; seed input type reuses the embedded seed entry; CLI --type is case-insensitive; shared option block for signal add/remove; clearer seed-file JSON error with the path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS
The backfill clears its whole recompute scope up front (clearAll / clearPortal), so the attributor's per-accession clear — needed only for the live-ingest replay path — was dead work repeated once per filing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS
sroussey
force-pushed
the
claude/sec-accredited-investor-portals-vtsw65
branch
from
July 16, 2026 03:16
4d27a2b to
7f218b9
Compare
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS
- Look up signals before clearing a filing's attribution rows and serialize the clear+write per accession (KeyedMutex), so a transient failure or a concurrent replay can no longer leave a filing stripped of rows. - Centralize the signature-role exclusion in pushAttributionCandidates and reject placeholder tokens (None, N/A, ...) inside normalizeNameSignal, so ingest, backfill, and CLI harvest identically by construction (with an idempotency test pinning normalize-twice parity). - Backfill: preload the signal table once (all or per portal), stream observations via the shared streamMatchingRows helper, and keep only candidates that match a loaded signal — memory is bounded by match count and unmatched accessions skip the filing lookup entirely. - CLI: address signals no longer inherit --country (ingest never has one), and --value for addresses must be a normalized pipe-joined hash. - Import: reject string live values in external seeds and report distinct portal count; drop the unused deletePortal method. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS
5 tasks
The CLI's machine-readable convention is the global --json flag (SEC_JSON_OUTPUT / isJsonOutput), not a per-command --format option; list and filings now follow it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an accredited-investor portal attribution subsystem for Form D filings by introducing curated portal + fingerprint (signal) storage, deterministic attribution writing, and CLI tooling to seed/curate/backfill attributions. This fits into the SEC ingest + storage layer by producing recomputable derived attribution rows from existing observations.
Changes:
- Introduces curated accredited-portal storage (portals, signals, Form D attributions) with normalization helpers and import-from-seed support.
- Adds a deterministic attribution engine (
PortalAttributor) plus an observation-based backfill to recompute attributions without raw filings. - Integrates attribution harvesting into the Form D ingest path and adds CLI commands + documentation/tests.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/storage/accredited-portal/SignalNormalization.ts | Normalizers for name/phone/address signals aligned with ingest output. |
| src/storage/accredited-portal/importAccreditedPortals.ts | Seed import (embedded or external JSON) with idempotent upserts and seed-signal refresh. |
| src/storage/accredited-portal/importAccreditedPortals.test.ts | Tests for seed import behavior and idempotency. |
| src/storage/accredited-portal/FormDPortalAttributionSchema.ts | TypeBox schema + DI token for derived Form D portal attributions. |
| src/storage/accredited-portal/FormDPortalAttributionRepo.ts | Repository wrapper for attribution CRUD and clearing scopes. |
| src/storage/accredited-portal/AccreditedPortalSignalSchema.ts | Signal schema keyed by (type,value) with seed/manual source tracking. |
| src/storage/accredited-portal/AccreditedPortalSignalRepo.ts | Signal repo helpers including “seed doesn’t overwrite manual” semantics. |
| src/storage/accredited-portal/AccreditedPortalSignalRepo.test.ts | Tests for normalization and signal repo behavior. |
| src/storage/accredited-portal/AccreditedPortalSchema.ts | Accredited portal schema + portal_id slug derivation helper. |
| src/storage/accredited-portal/AccreditedPortalRepo.ts | Portal repo helpers including seed upsert preserving curated fields. |
| src/storage/accredited-portal/AccreditedPortalRepo.test.ts | Tests for portal repo operations and slug generation. |
| src/sec/forms/exempt-offerings/Form_D.storage.ts | Harvests attribution candidates during Form D ingest and safely invokes attribution. |
| src/sec/forms/exempt-offerings/Form_D.attribution.test.ts | Integration tests verifying ingest-time attribution writes. |
| src/resolver/PortalAttributor.ts | Deterministic matching engine with scoped/unscoped recompute semantics and locking. |
| src/resolver/PortalAttributor.test.ts | Tests for attribution behavior, tie-breaking, deduping, and recompute behavior. |
| src/resolver/backfillFormDAttribution.ts | Backfill sweep recomputing attributions from stored observations + signals. |
| src/resolver/backfillFormDAttribution.test.ts | Tests for backfill correctness, clearing semantics, and scoping. |
| src/data/accreditedPortalsSeed.ts | Embedded seed list of accredited-investor portals. |
| src/config/TestingDI.ts | Registers in-memory storages for new accredited-portal tables in tests. |
| src/config/setupAllDatabases.ts | Ensures new storages are set up in integration runs. |
| src/config/resetAllDatabases.ts | Ensures new storages are cleared on resets. |
| src/config/DefaultDI.ts | Registers persistent storages for the new accredited-portal tables. |
| src/commands/index.ts | Wires accredited-portal commands into the CLI. |
| src/commands/accreditedPortal.ts | CLI for importing portals, curating signals, backfilling attribution, listing filings. |
| README.md | Documents accredited-investor portals and associated CLI workflows. |
| CLAUDE.md | Adds architecture + operational notes for the attribution system. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+26
to
+29
| matched_signal_type: Type.String({ | ||
| maxLength: 16, | ||
| description: "Strongest matching signal kind (address > phone > name)", | ||
| }), |
| ); | ||
| return null; | ||
| } | ||
| return { signal_type: "address", signal_value: opts.value.toLowerCase() }; |
| } | ||
| for (const p of portals) { | ||
| console.log( | ||
| `${p.portal_id}\t${p.name}\t${p.live ? "live" : "closed"}\t${p.url ?? ""}\t${p.brand ?? ""}` |
Comment on lines
+253
to
+256
| if (!opts.all && !opts.portal) { | ||
| fail("pass --all to recompute everything, or --portal <id> for one portal"); | ||
| return; | ||
| } |
Comment on lines
+389
to
+392
| // A related person's address is a portal fingerprint even when the person is | ||
| // a genuine individual (fund managers list the portal's back office). Their | ||
| // human name can't collide with normalized portal name signals, so it is | ||
| // safe to offer as a candidate too. |
- Trim pasted address_hash_id values before storing - Constrain matched_signal_type to the three signal-kind literals - Make attribute's --all and --portal mutually exclusive - Render null portal live status as 'unknown', not 'closed' - Soften an overclaiming comment about person-name collisions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS
…ration hardening - Register portal-attributor as a resolver component: rows stamp the active slot's semver, 'sec version coverage resolver portal-attributor' reports the share of attribution rows at a version, and drop-previous purges rows at the retired version - Add 'sec accredited-portal suggest': surfaces address/phone values recurring across many distinct Form D filings that are not yet curated signals - Add 'sec accredited-portal set' for curated portal fields (cik, notes) - Require explicit --country for phone signals (region-sensitive parsing) - Widen address-hash columns (64 -> 512) and phone columns (20 -> 32) in the observation and canonical-junction schemas so Postgres can store full address_hash_id values (schema-only; no deployed data) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a curated accredited-investor portal system that matches Form D filings to known platforms (AngelList, Forge, EquityZen, etc.) by fingerprinting their reused entity names, phone numbers, and addresses. Portals are bootstrapped from an embedded seed (derived from the embarc repo) and maintained via CLI commands; attributions are computed deterministically from stored observations and are fully recomputable when portal signals change.
Key Changes
Portal curation infrastructure: New
accredited_portal,accredited_portal_signal, andform_d_portal_attributiontables with repos and schemas. Portals are keyed by a name-derived slug (portal_id) rather than a CIK since they never register with the SEC.Signal normalization: Shared normalizers (
SignalNormalization.ts) that delegate to existing ingest-path helpers (company name, phone, address) so stored signals match filing values by exact string equality. Signals are keyed by(type, value)and re-point on re-add.Attribution engine (
PortalAttributor.ts): Matches filing candidates against the signal table with deterministic signal-strength ordering (address > phone > name) to break ties. Writes one row per matched portal per filing, recording all corroborating roles and the strongest matched signal type.Form D ingest integration: Modified
Form_D.storage.tsto harvest name/phone/address fingerprints from issuer and related-person roles viacollectAttributionSignals, feeding them to the attributor after processing. Signatures are deliberately skipped (signers are individuals and add noise).Observation backfill (
backfillFormDAttribution.ts): Recomputes attributions from stored observations (extractor_id "D") so re-attribution after signal changes never needs raw filings. Clears the affected scope (one portal or the whole table) first so removed signals drop stale attributions. Mirrors ingest normalization exactly (drops bad-data placeholders, skips signatures).CLI commands (
accreditedPortal.ts):sec accredited-portal import [file]— bootstrap/refresh from embedded seed or external JSONsec accredited-portal signal add/remove— curate portal fingerprintssec accredited-portal signal list— inspect stored signalssec accredited-portal backfill— recompute attributions after signal changesEmbedded seed data (
accreditedPortalsSeed.ts): 30+ curated portals (AngelList, Forge, EquityZen, Hiive, OurCrowd, Republic, etc.) with metadata (URL, live status, featured flag).Tests: Comprehensive coverage of signal normalization, portal/signal repos, attribution matching, Form D ingest integration, and observation backfill.
Documentation: Updated CLAUDE.md with accredited-portal architecture and CLI usage.
Implementation Details
pushAttributionCandidateshelper is shared between ingest and backfill so both paths harvest identically — a recompute cannot attribute differently than first ingest.angellist,forge-global) derived from display names, enabling CLI references by either slug or name.https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS