Conversation
Closed
7 tasks
Add `sec eval extract`, a correctness/speed/cost sweep that ranks extraction
models over committed golden fixtures, so we can find the cheapest/fastest model
that still extracts correctly. `scoreExtraction` aligns candidate rows to the
expected set on a key field and reports agreement/recall/precision (deduping
rows first); `modelPricing` estimates cost from char-count (no token usage is
exposed); latency is measured wall-clock.
Register models on demand by id shape: a HuggingFace `org/name` id maps to an
`HF_TRANSFORMERS_ONNX` record, otherwise Anthropic. Both declare the `json-mode`
capability `StructuredGenerationTask` gates on. Wire the HuggingFace Transformers
ONNX provider worker-backed (`hftWorker.ts`) so the heavy graph never runs on the
main thread, and patch the too-old bundled jinja so newer `{% generation %}`
chat templates compile. The local default is LiquidAI LFM2.5-350M (fast enough
for the default 3-way); Qwen3-4B is available as a stronger-but-slow baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `sec eval s1`: run a `--reference` model (e.g. claude-sonnet-5) as the "truth" over real committed S-1 sections and score each `--candidate` on agreement/recall/precision against it. `realSections.ts` segments the HTML into management / beneficial-ownership / related-party prose; the reference retries a few times per section to survive intermittent strict-schema failures, and a section it still fails is dropped from scoring. Per-section progress streams to stderr so a long local-model run isn't blind. Record the large-N verdict in CLAUDE.md: LFM2.5-350M is a useful free/fast local first-pass but not a sonnet substitute for production extraction. Move the test runner to vitest (so we can run under node when needed) and drop the temporary probe scripts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a third AI provider, node-llama-cpp (GGUF), worker-backed via `llamaCppWorker.ts` so the native llama.cpp binding (Metal on Apple Silicon) runs off the main thread. Route models to it with an explicit `gguf:` id prefix (absolute path or relative to the GGUF models dir); its `json-mode` is grammar-constrained, so structured extraction stays schema-valid even for thinking models. Context size and models dir are env-configurable (`SEC_GGUF_CONTEXT`, `SEC_GGUF_DIR`). Import the provider registration functions from the `workglow` mega package (`workglow/anthropic/runtime`, `workglow/hf-transformers[/runtime]`) instead of the scoped `@workglow/*` packages, per project convention. node-llama-cpp has no mega subpath yet, so it stays scoped (as `@workglow/cli` does). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Register the three cloud providers (OPENAI / GOOGLE_GEMINI / XAI) so `sec eval` can compare the GPT, Gemini, and Grok families head-to-head with the Anthropic tiers and the local models. secModelRecord dispatches by id shape (gpt-*/gemini-*/grok-*), each record declares the json-mode capability explicitly (the installed provider can't infer it for newer ids), and registerSecProviders wires the three inline cloud registrations defensively. modelPricing estimates per-provider cost so the ranking stays meaningful. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017VavWmLd39RvfEZoBWywcR
scoreExtraction now returns an ExtractionDiff alongside the aggregate score: expected/reference rows the candidate missed, distinct rows it invented, and per-field value mismatches (expected -> got, raw un-normalized values). The CLI prints these grouped by model after the ranking tables so "why is the score not 100%" is answerable without re-reading fixtures; --no-details hides them and --format json carries the diff for machine use. The eval commands run through EvalExtractTask / EvalS1Task via withCli so a multi-model cloud sweep shows task-graph progress instead of being silent until the final table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017VavWmLd39RvfEZoBWywcR
Models agree on a person's role but phrase the title inconsistently
("Chairman of our board of directors", "Member of the Board of Directors",
"and a director") and a single model is not even consistent call to call.
Add normalizeManagementTitle with an extensible KNOWN_TITLE_FIXES list,
applied to every extracted title after the model returns (source_span stays
verbatim), plus a matching prompt nudge toward the same canonical form. The
normalizer is idempotent and unit-tested against every phrasing the
cross-model eval surfaced; it brings all cloud models to 100% field agreement
on the management fixtures with no per-model variance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017VavWmLd39RvfEZoBWywcR
Follow-through on the node-llama-cpp provider fixes for local extraction: - Lower the default SEC_GGUF_CONTEXT from 32768 to 8192. A 32k KV cache for a dense 12-14B GGUF model exceeds the Metal working-set budget even on 64GB, so the context failed to allocate and the model would not load. 8k loads every tested local model; raise the env var for large real S-1 sections (the provider now evicts other cached models on a VRAM error to help it fit). - Unload a local model between eval candidates (runExtractionEval) via ModelDownloadRemoveTask, so a multi-model sweep does not accumulate VRAM/RAM. Best-effort and scoped to LOCAL_LLAMACPP (memory-only unload; never deletes weights); auto-evict covers correctness if unload is unavailable. - Terminate worker-backed AI provider threads at CLI teardown, so a command that touched a local model exits promptly instead of hanging on live worker threads until their idle timeout (~15 min). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moving `eval extract`/`eval s1` onto withCli routed per-item progress into the task-graph UI, which is suppressed when output is piped (`--format json` / background) — so a long local-model run was blind until the final table. Mirror progress to stderr when stdout is not a TTY; stdout stays clean for the table/JSON output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A person who is "Chief Executive Officer and Director" holds two roles, not one
concatenated string. Model titles as `titles: string[]` end to end, per
SPEC.md/ARCHITECTURE.md.
- normalizeManagementTitles() splits a compound title on ,/;/&/and// (Oxford-
comma safe), canonicalizes each role, and de-dupes. Also adds two title fixes:
a bare "Board of Directors" -> "Director", and a trailing "Chairman"
("Chairman", "... and Chairman") -> "... of the Board of Directors" (a
committee chair is left untouched).
- Extraction: management schema `title` -> required `titles` array; the prompt
asks for a list of distinct roles.
- Storage: person_observation.title column -> `titles` array (native Type.Array);
PersonClaim.titles + observePerson; all 13 observePerson callers pass a role
list (management splits; other forms wrap their single role). No data
migration/version bump — the DB is empty (see the Form_S_1 version note).
- Eval: fixtures use `titles` arrays; scoreExtraction scores an array field per
element (weight = expected role count, credit = set intersection), so a model
that finds only some of a person's roles gets partial credit.
Full suite green; adds tests for the split, the canonicalization rules, and the
array-field scoring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ompt - scoreExtraction diff: render an array field as a bracketed, quoted list (["Chairman of the Board of Directors", "Director"]) so a multi-element array is visually distinct from a single string that merely contains a comma — the data was already split correctly, but the joined display made it ambiguous. - Management prompt: state each role must be a SEPARATE array element (never a comma/"and"-joined string), include only roles held at THIS company (not prior employers), and never invent a role that isn't explicitly stated — the local model was misattributing prior-employer roles and hallucinating titles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The score was pure recall of expected field-values, so a model that found every expected value AND invented extras (an extra `titles` role, a misattributed prior-employer title) still reached 1.0. Redefine `score` as the F1 of field-value precision and recall: it drops both when a value is missed and when one is invented. Track `candidateFieldValues` (distinct produced values within matched rows) as the precision denominator. Whole invented rows remain penalized separately by the row-level `precision`. Updates the affected scorer tests and the CLI legend (score/agree). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…irector") A board chairman already sits on the board, so a bare "Director" listed alongside a "... of the Board of Directors" role is redundant and is dropped in normalizeManagementTitles — e.g. ["Chairman of the Board of Directors", "Director"] -> ["Chairman of the Board of Directors"]. A plain officer role does NOT imply board membership, so ["Chief Executive Officer", "Director"] and ["Chief Financial Officer", "Director"] keep both. This runs on both the candidate and the reference, so the chairman+director case is no longer scored as over-production, while a genuinely invented directorship still is. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Small on-device models (node-llama-cpp GBNF, HF Transformers ONNX) cannot
reliably echo the per-call verification token — a grammar-constrained 350M
tends to emit the schema's ^[0-9a-f]{16}$ pattern as the value rather than
the actual nonce — so every local extraction dead-lettered on NONCE_MISMATCH.
Centralize the nonce lifecycle in runGuardedExtraction: for local providers it
omits the nonce from the preamble and strips nonce_seen from the output schema,
skipping verifyNonce; cloud providers are unchanged (plant + verify). Every
other injection defense — the untrusted fence, the multi-pass defang, and the
downstream source-span gate — still applies to local providers.
Verified: LFM2.5-350M-ONNX now extracts management rows on all 4 committed S-1
sections (was: dead-lettered on every one).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both eval commands name the models under test the same way now: `sec eval extract` and `sec eval s1` take `--models`. `--models` reads cleanly in both — in `s1` the models are scored against `--reference`; in `extract` against the golden fixtures. (The internal reference/candidate distinction is unchanged.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three issues surfaced by a haiku-vs-sonnet oracle run: 1. Scorer counted the same person twice. `normalize()` folded case/whitespace but not typographic punctuation, so sonnet's curly "Frank D’Angelo" (U+2019) and haiku's straight "Frank D'Angelo" (U+0027) keyed to different rows — showing as one missing + one extra. Fold smart quotes/primes/dashes to ASCII. 2. Director nominees were modeled inconsistently (sonnet ["Director Nominee"], haiku []). The management prompt said nothing about nominees. Capture nominee status as a distinct role: a plain board nominee -> "Director Nominee"; a nominee to a specific board role -> "<role> (Nominee)" (e.g. "Chairman of the Board of Directors (Nominee)"). Enforced in both the prompt and the post-model normalizer (idempotent, since Title Case would lowercase the parenthesized suffix). 3. The prompt now explicitly excludes advisors/consultants. A 42k SPAC "Management" segment contained both the officer/director table AND an advisors subsection; the sonnet reference intermittently extracted the advisors instead (making a correct candidate look 100% wrong). With the exclusion, models consistently return the directors/officers only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The scorer keyed rows on the raw name string, so a model that wrote
"Frank Martire, III" and one that wrote "Frank Martire III" (or "Richard J.
Boyle, Jr." vs "Richard J Boyle Jr") counted the SAME person as one missing +
one extra. Drop commas and non-decimal periods in normalize(); a period flanked
by digits ("10.00") is preserved so numeric strings stay distinct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ariants The person resolver keys on first|middle|last|suffix, so two spellings of the same person split into two canonical rows. Two gaps: - Typographic apostrophe: "Frank D’Angelo" (U+2019) case-folded to "D’angelo" while "Frank D'Angelo" gave "D'Angelo" — different last names. Fold curly/ prime apostrophes to ASCII BEFORE parsing so fixCase keys off one character. - Initial/suffix periods: "Richard J. Boyle, Jr." gave middle "J.", suffix "Jr." vs "J"/"Jr" for the unpunctuated form. Strip periods and commas from the parsed name parts (apostrophes/hyphens preserved). No data migration / version bump needed (no data yet). Existing tests updated to the new period-free suffix; added regression tests that the variants collapse. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract foldTypographicPunctuation into dataCleaningUtils (smart quotes, curly apostrophes, primes, en/em/figure dashes -> ASCII) and apply it in production normalizeCompanyName (previously only normalizePerson had an apostrophe-only fold). A company written "Macy’s" (U+2019) or with an en-dash now shares a resolver key with its ASCII twin, so glyph variants don't split canonical companies. normalizePerson and the eval scorer now reuse the same helper, so eval and resolver normalization agree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
) PrismML Bonsai 27B (Qwen3.6-based, Apache-2.0) runs through the existing node-llama-cpp `gguf:` path — no special model id or route is needed, so this is documentation only. Adds a "Evaluating Bonsai 27B (local GGUF)" note to the model-comparison harness section: where to fetch a GGUF quant, the exact `sec eval s1 --models "gguf:..."` command, VRAM/context guidance for a 27B, and that the grammar-constrained llama.cpp json-mode keeps this thinking model's structured output schema-valid. Claude-Session: https://claude.ai/code/session_011bbjSr4rMxY6L6HTSTbNmw Co-authored-by: Claude <noreply@anthropic.com>
A harness commit predating main's TypeBox upgrade carried an outdated "typebox": "1.0.55" line that overrode the rebase conflict resolution, silently reverting main's 1.3.6 upgrade at the branch tip. Restore 1.3.6 and resync the lockfile. tsc + full suite (1715) pass on 1.3.6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Use the workglow mega-import re-exports (workglow/node-llama and workglow/node-llama/runtime) instead of the @workglow/node-llama-cpp subpackage directly, matching how every other provider is loaded (workglow/anthropic, workglow/openai, workglow/hf-transformers, …). Drop the direct node-llama-cpp dependency and its override — it resolves transitively via workglow → @workglow/node-llama-cpp (which depends on node-llama-cpp ^3.19.0), still installing 3.19.0. No new packages. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
workglow 0.3.25's ITabularStorage adds `updateWhere`, which the read-only wrapper didn't implement — breaking `tsc` (TS2420). Add it to the write-no-op block returning undefined (a read-only store updates no row), matching the existing put/delete no-op convention. Clears the last build error on harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011bbjSr4rMxY6L6HTSTbNmw
Bring back the LOI stage between searching and deal_announced. No 8-K item code carries an LOI, so processLoi8K (extractor id "loi") AI-detects non-binding letter-of-intent / agreement-in-principle language in known-SPAC 8-K narratives (items 1.01/7.01/8.01 now escalate to the full submission .txt, sharing the redemption path's escalation), records a per-accession spac_loi_extraction row, and emits an loi event dated by the narrative's stated LOI date (else report/filing date). deriveDeals opens/dates the attempt from the event; the rollup lifts loi_date and status "loi" onto the spac row, with a later definitive agreement superseding the stage. extractLoi rides runGuardedExtraction, so the nonce is planted and verified for cloud providers and omitted for local ones. "No LOI reported" is the expected outcome for most trigger 8-Ks, so its MODEL_EMPTY dead-letter is auto-resolved; low-confidence, unverified-span and nonce-mismatch entries stay pending. Model via SEC_LOI_MODEL (default claude-sonnet-5), floor via SEC_LOI_CONFIDENCE_FLOOR (falls back to SEC_S1_CONFIDENCE_FLOOR). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Umw9tRY72uvSaMjv16NxxT
New `sec editorial` command group for the hand-curated fields with no SEC-filing source: - `editorial set <cik>` writes url_sponsor / url_spac / details onto the spac row via SpacReportWriter.recordEditorial, which rebuilds at the row's own as_of anchor: values overwrite on re-import but the anchor never advances, and no automated record* writer carries these fields, so filing replays can never null them. Changes are versioned into spac_history / ChangeLog with change_source "editorial". - `editorial set-family-description <name> <text> --kind <kind>` writes the version-independent family_description table using the same normalizer as the resolver/alias commands. - `editorial import <csv...>` ingests the two CSV formats (spac editorial fields; family descriptions), skipping CIKs with no spac row unless --create-missing (a spac row marks the CIK a known SPAC), with --dry-run validation. data/editorial/spac-editorial.csv carries the 17 real website rows extracted from embarc's legacy JSON by a one-off script (sec.gov links excluded as merge pollution from embarc's combineSources; real sponsor sites come from the legacy url_sponsors array). family-descriptions.csv is a header-only template — embarc has no family blurb data. Commander resolves `--dry-run` against the program-level global option, never a duplicate subcommand declaration, so actions merge isDryRun(); dryRunRouting.test.ts pins that routing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Umw9tRY72uvSaMjv16NxxT
Replace the bespoke backfill tasks (redemptions, merger proxies) with one engine so a new extractor gets historical recovery for free: `sec extractor backfill <extractorId> [--force] [--dry-run]`. BackfillExtractorTask / runExtractorBackfill resolve a per-extractor BackfillDescriptor: every form-routed extractor id is backfillable by default over all filings of its forms; sub-extractors with narrower candidate sets register a selector (redemption / loi: known-SPAC trigger-item 8-Ks), and extractors whose recorded success can be a gated no-op override the needing-work predicate (merger-proxy: candidates lacking a spac_merger_extraction row). The default predicate is the bulk extractor_runs anti-join at the active version; each survivor re-runs ProcessAccessionDocFormTask so the full form pipeline (and any sub-extractors it gates) runs. Per-filing failures are isolated and cancellation raises TaskAbortedError. `sec spac backfill-redemptions` / `backfill-lois` / `backfill-merger-proxies` are aliases over the same engine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Umw9tRY72uvSaMjv16NxxT
Two heading-detection gaps surfaced by real 2021-era SPAC S-1 markup: - StyleResolver only read the CSS text-align property, so pre-CSS EDGAR headings centered with the ALIGN attribute (<P ALIGN="center"><B>The Offering</B></P>) never counted the centered trait and could not become heading candidates. The attribute now folds into the resolved style (CSS still wins when both are present), mirroring the <font size> handling. - assignHeadingLevels ranked signatures by first appearance capped at 6, so cover-page one-off styles consumed the low levels and real section headings collapsed onto level 6 with their own sub-headings — a sibling-level sub-heading then stripped its parent's body (a fixture's MANAGEMENT section went empty). Levels now rank by visual prominence (size, caps, centering, weight) with distinct tiers merged evenly across 1..6, preserving monotone order so a sub-heading can never close its parent's section. Fixture s1_1848507 now segments "The Offering" (2,909 chars carrying the unit terms); every other golden fixture resolves the same sections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Umw9tRY72uvSaMjv16NxxT
- `loi` joins EVAL_EXTRACTORS (detection-style single-object extractor; keyField target_name, scored on target_name + loi_date) with eight golden 8-K narrative fixtures — three LOI positives, five confusable negatives (definitive agreement, vote/redemption results, LOI termination, non-business-combination LOI, trust mechanics) — so `sec eval extract --extractor loi` ranks any model set on the prompt. - `offering-terms` joins EVAL_EXTRACTORS (positional single-object, scored on the objective unit-terms numerics) and maps to the segmenter's "The Offering" section for the `sec eval s1` oracle. - New `sec eval unit-terms`: scores offering-terms extraction against embarc's hand-curated unit structure as TRUTH (not a reference model) over the committed real S-1 fixtures. The truth set (src/eval/mock_data/embarc-spac-unit-terms.csv, 1,283 CIKs flattened from embarc's details maps by a one-off script) is deliberately NOT imported into the database — the S-1/424 extraction derives those figures from filings; embarc's curation is the independent yardstick. Scored numerics round to 2 decimals on both sides since scoreExtraction compares numbers exactly and 1/3 repeats. The report reuses the extract-eval summary shape so the ranking table renders identically (summarizeModelRuns exported; printTable takes a legend). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Umw9tRY72uvSaMjv16NxxT
Two fixes for management title convergence, verified against a gpt-5.4-mini / gemini-3-flash head-to-head: 1. Prompt: a person's roles are often split between the summary table and the bio's "has served as our X and Y" sentence. Instruct the model to take the UNION of roles stated in EITHER at THIS company. This was the John Lewis miss (table "CFO", bio "CFO and Secretary" — gpt-5.4-mini dropped Secretary); both candidates now return the full role set (100%). 2. `sec eval s1 --reference golden` scores candidates against committed human-verified labels (src/eval/goldenS1Labels.ts) instead of a live reference model — deterministic, $0, and it measures CORRECTNESS rather than agreement-with-a-wandering-sonnet. Labels cover the 4 committed management sections in canonical (normalizeManagementTitles) form; a test pins them canonical/covering. Authoring the labels + running candidates flushed out the truth on the ambiguous Haldeman cell (three models independently read the "Director" as seated) and confirmed a real gemini over-production on William Sherman. Both candidates now match golden truth 100%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The grammar guard only forced minItems:1 on top-level arrays (people), so a local grammar model could still take the `[]` shortcut on the nested people[].titles array — observed with qwen3.6-35b-a3b returning every person with a full bio but titles:[]. Extend the transform to also force minItems:1 on nested arrays-of-strings inside a top-level array's row objects (titles), while leaving nested arrays-of-objects (related-party parties[].transactions) empty- able so a row without one isn't forced to hallucinate. Renamed requireNonEmptyTopLevelArrays -> requireNonEmptyGrammarArrays, exported, unit tested. Verified: with the fix the model fills real roles (President / CFO / Director Nominee) instead of []. (Separately, qwen3.6-a3b is a *thinking* model and still emits placeholder "Title1" junk for multi-role people under grammar constraints — a model-fit issue json-mode can't fix, not this guard.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Investigation into whether structured-XML (edgarSubmission) 8-Ks occur in
EDGAR. An empirical sweep of 157 real 8-Ks across ~37 filers — large-cap
issuers plus 25+ SIC 6770 blank-check SPACs, including 40 redemption-relevant
trigger-item (5.07/2.01/8.01) vote/closing 8-Ks — found zero that parse to a
non-empty edgarSubmission formData. Modern 8-K bodies are XHTML
(<?xml …?><html …>), legacy ones are SGML <DOCUMENT> text.
Both deferred SPAC-redemption findings (escalation gate vs processForm8K
trigger set; full-.txt escalation dropping an XML periodOfReport) are gated on
XML 8-Ks existing and diverging from metadata, which they don't. Not a bug.
Add clarifying comments at the three touch points (Form_8_K.parse, the
extractItemCodes / effectiveReportDate paths in Form_8_K.storage, and the
escalation gate in ProcessAccessionDocFormTask) documenting that the
submissions-API items/report_date metadata is authoritative for 8-Ks because
the body is never edgarSubmission XML. No behavior change; the existing
"HTML primary docs parse to {}" regression test locks the finding.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EniG8S5aWSQRVmrXKTbncs
…) (#192) * refactor(types/edgar): replace generated TS enums with as const objects The machine-generated EDGAR type modules under src/types/edgar/ emitted TS `enum` declarations, which the project convention forbids. Rewrite each `export enum Name { ... }` into an `as const` object plus a `keyof typeof` union type, preserving both the value namespace (`Name.Member`) and the type (`Name`) that an enum provided. None of these enums are imported as values outside the generated files, so the rewrite is transparent. Add a reproducible `gen:types` script (scripts/gen-types.ts) that post-processes the xgen output to perform this conversion. It is idempotent, so it is safe to re-run after regenerating the modules with xgen. Closes #142 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012BGH1bWRbHV2WRrF3k1VVw * chore: remove gen:types post-process script Drop scripts/gen-types.ts and its package.json entry. The types/edgar modules have already been converted from TS enums to `as const` objects; the one-off post-processor is no longer needed. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012BGH1bWRbHV2WRrF3k1VVw --------- Co-authored-by: Claude <noreply@anthropic.com>
* feat(cli): honor --json flag in status and error output
CLI status/error output was always pretty-printed, so integrations could
not machine-parse it. Reintroduce the global `--json` flag and route it
through a new SEC_JSON_OUTPUT DI token (mirroring SEC_DRY_RUN):
- statusMessage() emits {"status","message"} JSON under --json instead of
the glyph-prefixed line, so runCommand's error path and the sec.ts
SecCliConfigurationError path both become parseable.
- The preAction hook and the init command register the flag into DI; an
isJsonOutput() helper reads it (defaulting to false when unregistered).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0174YErFdh2ZSGaUaCwhdoaj
* docs(cli): fix statusMessage JSON example; test --json option routing
Address Copilot review on #193:
- Correct the statusMessage JSDoc example to valid JSON
({"status":"error","message":"..."}).
- Pin that a post-subcommand --json routes to the program-level option
(the Commander nuance already guarded for --dry-run), so a Commander
upgrade cannot silently break JSON output in `sec <cmd> --json`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0174YErFdh2ZSGaUaCwhdoaj
---------
Co-authored-by: Claude <noreply@anthropic.com>
…xture Commits the Churchill Capital Corp XII 424B4 primary-doc HTML — the priced companion to the committed `s1_2114227_*` registration fixture — and a golden test that pins the deterministic half of the priced-prospectus pipeline that feeds the AI offering-sections pass: - segmentation into The Offering / Underwriting / Use of Proceeds (the exact sections the priced-424 AI extraction reads); - fee-prepaid SPAC 424B4 (Rule 456(a)) carries no inline XBRL / fee exhibit, so the untagged-body path is pinned as the production reality; - the final priced terms in the prose (36,000,000 units @ $10.00, Citigroup underwriter) — distinct from the S-1's registered terms that `sec issuer deal` compares against. Documented in the mock_data SOURCES.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LTcULExNZgjvoe7dAUK5h7
…lobs The 424B4 golden fixture landed under mock_data/s1/, where three S-1 helpers auto-discover every .htm and assume it is an S-1 with management sections (realSections.loadRealS1Sections, parseEdgarHtml.golden EXPECTED map, goldenS1Labels coverage). A priced prospectus has none of those, so it broke all three. Move the fixture to its own mock_data/424/ directory (with its own SOURCES.md) so the S-1 globs no longer sweep it in, and point Form_424.golden.test.ts at the new path. Restores mock_data/s1/SOURCES.md to its original content. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LTcULExNZgjvoe7dAUK5h7
…#195) * feat(xbrl): ISO date transforms + CIK/dimension query coverage (#154) Hardening: register the ixt/ixt-sec date transforms so non-numeric date facts (e.g. dei:DocumentPeriodEndDate tagged ixt:date-monthname-day-year-en) normalize to ISO-8601 instead of being left as locale strings. Both the TR1 concatenated (datemonthdayyearen, dateslashus/dateslasheu) and TR3/TR4 hyphenated spellings are handled; an unparseable date keeps its trimmed raw text rather than blanking the fact. Query coverage: expose `sec query xbrl --cik <cik>` to read a concept's series across all of an issuer's filings (the query function already supported CIK; the CLI only took an accession), ordered by (accession, fact_index) with an Accession column. Surface each fact's dimensional qualifiers as a compact Axis=Member column. Tests: date-transform cases (all spellings + unparseable fallback), a dei date fact normalized end to end, formatXbrlDimensions rendering and malformed-JSON degradation, and CIK-path ordering (unfiltered and concept-filtered). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rEbCs1VKZQ4DJ6CfQojT * fix(xbrl): address review — reject impossible dates, strict CIK, ORDER BY pushdown - toIsoDate: reject impossible calendar dates (Feb 30, Apr 31) via a UTC round-trip check instead of emitting an invalid ISO string like "2024-02-30"; unparseable dates keep their raw text. - query xbrl --cik: require an all-digit string (matches parseCikArg in the fetch group) so "123abc" errors instead of being silently read as 123. - queryXbrlFacts unfiltered path: push ORDER BY (accession_number, fact_index) down to storage so offset/limit pagination is consistent across pages, rather than sorting only the returned page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rEbCs1VKZQ4DJ6CfQojT --------- Co-authored-by: Claude <noreply@anthropic.com>
#150, #151) Implements three SPAC roadmap items: #149 AI content classifier for miscoded SPACs: an AI classifier behind the existing S1Classification `classifier_source = "ai"` seam catches SPACs filed under a miscoded/absent SIC. Gated on a cheap blank-check keyword heuristic so the AI call stays rare; a confident SPAC verdict flips is_spac, overwrites the classification row, and mints the known-SPAC row. Env: SEC_S1_CLASSIFIER_MODEL / SEC_S1_CLASSIFIER_CONFIDENCE_FLOOR. Added to the eval harness. #150 Sponsor promote economics: extracts founder (Class B) shares + percentage, private-placement warrant count/price/public coverage, and trust per-share/total from the prospectus into a new spac_promote_terms table, via the shared offering-sections runner (S-1 and priced-424). Added to the eval harness. #151 De-SPAC linkage: on a completed combination the rollup derives surviving_name/current_name from the deal target, and recordDeSpacLinkage populates surviving_name/post_merger_sic/post_merger_tickers from the SPAC CIK's post-close entity metadata on the item-2.01 8-K. New `sec spac backfill-despac` refreshes completed SPACs after submissions catch up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KqMBp35YzRXtkJMu4Xa4zm
Address review: recordDeSpacLinkage overwrote surviving_name / post_merger_sic / post_merger_tickers on every run whenever entity data diverged from the SPAC-era value, contradicting the "cannot regress a populated linkage" contract, and wrote post_merger_tickers without checking divergence from spac_tickers or sorting (non-deterministic order). - surviving_name: upgrade the deal-target fallback (or fill an empty slot) once; never overwrite an already entity-sourced snapshot on a later replay/rebrand. - post_merger_sic: write-once (only when the slot is still null). - post_merger_tickers: write-once, deduped + sorted for determinism, and only when the symbol set diverges from the SPAC-era tickers. Adds tests for the later-rebrand no-mutation case and the SPAC-era-ticker non-leak case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KqMBp35YzRXtkJMu4Xa4zm
Correctness fixes from a multi-agent review of the branch:
- s1: a confident `is_spac: true` / `is_loi: true` with a null source_span
was returned as null, which the caller auto-resolved as the expected
negative — silently discarding a correctly-identified SPAC/LOI with no
dead-letter left to triage. Throw so it dead-letters MODEL_INVALID_OUTPUT.
- s1/424: `sponsor-promote` was dead-lettered for every filing but run with
`skip: !isSpac`, and a skipped section never markResolves, so a non-SPAC
entry could never drain and every retry re-billed the full AI extraction.
Gate the section-name list on isSpac.
- s1: the spac-classification dead letter was recorded on the raw-HTML
heuristic but resolved only under the narrower summary-prose gate; resolve
the entry when the classifier declines to run.
- s1: split a compound title on "and"/"&" only when both sides end in a role
head noun — "Chief Legal & Administrative Officer" was being shredded into
two roles the person does not hold.
- html: convert %/em font sizes to points. Ranking read "120%" as 120pt, so a
slightly-enlarged heading outranked every genuinely larger one and inverted
the hierarchy the segmenter feeds to the AI extractors.
- eval: a field null in BOTH reference and candidate credited the F1 numerator
without the candidate denominator, so `score` exceeded 1 (1.6 / 1.333) and
ranked a model that emits nothing above one that fills fields in.
- eval: report extractors with no S-1 section mapping as skipped instead of
silently scoring zero sections.
- editorial: reject a blank cik cell — Number("") is 0 and Number.isInteger(0)
is true, so a blank imported under CIK 0 and marked it a known SPAC.
- cli: `sec query persons` read the renamed `title` field, blanking the column;
render list-valued cells as "A, B".
- s1: guard the named-entity table against inherited Object.prototype keys — a
filer-planted `&constructor;` stringified a function into the model prompt.
- db: `sec db setup --dry-run` really executed DROP TABLE form_8k_events (raw
SQL reaches around the read-only wrapper) while the recreate was no-op'd.
Guard the migration and the raw view/ALTER block.
- spac: surviving_name was derived from the deal target, persisted, then read
back as if explicit, so a superseding proxy could never correct it. Record
its provenance so deal-derived values re-derive and entity snapshots persist.
Schema migrations are intentionally omitted and extractor/resolver versions
stay at 1.0.0: the database is empty, so there is nothing to migrate or
re-resolve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`sec eval extract --extractor beneficial-ownership` validated the name against EVAL_EXTRACTORS, selected zero fixtures, and then printed a ranked table with 0/0 runs plus "no row/field disagreements — every scored run matched the expected" — an affirmative claim of correctness from nothing run, exit 0. Same silent-zero class as the realSections gap fixed in 45fb2d3, on the sibling code path. Registration in EVAL_EXTRACTORS does not imply a committed fixture: beneficial-ownership, related-party, and offering-terms are all registered with none. Throw from selectFixtures instead, before models are registered, and build the --extractor help from the extractors that actually have fixtures so the CLI stops advertising unscorable ones. This surfaces the gap rather than closing it — those three extractors still need fixtures, and golden truth (src/eval/goldenS1Labels.ts) still covers only the 4 management sections, so the beneficial-ownership / related-party figures in the CLAUDE.md verdict table remain sonnet-oracle-relative rather than measured against human-verified labels. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n truth
Authoring golden labels for the 5 committed beneficial-ownership sections
surfaced a production data-quality bug, so this fixes the root cause first.
Ownership tables end in an "All officers and directors as a group (N)"
subtotal. The prompt never said whether to emit it, and sonnet is inconsistent
— it emits the row for 4 of the 5 committed sections and omits it for the 5th.
When emitted it comes back as owner_kind "company", and Form_S_1.storage's
persist path hands every company row to observer.observeCompany(), which
resolves it into the CANONICAL COMPANY TIER. So we were minting canonical
companies named "All executive officers, directors and director nominees as a
group (five individuals)", with s1:beneficial-owner relations and an ownership
row whose aggregate share count double-counts the members listed above it.
The eval could not catch this: the oracle is the model making the mistake.
- Pin the convention in the prompt (no subtotal row; no footnote markers or
parenthetical annotations in the name) and enforce it in code with
isOwnershipGroupSubtotal — a prompt is not a guarantee, and this row reaches
the canonical tier. "Allstate"/"Alliance Group" style names are unaffected;
only the "as a group" subtotal phrasing matches.
- Commit golden labels for all 5 beneficial-ownership sections, transcribed
from the filing tables and cross-checked against an independent model read.
- Generalize GoldenRow (management rows carry titles; ownership rows are
name-only, matching the extractor's compareFields) and extend the tests to
both extractors, incl. that no label is a subtotal or carries a parenthetical.
- Add the two missing beneficial-ownership fixtures, so `sec eval extract
--extractor beneficial-ownership` scores something. Both exercise the
subtotal and footnote-marker conventions.
Measured after the fix (sec eval s1 --reference golden --extractors
beneficial-ownership): sonnet-5 and haiku-4-5 both score 100% agreement /
recall / precision over all 5 sections — haiku at ~2.8x lower cost.
LFM2.5-350M scores far worse than the sonnet-oracle run suggested: 3 of 5
sections hard schema-fail and 28 of 33 owners are missed, including a
pretraining-memorized hallucination ("Churchill Sponsor XII LLC" returned for
an unrelated issuer's table). CLAUDE.md's verdict is updated accordingly, and
its remaining oracle-relative figures are now marked as such.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLAUDE.md no longer mentions LFM2.5/LiquidAI. The local-model verdict block is replaced by the finding that actually matters for model choice: haiku-4-5 matches sonnet-5 at 100% on the golden beneficial-ownership sections for ~2.8x less, and small local models hard schema-fail and hallucinate pretraining- memorized entities. The debunked "~100% entity recall" claim is gone from the Constants.ts comment too. Scrubbing the docs alone would have left them untrue: both eval sweeps still ran the local model by DEFAULT — `sec eval extract` as the third leg of its "3-way", and `sec eval s1` as its default --models candidate. So every default sweep burned minutes per section on a model that is not a production candidate. Both now default to cloud (extract: haiku vs sonnet; s1: haiku against the sonnet reference). The HFT path itself is untouched and still opt-in via SEC_HFT_MODEL: the provider, worker, jinja chat-template patch, and the fallback repo id all remain, so a local model can still be ranked by passing it explicitly. The patchHftChatTemplate comment still cites the LFM2.5 template family because that is the concrete markup the workaround exists for. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A model oracle caps every candidate at its own accuracy, so the reference
should be the strongest model available rather than a mid-tier one that
candidates are then measured against. `sec eval s1 --reference` now defaults to
claude-opus-4-8 (pricing and ANTHROPIC routing already resolve for that id;
`--reference golden` is unaffected and remains the deterministic path).
Verified on the committed beneficial-ownership sections: opus completes 5/5 and
returns exactly 33 rows — the same roster as the committed golden labels.
Docs no longer name a specific model as "the oracle"; the surrounding prose is
generalized ("the strongest model", "not merely reference-like") so it does not
go stale on the next model refresh. Production extractor defaults
(SEC_S1_MODEL etc.) are untouched — this is the eval reference only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
"V-Cube, Inc. and Naoaki Mashita" is a company AND a person. Keeping it as one row put a value that is plainly two names into `name` — and the S-1 persist path resolves a company row straight into the canonical company tier, so that combined string became a single bogus canonical company. Same class of defect as the "as a group" subtotal row, just less obvious. Footnote 5 of that filing attributes the shares separately (1,520,000 to V-Cube, 45,942 to Mr. Mashita), so splitting is stated by the filing rather than synthesized. - Prompt: `name` must hold exactly one owner; a cell naming several yields one row per owner, never a combined "X and Y". No code guard here, unlike the subtotal case — a name splitter cannot safely tell "V-Cube, Inc. and Naoaki Mashita" from "Johnson and Johnson", so this is prompt + golden truth only. - Golden: that section is now 7 rows (34 total, was 33). This was measurably ambiguous, not merely theoretical: haiku scored 100% on the section against golden in one run and split the row in the next — same model, same input. With the convention stated, opus AND haiku both score 100% agreement / recall / precision over all 5 sections with zero disagreements. Also fixes the diff renderer that hid this: keyList joined keys with ", " unquoted, so the two rows ["V-Cube, Inc.", "Naoaki Mashita"] printed as `V-Cube, Inc., Naoaki Mashita` — indistinguishable from one comma-containing name, which is exactly the distinction these diffs exist to show. scoreExtraction's displayValue already bracket-quotes arrays for this reason; keyList now quotes each key too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…5.10.0 Reconcile deps with main's 'chore: update deps' after rebasing harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ess (#198) * feat(models): download model weights before use across commands Local model weights must be present on disk before generation, but the providers differ on when that happens: cloud models have nothing to download, HuggingFace ONNX auto-fetches on first generation, and node-llama-cpp (GGUF) loads its model_path directly and never fetches at generation. Add ensureModelDownloaded as the single seam that normalizes this — it runs ModelDownloadTask for the local providers (no-op for cloud, memoized per model id) and skips a bare-path GGUF whose file is assumed on disk. Wire it into runStructured (the chokepoint every extractor funnels through) so normal runs fetch weights before use, and prefetch in the eval loops before their timed sections so download time isn't charged to a model's measured latency. Make GGUF ids fetchable rather than pre-staged: a gguf: id may now be a remote URI (gguf:hf:org/repo:quant or an https URL), which secModelRecord turns into a model_url download source plus a local model_path/models_dir under the GGUF models dir. Plain gguf: paths stay load-directly local files, unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011HoQKRtpU7mGpK8Dah9ySm * feat(models): render model-download progress through the CLI task UI The download seam used a throwaway execute context, which silently discarded the download run-fn's progress events — a multi-GB GGUF/ONNX fetch looked like a hang. Thread the running task's real IExecuteContext into ensureModelDownloaded and on to ModelDownloadTask.execute: the download's phase events are forwarded to context.updateProgress, which the @workglow/cli progress UI (withCli) already renders, and context.signal now aborts a long download on Ctrl-C. Add prefetchModel, the best-effort wrapper the CLI-task boundaries call to fetch weights (with visible progress) before the work begins: - eval sweeps pass their task context so `sec eval` shows download progress; - the AI form processors (S-1, 424, merger-proxy, redemption, LOI) prefetch once after resolving their model, via a context threaded through the shared storageArgs in ProcessAccessionDocFormTask. runStructured keeps a context-less ensureModelDownloaded call as a per-section correctness safety-net (it downloads silently only if a model was never prefetched, e.g. a sub-extractor's distinct model); the progress-bearing fetch lives at the task boundary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011HoQKRtpU7mGpK8Dah9ySm * feat(s1): surface per-section generation progress on the task row Thread the running task's IExecuteContext down the extraction stack (processFormS1 / runOfferingSections / the 424, merger-proxy, redemption and LOI processors → extract* → runGuardedExtraction → runStructured), so each section's AI generation reports progress on the CLI task row instead of running silently. The context alone is not enough: StructuredGenerationTask.execute() keeps only the finish event and drops the phase events, so a threaded context would still see nothing. runStructured now drives the task's executeStream itself — identical result (same retry/validation loop, same finish) — and forwards the Preparing/Generating phases to context.updateProgress. Each section's row now shows the model actively working; the download progress already added shows alongside it. context is optional throughout: the eval sweeps and unit tests call the extract* functions with the two-arg (no-context) signature and fall back to the self-contained stub, so behavior there is unchanged. All side effects are preserved, so the form-pipeline tests that pin processFormS1/ProcessAccessionDocFormTask stay green. Adds a regression test asserting a threaded context receives the phase progress. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011HoQKRtpU7mGpK8Dah9ySm * refactor(models): drive model tasks through run(); address PR review Use the task run() lifecycle instead of hand-driving execute()/executeStream for the three model tasks: - runStructured runs StructuredGenerationTask.run() and forwards its Preparing/Generating phases to the caller's context.updateProgress via the runConfig.updateProgress hook (caching disabled to match execute()'s never-cache semantics). run() also fails loudly — a generation that can't finish throws rather than returning {}, so the prior executeStream finish-guard concern no longer applies. - ensureModelDownloaded runs ModelDownloadTask.run(); unloadLocalModel runs ModelDownloadRemoveTask.run() (dropping its throwaway stub context). Address the PR review: - Memoize ensureModelDownloaded on a robust key (model_id ?? model ?? provider_config.model_url ?? model_path), mirroring resolveModelId, so a record without model_id still downloads once instead of every call. - isRemoteGgufUri matches hf: case-insensitively. - ggufCacheFileName folds the whole source path into the cache filename so distinct remote GGUF sources (same repo name / different org, or two repos each holding model.gguf) don't collide on disk. Adds tests for the model-identified-by-`model` memo path, uppercase HF:, and same-repo-name/different-org cache-target uniqueness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011HoQKRtpU7mGpK8Dah9ySm --------- Co-authored-by: Claude <noreply@anthropic.com>
…renderer Adds runWorkflowCli — a shared helper that pipes tasks into a Workflow terminated by an OutputTask sink and runs it through @workglow/cli (renderWorkflowRun progress UI on a TTY, plain run when piped) — plus task classes for every query/db subcommand. Commands now only parse arguments, run the graph, and render the collected output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcsARaK3c9Ta89y2GydQan
…197) * Add accredited-investor portal tables, seed bootstrap, and Form D attribution Accredited-investor portals (AngelList, Forge, EquityZen, ...) never register with the SEC, so a curated tier bootstraps from an embedded copy of the embarc portals-accredited list: accredited_portal (slug-keyed), plus accredited_portal_signal fingerprints (normalized entity names, phone international numbers, address hash ids) and derived form_d_portal_attribution rows. processFormD harvests issuer / related-person / sales-compensation candidates and PortalAttributor matches them exactly against the signal table (address > phone > name); 'sec accredited-portal attribute' recomputes attributions from stored observations with clear-then-recompute semantics. New 'sec accredited-portal' CLI group: import, list, signal add/list/remove, attribute, filings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS * Address code-review findings on accredited-portal attribution Correctness: unscoped attribution now clears the filing's prior rows before recomputing (re-ingest replays self-heal after signal changes); the backfill filters bad-data placeholder tokens out of person name parts exactly like the ingest path (shared pushAttributionCandidates builder ends the divergence); the matches audit trail keeps every corroborating filing role and the strongest-match tie-break is deterministic. Efficiency: signal lookups batch through getBulk; clearPortal/clearAccession use deleteSearch instead of row-by-row deletes; the backfill streams observations with keyset pagination instead of materializing both tables. Cleanup: dead signal-type const objects removed; seed input type reuses the embedded seed entry; CLI --type is case-insensitive; shared option block for signal add/remove; clearer seed-file JSON error with the path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS * Skip redundant per-accession clear during backfill attribution The backfill clears its whole recompute scope up front (clearAll / clearPortal), so the attributor's per-accession clear — needed only for the live-ingest replay path — was dead work repeated once per filing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS * Drop featured column from accredited_portal Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS * Address xhigh code-review findings on portal attribution - Look up signals before clearing a filing's attribution rows and serialize the clear+write per accession (KeyedMutex), so a transient failure or a concurrent replay can no longer leave a filing stripped of rows. - Centralize the signature-role exclusion in pushAttributionCandidates and reject placeholder tokens (None, N/A, ...) inside normalizeNameSignal, so ingest, backfill, and CLI harvest identically by construction (with an idempotency test pinning normalize-twice parity). - Backfill: preload the signal table once (all or per portal), stream observations via the shared streamMatchingRows helper, and keep only candidates that match a loaded signal — memory is bounded by match count and unmatched accessions skip the filing lookup entirely. - CLI: address signals no longer inherit --country (ingest never has one), and --value for addresses must be a normalized pipe-joined hash. - Import: reject string live values in external seeds and report distinct portal count; drop the unused deletePortal method. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS * Use the global --json flag for accredited-portal JSON output The CLI's machine-readable convention is the global --json flag (SEC_JSON_OUTPUT / isJsonOutput), not a per-command --format option; list and filings now follow it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS * Address Copilot review comments on accredited-portal CLI and schema - Trim pasted address_hash_id values before storing - Constrain matched_signal_type to the three signal-kind literals - Make attribute's --all and --portal mutually exclusive - Render null portal live status as 'unknown', not 'closed' - Soften an overclaiming comment about person-name collisions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS * Address deferred items: attributor versioning, signal suggestions, curation hardening - Register portal-attributor as a resolver component: rows stamp the active slot's semver, 'sec version coverage resolver portal-attributor' reports the share of attribution rows at a version, and drop-previous purges rows at the retired version - Add 'sec accredited-portal suggest': surfaces address/phone values recurring across many distinct Form D filings that are not yet curated signals - Add 'sec accredited-portal set' for curated portal fields (cik, notes) - Require explicit --country for phone signals (region-sensitive parsing) - Widen address-hash columns (64 -> 512) and phone columns (20 -> 32) in the observation and canonical-junction schemas so Postgres can store full address_hash_id values (schema-only; no deployed data) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdQcWqsBKVGXh12C17rrXS --------- Co-authored-by: Claude <noreply@anthropic.com>
Every subcommand now runs its work as a Workflow task graph via the shared runWorkflowCli helper (tasks + OutputTask sink, rendered with @workglow/cli's renderWorkflowRun progress UI on a TTY, plain run when piped). Business logic that previously lived inline in Commander actions moved into ~35 new task classes under src/task/ (query, db, versioning ceremonies, resolve, canonical/family aliases, spac report and backfills, editorial set/import, fixtures, init apply, per-CIK facts), leaving commands to parse arguments, run the graph, and render the collected output. sync and bootstrap now build a single pipeline per invocation instead of sequential runs. CLI output strings and exit codes are preserved verbatim; helpers that tests import (assembleSpacReport, compareIssuerDeal, family lookups, resolveCanonical*Ref, countEligibleDeadLetters, buildEnvConfig) stay exported. UpdateAllSubmissionsTask/UpdateAllCompanyFactsTask gained input schemas since graph-driven runs validate inputs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcsARaK3c9Ta89y2GydQan
Co-authored-by: Steven Roussey <sroussey@gmail.com>
Co-authored-by: Steven Roussey <sroussey@gmail.com>
…pe tier wiring Findings from code review of the graph-ification refactor: - Expected user-errors (unknown canonical/family names, self-alias, missing spac row) now come back as output ports instead of throws. On a TTY the workflow renderer intercepts thrown errors with process.exit(1), bypassing command error handling and CLI teardown; as data, the commands render the same 'error: ...' text and exit code on both TTY and piped runs. This also removes editorial set's fragile string-prefix error classification. - editorial import records a mid-file import failure as a fatalError result entry (stopping the sweep) so summaries for already-committed files are still reported instead of being lost with the throw. - Removed the duplicate SetFamilyDescriptionTask; editorial reuses FamilyDescriptionSetTask, whose schema validates the kind enum. - New canonicalTier deps seam collapses the person/company copy-paste in the three canonical alias tasks (mirrors familyTier); the canonical reference resolvers move there (command re-exports them). - issuerCiksByFamilyName lives in familyTier now; IssuersByFamilyTask and both command modules delegate to it (removes the task-to-command layering inversion and module cycles). - countEligibleDeadLetters lives with ListDeadLettersTask (extractor group re-exports it); VersionPromoteTask no longer imports from the CLI layer. - ceremonyRepos() factory replaces the registry/event-repo wiring duplicated across the five version-ceremony tasks. - ResolveObservationsTask reports a skipped count and shares one resolve loop across kinds. - FamilyAliasListTask only requires resolverVersion for orphan listing; the sponsor alias-list no longer passes an inert version. - runWorkflowCli documents the piped-port-name discipline and the expected-errors-as-data convention. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcsARaK3c9Ta89y2GydQan
Co-authored-by: Steven Roussey <sroussey@gmail.com>
…efactor-jocdah Claude/sec commands graph refactor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a comprehensive evaluation framework for comparing AI model performance on SEC extraction tasks, along with support for multiple cloud and local model providers beyond Anthropic.
Key Changes
Evaluation Framework
sec eval extractcommand to benchmark models against fixture-based golden datasetssec eval s1command to score models against real S-1 prospectus sections using an oracle reference modelscoreExtraction.ts: Field-level scoring with F1 metrics, entity recall, precision, and concrete diff reporting (missing/extra/mismatched rows)runExtractionEval.ts/runOracleEval.ts: Orchestrate multi-model evaluation sweeps with progress reporting and cost estimationmodelPricing.ts: Per-model cost estimation from character counts and public pricing (OpenAI, Anthropic, Google, xAI)fixtures.ts: Registry of extractors (management, beneficial ownership, related party) with golden expected outputsrealSections.ts: Load real S-1 HTML sections from committed mock data for oracle evaluationMulti-Provider Model Support
registerModels.tsto dispatch model IDs to appropriate providers:gguf:prefix → node-llama-cpp (GGUF) local modelsorg/nameformat → HuggingFace Transformers ONNXgpt-*/o*→ OpenAIgemini-*→ Google Geminigrok-*→ xAISecHftModelDefault(LFM2.5-350M) as a local baseline model for cost-free comparisonregisterProviders.ts: Register OpenAI, Google Gemini, xAI, HuggingFace Transformers, and node-llama-cpp providershftWorker.ts) to isolate heavy@huggingface/transformersgraph from main threadpatchHftChatTemplate.ts: Strip{% generation %}tags from HFT chat templates (post-0.5.6 compatibility)Management Title Normalization
normalizeTitle.ts: Deterministic post-model canonicalization of management titles (e.g., "Member of the Board of Directors" → "Director", possessive board refs → "the Board")sectionExtractors.tsto normalize extracted titles before storageSchema & Storage Updates
title(scalar string) totitles(array of strings) in management extraction schema to capture multiple rolesPersonObservationSchemaand all dependent storage/resolver code to usetitlesarrayCLI Integration
evalcommand group withextractands1subcommandsEvalExtractTask/EvalS1Task: Workglow task wrappers for the evaluation harnessTesting & Utilities
scoreExtraction.test.ts: Field-level scoring, entity recall, precision, duplicate deduplicationnormalizeTitle.test.ts: Title normalization patternsrealSections.test.ts: Real S-1 section loadingpatchHftChatTemplate.test.ts: Chat template tag strippingunloadModel.ts: Safe worker model unloading between eval candidatesworkers.ts: Worker thread lifecycle managementNotable Implementation Details
confidence,source_spanare ignored); values are normalizedhttps://claude.ai/code/session_011bbjSr4rMxY6L6HTSTbNmw