Skip to content

feat: add the instrument hub and record provenance - #1807

Open
thomasbeaudry wants to merge 10 commits into
mainfrom
feat/datahub-instrument-hub
Open

thomasbeaudry wants to merge 10 commits into
mainfrom
feat/datahub-instrument-hub

Conversation

@thomasbeaudry

@thomasbeaudry thomasbeaudry commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Adds an instrument view of collected data alongside the subject hub from #1654 (merged). Users can inspect one instrument edition across subjects, or open a series to see what its members collected.

  • Browse sortable instrument listings with previews, minimum-record and collection-date filters, and listing downloads.
  • Drill into instrument records with collection-method/series filters, sortable columns, and return links through series, instruments, and subjects. URL filters survive table/chart tab switches and apply to record downloads.
  • Explore distributions and scatter plots; choose shared chart palettes on instrument and subject graphs, and download PNGs with headings and legends. Subject graphs also include non-form instruments.
  • Show collection method and series in record tables. Record exports include instrument name/edition and provenance; wide series exports retain every member's measure columns.
  • Add permission-scoped instrument/series summaries and series-record queries.

Merging main

#1654 merged, which conflicted this branch. 7b17710e4 merges main (432dec811). Four conflicts:

  • instrument-records.service.ts — main deleted InitialMessage as unreachable while this branch renamed InitData to InitPayload/InitInstruments. Kept this branch's names without the deleted type; main's removal of the message.success check on the INIT acknowledgement merged cleanly.
  • instrument-records.service.test.ts — took main's deletion of the test pinning a defect main has since fixed.
  • subjects/$subjectId/graph.tsx — took this branch's renderChartPng rewrite whole. main's side was a cleanup inside the html2canvas block that rewrite replaces.
  • datahub.visualization.selectMeasures — main removed it as dead once the subject graph stopped using it, but the instrument graph still reads it, so it is restored in 5657bbc84.

This branch forked before two gates main has since added, so it had never been measured against either:

  • The unit test layout rule (odc/unit-test-names-subject). Three flat specs under apps/web/src/__tests__/ moved beside the files they test.
  • 100% unit coverage (pnpm test:coverage). 21 files were short, seven of them at 0%: both chart components, SelectChartPalette, and four instrument hub route files.

Defects the coverage gate exposed

Three branches turned out to be unreachable because the code around them was wrong:

  • The scatter tooltip rendered the collection date as a raw epoch. formatter compared the row name against 'time', but recharts passes the axis name, so the comparison never held and both rows fell through to the measure label. The labelFormatter that did the date formatting is never called for a ScatterChart at all. The date is now formatted in the row recharts actually names after the x axis.
  • instrument-hub.spec.ts expected an em-dash in the series cell. f6d3f8295 changed that cell to a translated "None" without updating the spec, which could not run while the branch conflicted.
  • subject-detail.spec.ts waited on the glob **/graph. The subject tab links now carry the chart palette, so clicking one lands directly on …/graph?palette=default and the URL never passes through a query-free state — a ** glob cannot match it, and the test timed out while sitting on the correct tab. The wait is now a regex allowing the query string.

Code removed rather than covered

Each of these is unreachable, so it is deleted rather than pinned by a test:

  • The isSeries branch of measureColumns in instruments/$instrumentId/table.tsx, which built an INSTRUMENT column for a DataTable that function's own early return means is never rendered.
  • The InstrumentTable toolbar's ?./?? fallbacks for the three columns it declares itself and seeds filters for in initialState.
  • getCategoricalColor's neutral fallback (the index is always below MAX_CATEGORICAL_GROUPS, and every theme holds six hues), binDistribution's second ?? 0 (that row is pre-seeded at zero for every series key), and the export mapping's ?? 'N/A' for a username the records mapping has already defaulted.
  • html2canvas, which renderChartPng replaced, together with its lockfile entries.
  • RecordChartPoint and DistributionRow, exported but used only in their own file.

Validation on e6d8421f8

  • pnpm lint — 34/34, and green again on an immediate second pass. The second pass matters: each workspace's lint is tsc && eslint --fix src, so tsc runs before the fixer and a single green pass says nothing about the file left on disk.
  • pnpm test:coverage — 4,649 passed and 4 expected failures; 100% statements, branches, functions and lines.
  • pnpm knip — clean, apart from the pre-existing turbo in ignoreBinaries hint.
  • pnpm test:e2e — 285/286 locally. The one failure is record-file-deletion.spec.ts, which 503s on record creation because a bare local stack starts no S3/rustfs backend. It passes in CI.
  • CI on e6d8421f8: every check green, including the lint-and-test job that runs lint, the coverage gate and the full Playwright suite.

Validation notes

  • An earlier local e2e run reported seven failures across unrelated specs and eleven tests that did not run. All 43 of those tests pass with --workers=1; it was contention on the shared database under six local workers, not a regression.
  • apps/web/src/route-tree.ts was checked against the routes on disk and is in sync — every instrument hub route is registered. The generator was not run, per the repository rule that the user regenerates it.
  • The replacement chart hues still have no renewed colourblind-separation validation.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Co-Authored-By: GPT-6

🤖 Generated with Claude Code

thomasbeaudry and others added 4 commits October 5, 2026 13:16
Second half of the datahub revamp, on top of the subject hub.

Record provenance: `Session.type` reaches the client on every record, and
the export carries `seriesId` beside a `seriesName` resolved to `en` by
the worker, which has no request language. Every record table gains
COLLECTION_METHOD and SERIES columns.

The instrument hub at `/datahub/instruments` lists one row per edition of
each FORM, INTERACTIVE and FILE instrument, with a detail page holding a
table and a chart over the records of that one edition. Filters live in
the URL and are applied inside the data hook, so the download cannot
disagree with the table.

`summarizeBySubject` is generalised into a private `summarizeRecords(by,
…)` now shared with `summary/by-instrument` and `summary/by-series`; the
subject view alone did not justify the parameter, which is why PR1
shipped it inline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Correctness:

- the kind filter rendered the raw enum, so the menu offered "FORM"
  while the cell beside it said "Form". Both now read one
  `useInstrumentKindLabels` map, keyed over `InstrumentKind` so a new
  kind fails to compile rather than reaching the screen untranslated
- the method filter exempted a record carrying no method, so unchecking
  every method still left rows on screen. `methods` becomes optional,
  matching `series`: absent means every method, and a record with none
  survives the unfiltered view but no explicit selection
- a series' member rows were all tagged FORM; the member's own kind is
  carried through instead
- the scatter chart read its mark ring off the tooltip's background, so
  restyling the tooltip moved the ring. `CHART_SURFACE` names it

Dead code from the WIP snapshot: an export-options ternary after the
early return that made it unreachable, and a doc block attached to
nothing in `tag-colors.ts`.

De-duplication: `useSeriesNames` and `useInstrumentTitles` were the same
four lines twice, so they collapse into `useInstrumentInfoById(params)`,
which carries the kind the member table needed. The two identical
summary mappers share `toInstrumentSummaries`. `COLOUR_BY_OPTIONS` is
now the single source the search param derives from. `InitData` is
renamed `InitInstruments`, since it names only the instruments.

`useSubjectRecordSummaryQuery` and `useInstrumentRecordSummaryQuery` are
deliberately left alone: the shape they share is the queryOptions-plus-
hook pairing that apps/web/AGENTS.md requires of every prefetched hook.

Tests: the seven new hooks and `InstrumentTable` gain unit tests. The
e2e asserting the hub lists no series is replaced by one that drives
what it now does, since the original passed only because the fixture
group had none.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`TabLink` navigated with `to` alone, so TanStack dropped the current
search params and the newly mounted tab silently widened the selection —
the opposite of what putting the filters in the URL was for.

The e2e that exists to catch this was passing anyway: it asserted the URL
matched /methods=/, and `methods` carried a default of all three
collection methods, so the param was present whether or not the filter
had survived. Making `methods` optional removed the default and exposed
the bug underneath.

Also stops the graph spec reaching the tab through `getPageModel`, whose
exact-URL assertion the route's own search-param defaults fail; it clicks
the tab as a user would instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thomasbeaudry thomasbeaudry changed the title Feat/datahub instrument hub feat: add the instrument hub and record provenance Oct 7, 2026

@gdevenyi gdevenyi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch has merge conflicts with main, so review is on hold.

Please merge or rebase on the current main, resolve the conflicts, run pnpm lint and pnpm test:coverage locally, and push. Review continues when the branch merges cleanly and CI is green.

Reviewed at commit f6d3f82.

thomasbeaudry and others added 6 commits October 16, 2026 15:36
…nt-hub

# Conflicts:
#	apps/api/src/instrument-records/__tests__/instrument-records.service.test.ts
#	apps/api/src/instrument-records/instrument-records.service.ts
#	apps/web/src/routes/_app/datahub/subjects/$subjectId/graph.tsx
…raph

`main` removed `datahub.visualization.selectMeasures` as dead code once the
subject graph stopped using it. The instrument hub's graph route reads it, so
the merge left the key missing and `tsc` rejecting the call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`main` now gates CI on 100% unit coverage, which this branch's new code did not
meet. Adds the missing cases and removes three fallbacks no input can reach.

The scatter tooltip's `formatter` compared the row name against `time`, but
recharts passes the axis name, so the comparison never held and the collection
date rendered as a raw epoch. Its `labelFormatter`, which did the formatting,
is never called for a scatter. The date is now formatted in the row recharts
actually names after the x axis.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…anches

Also drops the export mapping's `?? 'N/A'` for a username the records mapping
has already defaulted, which no input could reach.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the instrument record table's filters, navigation and row actions, the
listing's preview and spreadsheet export, and the remaining chart branches.

Removes three things no input reaches: the `measureColumns` branch building an
INSTRUMENT column for a series, whose page returns the member listing before
that table is ever rendered; the toolbar's fallbacks for columns it declares
itself and seeds filters for; and `html2canvas`, which `renderChartPng`
replaced. Two types exported only for their own file go with them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d urls

Two specs that could not run while the branch conflicted with main:

- The series cell renders a translated "None" rather than an em-dash, which
  `f6d3f8295` changed without the spec following.
- The subject's tab links now carry the chart palette, so a link click lands on
  `…/graph?palette=default` and never passes through a query-free url. A `**`
  glob cannot match that, so the wait is a regex allowing the query string.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants