Skip to content

feat(export): analysis files in the study ZIP (JSONL + CSV for AI-assisted analysis) - #101

Merged
linxule merged 5 commits into
mainfrom
feat/analysis-export
Oct 10, 2026
Merged

linxule merged 5 commits into
mainfrom
feat/analysis-export

Conversation

@linxule

@linxule linxule commented Oct 10, 2026

Copy link
Copy Markdown
Owner

For researchers who analyse exports with Claude Code or another AI tool, or in a spreadsheet (Yinghua's use case). Every study ZIP, single-study and all-studies, on Node and Cloudflare, now also contains analysis/:

  • interviews.jsonl: one object per interview (openinterviewer.analysis.v1) with study context from the interview's own collection snapshot, consent/provider provenance, analysis, profile fields with their historical labels, and turns with exact text. This is the file to quote from.
  • interviews.csv: one row per interview; one profile:<fieldId> column per field seen.
  • profile_fields.csv: long form, safe across revisions.
  • turns/NNN.csv: one per interview. A single turns table could reach ~830 MiB at the 500-interview cap, too much for a Worker.
  • README.md: a data dictionary, plus the Markdown export's consent/provider warning from the same constant.

Details

  • CSV format: UTF-8 BOM, CRLF line endings, text cells through csvCell. Numbers stay numbers, a fix from the Kimi review.
  • Turn numbering: turn indexes match the analysis citation numbering.
  • Dates: the per-interview Markdown in the ZIP now uses ISO UTC instead of the server's locale.

Cloudflare path. The ZIP writer takes chunked entries, with incremental CRC and size checks. The Worker streams the combined files by re-walking the same export snapshot; every page read re-checks row fingerprints, and verifyExportSequence still runs before the closing records. The cost is about 5× snapshot reads per export, bounded.

Not included: no adapter for third-party pipelines. shipaleks/ux-research-pipeline is "all rights reserved"; its documented input is a per-interview JSON array that a researcher can derive from interviews.jsonl.

Reviews: Kimi approved, no blockers. Accepted: numbers in CSV. Noted, not changed: the 5× read cost, and the replay identity check relying on the Durable page fingerprints.

npm run check:cloudflare: 20/20 lanes on a clean tree at the head commit, including the artifact journey that now asserts the analysis entries through the built Worker.

🤖 Generated with Claude Code

linxule and others added 5 commits October 10, 2026 15:28
interviews.jsonl (exact text, the quotation source), interviews.csv,
profile_fields.csv and per-interview turn CSVs, plus a README that repeats
the Markdown export's consent/provider warning from the shared constant.
CSVs: UTF-8 BOM, CRLF, every cell through csvCell. Turn indexes match the
analysis citation numbering.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…orts

The ZIP writer accepts chunked entries with incremental CRC and size checks,
so the Worker streams the combined files instead of holding rows; turns are
sharded per interview (a single table's bound is ~830 MiB at the 500-interview
cap). The Durable path rereads the same export sequence and still verifies it
before the closing records. Per-interview Markdown dates are now ISO UTC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The formula guard prefixed an apostrophe to negative numbers (offset_s can be
negative), so spreadsheets read them as text. A finite number cannot carry a
formula; text cells are still guarded. From the Kimi review.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… files

The ST-08 journey counted every .md in the archive; analysis/README.md is now
one more. Assert the analysis entries exist instead, with one turn shard and
one JSONL line per interview, through the built Worker.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@linxule
linxule force-pushed the feat/analysis-export branch from 46a5f67 to 18a94ed Compare October 10, 2026 13:29
@linxule
linxule merged commit c597b35 into main Oct 10, 2026
6 checks passed
@linxule
linxule deleted the feat/analysis-export branch October 10, 2026 13:40
menotbobbybrown pushed a commit to menotbobbybrown/openinterviewer that referenced this pull request Oct 11, 2026
On-device voice input (linxule#100), analysis files in the study export (linxule#101),
OpenNext 1.20.10 / wrangler 4.148.0 with Next held at 16.3 (linxule#98), provider
SDKs (linxule#95). Code-only release; rollback to 5.3 refuses device-mode studies.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant