DDI round trips: a CDL codebook that converts back to the same form

[jstet]

Goal: a DDI Codebook that formtransform writes carries the whole form, so XLSForm → DDI → XLSForm gives back the same form. "Same" is judged on the Instrument model, not on bytes.

Approach (B: every field in the DDI itself)

The DDI stays the single source of truth. Everything the form needs goes into standard DDI elements where one fits, and otherwise into CDL-typed <notes>. The DDI 2.5 XSD provides type and subject on <notes> for exactly this kind of controlled vocabulary. An edit made in qwacback (concept tags, merged studies) therefore flows back into the form.

Rejected: embedding the whole form as JSON in one note (A). It's exact by construction, but it creates a second source of truth that drifts as soon as qwacback edits the DDI.

Parts

1. Convention: carry relevant/constraint in DDI as typed <notes> plus <universe> prose #151: relevant / constraint / constraint_message as typed notes plus <universe> prose. See the comment there for corrections.
2. DDI round trips: keep question order, every group, and notes apart from hints #152: survey order, a <varGrp> for every group, notes kept apart from hints.
3. DDI round trips: every form field in standard DDI where possible, cdl: notes only for the rest #153: every other field as a cdl: note (type, required, default, appearance, parameters, list name, settings), plus a registry guard that no Instrument field is unmapped.
4. DDI round trips: DDI → Instrument parser, ddi2xlsform, and a round-trip contract #154: DDI → Instrument parser, ddi2xlsform, the round-trip contract test (fixtures + generated forms), Schematron rules for the vocabulary, and the ARCHITECTURE.md scope change.

1–3 are independent emitter changes, each useful on its own for codebook readers. 4 needs them.

Limits

• Only DDI that formtransform wrote round-trips. Any other DDI converts best-effort, with a warning per missing field.
• Byte-identical round trips are not a goal.

Decisions for qwacback

• The note vocabulary (Convention: carry relevant/constraint in DDI as typed <notes> plus <universe> prose #151 first).
• Whether qwacback's DDI → XLSForm export calls formtransform's ddi2xlsform through the ddi-emitter sidecar instead of keeping its Go converter.
• The Schematron change on count(notes) <= 1.

[jstet]

qwacback's decisions:

1. Note vocabulary: Convention: carry relevant/constraint in DDI as typed <notes> plus <universe> prose #151's shape with the corrections is confirmed. See the comment there, including the type="section" request for DDI round trips: keep question order, every group, and notes apart from hints #152.
2. ddi2xlsform: yes. qwacback will remove its Go DDIToXLSForm completely and call formtransform's ddi2xlsform through the ddi-emitter sidecar, so formtransform alone owns the DDI ↔ XLSForm mapping in both directions. What qwacback needs from DDI round trips: DDI → Instrument parser, ddi2xlsform, and a round-trip contract #154:
   • Input shapes: qwacback's XLSForm endpoints convert a single question (<var>, <varGrp> or <dataDscr> fragment) as well as a whole study (<codeBook>). Either ddi2xlsform accepts fragments, or qwacback wraps them into a minimal codebook. Please say which you prefer.
   • Non-CDL DDI: qwacback's seed studies are hand-written DDI without cdl: notes. Best-effort conversion with warnings is fine, but it must not refuse them.
   • Output: the sheets as JSON (survey, choices, settings rows), which qwacback returns as-is, not only .xlsx.
   • Warnings: returned, so qwacback can pass them to the API client.
   • Before switching, qwacback will compare ddi2xlsform's output with the Go converter on its seed studies, then delete the Go code.
3. Schematron count(notes): count(notes[not(@type)]) <= 1 per xml:lang is fine for qwacback's seed studies.

qwacback's side (storing <universe>/typed notes/section groups, then switching the export) will be tracked in a qwacback issue.

[jstet]

Principle, from the maintainer: standard DDI wherever possible. cdl: notes only where DDI 2.5 has no home for a field. #153 now maps each field that way, using standard slots: var/@dcml, varFormat/@category, valrng/range, qstn/@seqNo, qstn/backward, varGrp type="section" + universe, and possibly postQTxt for hints. What remains as cdl:: the relevant / constraint XPath, constraint_message, required, default, appearance, a range's step, the style setting, and possibly the or_other shorthand. qwacback's answers are recorded on #152 (section) and #154 (fragments, non-CDL DDI, JSON output, warnings).

[jstet]

All four parts are merged: #151 (logic, PR #156), #152 (structure, PR #157), #153 (fields, PR #158, breaking: the hint moved from preQTxt to postQTxt), and #154 (ddi2xlsform plus round-trip contracts, PR #159). A codebook formtransform writes now gives back its form, checked on the Instrument model for every fixture and for fast-check-generated forms. The known losses are listed in src/pipelines/ddi2xlsform/README.md. It isn't released yet.