Skip to content

feat(ddi2xlsform): DDI back to XLSForm, round trips proven on the model (#154) - #159

Merged
jstet merged 1 commit into
mainfrom
feat/154-ddi2xlsform
Sep 27, 2026
Merged

jstet merged 1 commit into
mainfrom
feat/154-ddi2xlsform

Conversation

@jstet

@jstet jstet commented Sep 27, 2026

Copy link
Copy Markdown
Member

Closes #154, and with it #155 (#151 logic, #152 structure and #153 fields are merged).

ddi2xlsform

const { survey, choices, settings } = ddiToXlsform(xml, { onWarning });

CLI: formtransform ddi2xlsform codebook.xml -o form.json.

What qwacback asked for (qwacback#37):

Requirement How it's met
Fragments <codeBook>, <dataDscr> or bare <var> / <varGrp> elements work, with no wrapping needed. A varGrp pointing outside the fragment gives ddi-reference-outside, not an error.
Non-CDL DDI Never refused; it's read as far as its standard elements go. Without seqNo or any cdl: note, it gets one ddi-field-missing warning for each field only CDL carries. An unknown responseDomainType becomes text (ddi-type-unknown). Only XML that isn't well-formed throws (ddi-invalid).
Output The { survey, choices, settings } JSON sheets, the shape lstsvToXlsform already returns. Warnings go through onWarning as Diagnostic[].
Standard DDI first Standard elements are read first, cdl: notes only where DDI has no element.
Browser src/utils/xmlParse.ts is a small XML reader with no dependency (no DOMParser, no Node). The browser-bundle test runs it.

Round trips, on the model

  • tests/ts/contract/ddiRoundtrip.test.ts runs every survey fixture and registry entity (29 cases) through two paths: XLSForm → DDI → Instrument, and XLSForm → DDI → XLSForm sheets → Instrument. In both, the result must equal the original Instrument up to the documented losses (canonicalInstrument.ts). A control test proves the comparison catches a dropped relevant, constraint or label.
  • ddiRoundtripGenerated.test.ts does the same with fast-check: 200 random forms per run, covering every simple type, selects with or_other and exclusive choices, notes, nested groups, grids, 1–2 languages, relevant, constraint, required, hints, appearances and parameters. It also held at 2000 runs locally.
  • Each survey's output is blessed as ddi2xlsform.json, pinned by a snapshot test, and validated with pyxform.
  • The known losses are listed in src/pipelines/ddi2xlsform/README.md:
    • list names,
    • the or_other shorthand (it comes back as the explicit pair),
    • note names and positions,
    • rows DDI has no variable for,
    • language display names,
    • defaults written as defaults,
    • choice columns and settings that aren't carried.

Found and fixed by the round trip

  • A note before a semi-open question (the select plus its _other text) was dropped from the DDI.
  • A per-language form_title was written as titl = [object Object]. It is now titl in the base language, plus parTitl xml:lang="…" (DDI's parallel title) for each other language.

Shared XLSForm emitter

  • The Instrument → XLSForm emitter moves from lstsv2xlsform to src/xlsform/fromInstrument.ts, so both reverse paths share it; pipelines may not import each other.
  • It now keeps group names, grid members' own fields, each shared list once, exclusive choices from the choice rows, form_id / version, and per-language titles.
  • lstsv2xlsform changes as a result:
    • A LimeSurvey group description comes back as the group's hint.
    • A group name that's already a valid XLSForm name is kept (Later) instead of lowercased. Any other is slugified and made unique, which also keeps lstsv2ddi's section IDs valid.

Schematron

New generated rules for the cdl: vocabulary:

  • only the registry's note types are allowed,
  • cdl:relevant and cdl:constraint need subject="xlsform-xpath", and cdl:setting needs a subject,
  • at most one note per type and language on each element (per subject on stdyDscr).

Each rule has a mutation test.

Docs

  • ARCHITECTURE and src/pipelines/README.md: "Why there is no ddi2xlsform" is now "The way back from DDI".
  • README: the directions, the API and a CLI example.
  • A new src/pipelines/ddi2xlsform/README.md.

fast-check is added as a dev dependency.

Checks

  • vitest: 1346 passed.
  • pytest validation and codegen: 147 passed, 3 xfailed (unchanged).
  • The schematron-worker tests (gradle) pass.
  • npm run validate and the drift check are clean.

🤖 Generated with Claude Code

…el (#154)

- `ddiToXlsform(xml, { onWarning })` / `formtransform ddi2xlsform`: a DDI
  codebook, or a <dataDscr>/<var>/<varGrp> fragment, → `{ survey, choices,
  settings }`. `src/instrument/fromDdi.ts` reads standard DDI first, then the
  cdl: notes; `src/utils/xmlParse.ts` is a dependency-free XML reader, so it
  runs in the browser bundle too.
- Never refuses DDI it can read: a non-CDL codebook gets one
  `ddi-field-missing` warning per field only CDL carries, an unknown
  responseDomainType is text (`ddi-type-unknown`), a fragment's reference
  outside it is `ddi-reference-outside`. Broken XML is `ddi-invalid`.
- The Instrument → XLSForm emitter moves to `src/xlsform/fromInstrument.ts`
  and is shared with lstsv2xlsform: groups keep their names, grid members
  their own fields, a shared list is written once, exclusive choices come
  from the choice rows too, form_id/version/per-language titles are kept.
  lstsv2xlsform: a LimeSurvey group's description comes back as its hint,
  and a group name that is a valid XLSForm name is kept (`Later`), else
  slugified and made unique.
- Round trips: every survey fixture and registry entity, and 200 random
  forms (fast-check), give back their Instrument after XLSForm → DDI, and
  after XLSForm → DDI → XLSForm sheets, up to the losses
  src/pipelines/ddi2xlsform/README.md lists. Each survey's output is
  blessed as ddi2xlsform.json and validated with pyxform.
- Schematron: the cdl: note vocabulary (known types, the subject each needs,
  one note per type and language), generated from the registry.
- Emitter fixes the round trip found: a note before a semi-open question
  was dropped, and a per-language form_title was written as
  "[object Object]"; it is now titl plus parTitl per language.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jstet
jstet merged commit b0148ef into main Sep 27, 2026
4 checks passed
@jstet
jstet deleted the feat/154-ddi2xlsform branch September 27, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DDI round trips: DDI → Instrument parser, ddi2xlsform, and a round-trip contract

1 participant