feat(ddi2xlsform): DDI back to XLSForm, round trips proven on the model (#154)

ARCHITECTURE.md

@@ -149,18 +149,16 @@ The migration runs in phases, each keeping every snapshot byte-identical:
    stores the expression text, not the AST: emitters that need structure
    (the EM transpiler) parse it, and XPath is what XLSForm writes back.
 
-### Why there is no `ddi2xlsform` or `ddi2lstsv`
+### The way back from DDI (`ddi2xlsform`)
 
-**DDI is the terminus of the pipeline graph for now.** It describes a *dataset*, not an *instrument*, and a reverse path needs the whole instrument. #155 plans one.
+A DDI codebook describes a *dataset*, not an *instrument*. A codebook formtransform wrote carries the whole instrument too (#155), so `ddi2xlsform` (#154) turns it back into its form: `src/instrument/fromDdi.ts` reads the standard elements and the `cdl:` notes into the Instrument, and the XLSForm emitter both reverse paths share (`src/xlsform/fromInstrument.ts`) writes the sheets. Any other DDI converts as far as its standard elements go, with a warning for each missing field. The round trip is tested on the model, never on bytes; `src/pipelines/ddi2xlsform/README.md` lists the losses. A `ddi2lstsv` would be `ddi2xlsform` → `xlsform2lstsv` and earns no module of its own.
 
-The canonical `Variable` (`src/ddi/types.ts`) is what survives an emit:
+What a CDL codebook carries:
 
 - **`relevant`, `constraint`, `constraint_message`, `required`** survive (#151). DDI Codebook 2.5 has no expression syntax, so each goes in twice: readable (`<universe>` prose, `<valrng>` for a simple numeric range) and exact, in a typed `<notes type="cdl:…">`. `convention:logicMapping` (`ddiEncoding`) defines the notes. A group's own condition is on its `varGrp`; a variable's `<universe>` states its groups' conditions too.
 - **Groups and order** survive (#152): every group is a `<varGrp>` (a plain one `type="section"`), nested through `@varGrp`, and the `<var>`s and data columns follow the survey.
 - **Every other form field** (#153, `convention:ddiFields`): standard DDI where it has a home (the hint as `postQTxt`, `guidance_hint` as `ivuInstr`, `varFormat/@category` for date and time, `var/@dcml="0"` for integer, `valrng` for a range, `qstn/@seqNo` and `qstn/backward`), else a typed note (`cdl:default`, `cdl:appearance`, `cdl:parameters`, a group's `cdl:hint`, `cdl:exclusive`, `cdl:setting`). A unit test fails when a model field has neither nor a documented loss. Not carried: `calculation` and other unlifted columns, a choice list's name, the `or_other` shorthand as such.
 
-Compare `lstsv2xlsform`, which *is* implemented: a LimeSurvey structure TSV carries `relevance`, `em_validation_q`, `mandatory`, `default` and the `!`/`T` type overrides. It is a form definition in a different dialect, so reversing it is a translation problem. Reversing a codebook without those pieces is a *reconstruction* problem, and they cannot be inferred from it.
-
 ## Development Workflow
 
 ### Adding a New Question Type

README.md

@@ -67,13 +67,15 @@ import {
   xlsformToDdi,
   lstsvToDdi,
   lstsvToXlsform,
+  ddiToXlsform,
 } from '@correlaid/formtransform';
 
 const bytes = await file.arrayBuffer();
 const tsv = await xlsformToLstsv(bytes);
 const xml = xlsformToDdi(bytes);
 const fromLs = lstsvToDdi(tsv);
 const { survey, choices, settings } = lstsvToXlsform(tsv);
+const back = ddiToXlsform(xml); // a codebook, or a <var>/<varGrp> fragment
 ```
 
 Each rejects input outside its subset with a `ConversionError` unless you pass
@@ -124,6 +126,9 @@ formtransform lstsv2ddi survey.tsv -o codebook.xml --data responses.csv
 
 # Convert from LimeSurvey TSV back to XLSForm (emitted as JSON sheets)
 formtransform lstsv2xlsform survey.tsv -o recovered.json
+
+# Convert a DDI codebook (or a <var>/<varGrp> fragment) back to XLSForm (JSON sheets)
+formtransform ddi2xlsform codebook.xml -o form.json
 ```
 
 ### Asking the library what exists
@@ -183,21 +188,22 @@ records how each maps onto it.
 | [LimeSurvey TSV](https://www.limesurvey.org/manual/Tab_Separated_Value_survey_structure) | **Deployment** — recreate the survey in LimeSurvey | type codes (`L`, `M`, `F`, `N`) |
 | [DDI Codebook 2.5](https://ddialliance.org/Specification/DDI-Codebook/2.5/) | **Documentation** — describe the resulting dataset | interval class + response domains (`category`, `multiple`) |
 
-**Supported directions:** four, one per module under `src/pipelines/` —
+**Supported directions:** five, one per module under `src/pipelines/` —
 `xlsform2lstsv` (deploy the survey), `xlsform2ddi` (document the dataset),
-`lstsv2ddi` and `lstsv2xlsform` (the reverse paths). All are lossy for some
+`lstsv2ddi`, `lstsv2xlsform` and `ddi2xlsform` (the reverse paths). All are lossy for some
 types: nested groups flatten in LimeSurvey, choice codes over 5 chars truncate,
 `select_multiple` becomes N binary variables, and the reverse paths cannot
 recover a select's authored `list_name`.
 
-**DDI has no way back yet:** there is no `ddi2xlsform` or `ddi2lstsv`. A CDL
-codebook carries skip logic, validation and `required`: each condition as a
-readable `<universe>` sentence (and a simple numeric range as `<valrng>`), plus
-the exact expression in a typed note such as `<notes type="cdl:relevant"
-subject="xlsform-xpath">` (`convention:logicMapping`). Groups, order, hints,
-defaults, appearances and parameters are in it too, in standard DDI where it
-has a place and typed notes where not (`convention:ddiFields`). The reverse
-parser that reads them back is #154.
+**DDI goes back to XLSForm:** a CDL codebook carries the whole form. Skip
+logic, validation and `required` are each a readable `<universe>` sentence
+(a simple numeric range also `<valrng>`), plus the exact expression in a typed
+note such as `<notes type="cdl:relevant" subject="xlsform-xpath">`
+(`convention:logicMapping`). Groups, order, hints, defaults, appearances and
+parameters are in standard DDI where it has a place and typed notes where not
+(`convention:ddiFields`). `ddiToXlsform` reads it back; any other DDI converts
+as far as its standard elements go, with a warning per missing field
+([`src/pipelines/ddi2xlsform/README.md`](src/pipelines/ddi2xlsform/README.md)).
 
 ## Errors and warnings
 

codegen/schematron.py

@@ -33,7 +33,20 @@ def rdt_of(slug: str) -> str | None:
     companion_type = other.get("companionType", "text")
     comp_ddi = registry.get(f"type:{companion_type}", {}).get("ddi", {})
 
+    # The `cdl:` note vocabulary (convention:logicMapping ddiEncoding, convention:ddiFields):
+    # each type, and the @subject a type requires (a fixed value, or `True` for "any").
+    logic = registry.get("convention:logicMapping", {}).get("rule", {}).get("ddiEncoding", {})
+    fields = registry.get("convention:ddiFields", {}).get("rule", {})
+    note_specs = [*logic.get("notes", {}).values(), *fields.get("notes", {}).values()]
+    note_types = sorted({n["type"] for n in note_specs})
+    subject = logic.get("noteSubject")
+    note_subjects = {
+        n["type"]: (n["subject"] if n.get("subject") == subject else True) for n in note_specs if n.get("subject")
+    }
+
     return {
+        "note_types": note_types,
+        "note_subjects": note_subjects,
         "cat_rdts": cat_rdts,
         "vg_types": vg_types,
         "category_rdt": rdt_of("select_one"),
@@ -175,11 +188,43 @@ def generate_schematron(registry: dict[str, Any], output: Path) -> None:
     </rule>
 """
 
+    note_test = " or ".join(f"@type = '{t}'" for t in f["note_types"])
+    note_msg = ", ".join(f["note_types"])
+    subject_rules = "\n".join(
+        (
+            f"        <assert test=\"not(@type = '{t}') or @subject = '{subj}'\">"
+            f'A {t} note needs subject="{subj}": its text is an expression in that syntax.</assert>'
+        )
+        if subj is not True
+        else (
+            f"        <assert test=\"not(@type = '{t}') or normalize-space(@subject) != ''\">"
+            f"A {t} note needs a subject: the name of what it holds.</assert>"
+        )
+        for t, subj in sorted(f["note_subjects"].items())
+    )
+    cdl_notes = f"""\
+    <rule context="%P%notes[starts-with(@type, 'cdl:')]">
+        <assert test="{note_test}">Note type "<value-of select="@type"/>" is not in the CDL vocabulary ({note_msg}).</assert>
+{subject_rules}
+    </rule>
+"""
+    # At most one note of each cdl: type per element and language (per subject on stdyDscr).
+    cdl_note_uniqueness = """\
+    <rule context="%P%var | %P%varGrp">
+        <assert test="every $t in distinct-values(%P%notes[starts-with(@type, 'cdl:')]/@type) satisfies every $l in distinct-values(%P%notes[@type = $t]/string(@xml:lang)) satisfies count(%P%notes[@type = $t][string(@xml:lang) = $l]) &lt;= 1"><value-of select="@name"/> has more than one note of one cdl: type in one language.</assert>
+    </rule>
+    <rule context="%P%stdyDscr">
+        <assert test="every $s in distinct-values(%P%notes[@type = 'cdl:setting']/@subject) satisfies count(%P%notes[@type = 'cdl:setting'][@subject = $s]) &lt;= 1">A setting has more than one cdl:setting note.</assert>
+    </rule>
+"""
+
     patterns = [
         ("uniqueness", uniqueness),
         ("essentials", essentials),
         ("logic", logic),
         ("other_variables", other_variables),
+        ("cdl_notes", cdl_notes),
+        ("cdl_note_uniqueness", cdl_note_uniqueness),
     ]
 
     out: list[str] = [

ddi-validation/schematron/ddi_custom_rules.sch

@@ -214,4 +214,34 @@
     </rule>
     </pattern>
 
+    <pattern id="cdl_notes">
+    <rule context="ddi:notes[starts-with(@type, 'cdl:')]">
+        <assert test="@type = 'cdl:appearance' or @type = 'cdl:constraint' or @type = 'cdl:constraint_message' or @type = 'cdl:default' or @type = 'cdl:exclusive' or @type = 'cdl:hint' or @type = 'cdl:parameters' or @type = 'cdl:relevant' or @type = 'cdl:required' or @type = 'cdl:setting'">Note type "<value-of select="@type"/>" is not in the CDL vocabulary (cdl:appearance, cdl:constraint, cdl:constraint_message, cdl:default, cdl:exclusive, cdl:hint, cdl:parameters, cdl:relevant, cdl:required, cdl:setting).</assert>
+        <assert test="not(@type = 'cdl:constraint') or @subject = 'xlsform-xpath'">A cdl:constraint note needs subject="xlsform-xpath": its text is an expression in that syntax.</assert>
+        <assert test="not(@type = 'cdl:relevant') or @subject = 'xlsform-xpath'">A cdl:relevant note needs subject="xlsform-xpath": its text is an expression in that syntax.</assert>
+        <assert test="not(@type = 'cdl:setting') or normalize-space(@subject) != ''">A cdl:setting note needs a subject: the name of what it holds.</assert>
+    </rule>
+    <rule context="notes[starts-with(@type, 'cdl:')]">
+        <assert test="@type = 'cdl:appearance' or @type = 'cdl:constraint' or @type = 'cdl:constraint_message' or @type = 'cdl:default' or @type = 'cdl:exclusive' or @type = 'cdl:hint' or @type = 'cdl:parameters' or @type = 'cdl:relevant' or @type = 'cdl:required' or @type = 'cdl:setting'">Note type "<value-of select="@type"/>" is not in the CDL vocabulary (cdl:appearance, cdl:constraint, cdl:constraint_message, cdl:default, cdl:exclusive, cdl:hint, cdl:parameters, cdl:relevant, cdl:required, cdl:setting).</assert>
+        <assert test="not(@type = 'cdl:constraint') or @subject = 'xlsform-xpath'">A cdl:constraint note needs subject="xlsform-xpath": its text is an expression in that syntax.</assert>
+        <assert test="not(@type = 'cdl:relevant') or @subject = 'xlsform-xpath'">A cdl:relevant note needs subject="xlsform-xpath": its text is an expression in that syntax.</assert>
+        <assert test="not(@type = 'cdl:setting') or normalize-space(@subject) != ''">A cdl:setting note needs a subject: the name of what it holds.</assert>
+    </rule>
+    </pattern>
+
+    <pattern id="cdl_note_uniqueness">
+    <rule context="ddi:var | ddi:varGrp">
+        <assert test="every $t in distinct-values(ddi:notes[starts-with(@type, 'cdl:')]/@type) satisfies every $l in distinct-values(ddi:notes[@type = $t]/string(@xml:lang)) satisfies count(ddi:notes[@type = $t][string(@xml:lang) = $l]) &lt;= 1"><value-of select="@name"/> has more than one note of one cdl: type in one language.</assert>
+    </rule>
+    <rule context="ddi:stdyDscr">
+        <assert test="every $s in distinct-values(ddi:notes[@type = 'cdl:setting']/@subject) satisfies count(ddi:notes[@type = 'cdl:setting'][@subject = $s]) &lt;= 1">A setting has more than one cdl:setting note.</assert>
+    </rule>
+    <rule context="var | varGrp">
+        <assert test="every $t in distinct-values(notes[starts-with(@type, 'cdl:')]/@type) satisfies every $l in distinct-values(notes[@type = $t]/string(@xml:lang)) satisfies count(notes[@type = $t][string(@xml:lang) = $l]) &lt;= 1"><value-of select="@name"/> has more than one note of one cdl: type in one language.</assert>
+    </rule>
+    <rule context="stdyDscr">
+        <assert test="every $s in distinct-values(notes[@type = 'cdl:setting']/@subject) satisfies count(notes[@type = 'cdl:setting'][@subject = $s]) &lt;= 1">A setting has more than one cdl:setting note.</assert>
+    </rule>
+    </pattern>
+
 </schema>

package.json

@@ -78,6 +78,7 @@
     "concurrently": "^10.0.3",
     "esbuild": "^0.28.1",
     "eslint": "^10.7.0",
+    "fast-check": "^4.10.2",
     "globals": "^17.7.0",
     "husky": "^9.1.7",
     "jscpd": "^5.0.12",

scripts/bless-survey-snapshots.mjs

@@ -1,7 +1,7 @@
 #!/usr/bin/env node
 /**
  * Re-bless the frozen whole-survey snapshots
- * (tests/fixtures/surveys/<name>/{tsv.tsv,ddi.xml}) using the locally built
+ * (tests/fixtures/surveys/<name>/{tsv.tsv,ddi.xml,ddi2xlsform.json}) using the locally built
  * library (dist/index.js).
  *
  * The per-entity snapshots under registry/entities/ pin *one question type* at a
@@ -30,9 +30,8 @@ if (!fs.existsSync(entry)) {
   console.error('dist/index.js missing — run `npm run build` first.');
   process.exit(1);
 }
-const { XLSFormToTSVConverter, XLSLoader, buildDdiXml } = await import(
-  pathToFileURL(entry).href
-);
+const { XLSFormToTSVConverter, XLSLoader, buildDdiXml, ddiToXlsform } =
+  await import(pathToFileURL(entry).href);
 
 // Fixed so a re-bless on another day is not a diff (the snapshot test scrubs it
 // too, but keeping the file stable makes `git diff` mean something).
@@ -96,7 +95,14 @@ for (const name of dirs) {
     });
     fs.writeFileSync(path.join(dir, 'ddi.xml'), ddi);
 
-    console.log(`blessed ${name} → tsv.tsv + ddi.xml`);
+    // And back (#154): the form ddi2xlsform gives, which pyxform validates.
+    const back = ddiToXlsform(ddi, { onWarning: () => {} });
+    fs.writeFileSync(
+      path.join(dir, 'ddi2xlsform.json'),
+      JSON.stringify(back, null, 2) + '\n',
+    );
+
+    console.log(`blessed ${name} → tsv.tsv + ddi.xml + ddi2xlsform.json`);
     written++;
   } catch (e) {
     // testA carries unimplemented types on purpose; a fixture that cannot be

scripts/bless.sh

@@ -5,7 +5,7 @@
 #   npm run bless -- tsv      # registry/entities/<slug>/tsv.tsv
 #   npm run bless -- ddi      # registry/entities/<slug>/ddi.xml
 #   npm run bless -- examples # meta.json + xlsform.xlsx (codegen)
-#   npm run bless -- surveys  # tests/fixtures/surveys/<name>/{tsv.tsv,ddi.xml}
+#   npm run bless -- surveys  # tests/fixtures/surveys/<name>/{tsv.tsv,ddi.xml,ddi2xlsform.json}
 #   npm run bless -- responses# tests/live/limesurvey/expected/ (needs docker)
 #
 # Snapshots are a manual gate: default codegen never touches ddi.xml/tsv.tsv, so

src/api.ts

@@ -18,6 +18,11 @@ import type { ChoiceRow, SettingsRow, SurveyRow } from './xlsform/types.js';
 import { XLSFormToTSVConverter } from './pipelines/xlsform2lstsv/index.js';
 import { buildDdiXml } from './pipelines/xlsform2ddi/index.js';
 import { lstsvToDdiXml } from './pipelines/lstsv2ddi/index.js';
+import {
+  ddiToXlsform as ddiToXlsformSheets,
+  type DdiToXlsformOptions,
+  type XlsformOutput,
+} from './pipelines/ddi2xlsform/index.js';
 import type { LstsvToDdiOptions } from './pipelines/lstsv2ddi/index.js';
 
 /** An XLSForm: the `.xlsx` bytes, or its loaded sheets. */
@@ -157,6 +162,18 @@ export function xlsformToDdi(
   });
 }
 
+/**
+ * DDI-Codebook 2.5 XML (a codebook or a fragment) → XLSForm sheets
+ * `{ survey, choices, settings }` (#154). Never refuses DDI it can read:
+ * each field it can't supply is a warning.
+ */
+export function ddiToXlsform(
+  xml: string,
+  options: DdiToXlsformOptions = {},
+): XlsformOutput {
+  return ddiToXlsformSheets(xml, { onWarning: onceEach(options.onWarning) });
+}
+
 /** LimeSurvey structure TSV → DDI-Codebook 2.5 XML. */
 export function lstsvToDdi(
   tsv: string,