feat(ddi): skip logic and validation in the codebook (#151)

ARCHITECTURE.md

@@ -151,15 +151,14 @@ The migration runs in phases, each keeping every snapshot byte-identical:
 
 ### Why there is no `ddi2xlsform` or `ddi2lstsv`
 
-Deliberate design decision. **DDI is the terminus of the pipeline graph** — it describes a *dataset*, not an *instrument*, so it does not carry the information a survey needs to run.
+**DDI is the terminus of the pipeline graph for now.** It describes a *dataset*, not an *instrument*, and a reverse path needs the whole instrument. #155 plans one.
 
-The canonical `Variable` (`src/ddi/types.ts`) is what survives an emit. Everything that makes a form behave is absent:
+The canonical `Variable` (`src/ddi/types.ts`) is what survives an emit:
 
-- **no `relevant`** — DDI Codebook 2.5 has no machine-readable expression syntax at all, so skip logic is dropped on the way in
-- **no `constraint`** — same reason
-- **no `required`, `default`, per-question `appearance`, `calculation`** (a question's `hint` and `guidance_hint` do survive, as `<preQTxt>` and `<ivuInstr>`)
+- **`relevant`, `constraint`, `constraint_message`, `required`** survive (#151). DDI Codebook 2.5 has no expression syntax, so each goes in twice: readable (`<universe>` prose, `<valrng>` for a simple numeric range) and exact, in a typed `<notes type="cdl:…">`. `convention:logicMapping` (`ddiEncoding`) defines the notes. Until every group has a `varGrp` (#152), a group's condition is ANDed into each variable's own.
+- **no `default`, per-question `appearance`, `calculation`, plain group structure** yet (#152, #153). A question's `hint` and `guidance_hint` do survive, as `<preQTxt>` and `<ivuInstr>`.
 
-Compare `lstsv2xlsform`, which *is* implemented: a LimeSurvey structure TSV carries `relevance`, `em_validation_q`, `mandatory`, `default` and the `!`/`T` type overrides. It is a form definition in a different dialect, so reversing it is a translation problem. Reversing DDI is not — it is a *reconstruction* problem, and the missing pieces cannot be inferred from a codebook.
+Compare `lstsv2xlsform`, which *is* implemented: a LimeSurvey structure TSV carries `relevance`, `em_validation_q`, `mandatory`, `default` and the `!`/`T` type overrides. It is a form definition in a different dialect, so reversing it is a translation problem. Reversing a codebook without those pieces is a *reconstruction* problem, and they cannot be inferred from it.
 
 ## Development Workflow
 

README.md

@@ -190,10 +190,13 @@ types: plain/nested groups flatten, choice codes over 5 chars truncate,
 `select_multiple` becomes N binary variables, and the reverse paths cannot
 recover a select's authored `list_name` or tell `integer` from `decimal`.
 
-**DDI is the terminus:** there is no `ddi2xlsform` or `ddi2lstsv`, by design. A
-codebook describes a *dataset*, not an *instrument* — it carries no relevance,
-constraint, required, default or appearance, so reversing it would emit a survey
-that looks right and behaves wrongly.
+**DDI has no way back yet:** there is no `ddi2xlsform` or `ddi2lstsv`. A CDL
+codebook carries skip logic, validation and `required`: each condition as a
+readable `<universe>` sentence (and a simple numeric range as `<valrng>`), plus
+the exact expression in a typed note such as `<notes type="cdl:relevant"
+subject="xlsform-xpath">` (`convention:logicMapping`). Defaults, appearances
+and the group structure are not in it yet, so a reversed form would still look
+right and behave wrongly. #155 tracks the rest.
 
 ## Errors and warnings
 

codegen/schematron.py

@@ -83,7 +83,7 @@ def generate_schematron(registry: dict[str, Any], output: Path) -> None:
         <assert test="%P%varFormat">Variable <value-of select="@name"/> is missing technical format (varFormat).</assert>
         <assert test="%P%concept[normalize-space(.) != '']">Variable <value-of select="@name"/> is missing a concept element.</assert>
         <assert test="not(%P%labl)">Variable <value-of select="@name"/> uses labl — use concept instead. labl is only for catgry elements.</assert>
-        <assert test="count(%P%notes) &lt;= 1">Variable <value-of select="@name"/> has multiple notes elements. Only one notes element per variable is allowed.</assert>
+        <assert test="every $l in distinct-values(%P%notes[not(@type)]/string(@xml:lang)) satisfies count(%P%notes[not(@type)][string(@xml:lang) = $l]) &lt;= 1">Variable <value-of select="@name"/> has more than one untyped notes element in one language. Only one untyped note per language is allowed; typed notes (convention:logicMapping, e.g. type="cdl:relevant") are not limited.</assert>
     </rule>
 
     <!-- Variable group essentials -->

ddi-validation/schematron/ddi_custom_rules.sch

@@ -29,7 +29,7 @@
         <assert test="ddi:varFormat">Variable <value-of select="@name"/> is missing technical format (varFormat).</assert>
         <assert test="ddi:concept[normalize-space(.) != '']">Variable <value-of select="@name"/> is missing a concept element.</assert>
         <assert test="not(ddi:labl)">Variable <value-of select="@name"/> uses labl — use concept instead. labl is only for catgry elements.</assert>
-        <assert test="count(ddi:notes) &lt;= 1">Variable <value-of select="@name"/> has multiple notes elements. Only one notes element per variable is allowed.</assert>
+        <assert test="every $l in distinct-values(ddi:notes[not(@type)]/string(@xml:lang)) satisfies count(ddi:notes[not(@type)][string(@xml:lang) = $l]) &lt;= 1">Variable <value-of select="@name"/> has more than one untyped notes element in one language. Only one untyped note per language is allowed; typed notes (convention:logicMapping, e.g. type="cdl:relevant") are not limited.</assert>
     </rule>
 
     <!-- Variable group essentials -->
@@ -54,7 +54,7 @@
         <assert test="varFormat">Variable <value-of select="@name"/> is missing technical format (varFormat).</assert>
         <assert test="concept[normalize-space(.) != '']">Variable <value-of select="@name"/> is missing a concept element.</assert>
         <assert test="not(labl)">Variable <value-of select="@name"/> uses labl — use concept instead. labl is only for catgry elements.</assert>
-        <assert test="count(notes) &lt;= 1">Variable <value-of select="@name"/> has multiple notes elements. Only one notes element per variable is allowed.</assert>
+        <assert test="every $l in distinct-values(notes[not(@type)]/string(@xml:lang)) satisfies count(notes[not(@type)][string(@xml:lang) = $l]) &lt;= 1">Variable <value-of select="@name"/> has more than one untyped notes element in one language. Only one untyped note per language is allowed; typed notes (convention:logicMapping, e.g. type="cdl:relevant") are not limited.</assert>
     </rule>
 
     <!-- Variable group essentials -->

registry/conventions/logicMapping.jsonld

@@ -30,7 +30,7 @@
       "@type": "GlobalConvention",
       "skos:prefLabel": "Relevance & validation logic: XLSForm ↔ LimeSurvey ↔ DDI",
       "rule": {
-        "description": "XLSForm expresses survey logic in an XPath subset; LimeSurvey uses ExpressionScript (EM). The transformation transpiles between them (implemented in src/pipelines/xlsform2lstsv/xpathTranspiler.ts). DDI Codebook has no machine-readable expression syntax — logic fields are dropped when emitting DDI and tools MUST report this as loss.",
+        "description": "XLSForm expresses survey logic in an XPath subset; LimeSurvey uses ExpressionScript (EM). The transformation transpiles between them (implemented in src/pipelines/xlsform2lstsv/xpathTranspiler.ts). DDI Codebook 2.5 has no expression syntax of its own, so a CDL codebook carries the logic in two parts (formtransform#151): readable prose in the standard element (`universe`, `valrng`) and the expression itself in a typed `<notes>`, the extension point the DDI XSD provides (\"The attributes for notes permit a controlled vocabulary to be developed ('type' and 'subject')\"). See ddiEncoding.",
         "fields": [
           {
             "concept": "relevance (conditional display)",
@@ -39,9 +39,9 @@
             "lsTsvColumn": "relevance",
             "lsSyntax": "ExpressionScript (EM)",
             "ddi": {
-              "closestElement": "var/universe",
-              "note": "DDI universe describes the subpopulation as prose, not as machine syntax. Currently NOT emitted — relevance is dropped in DDI output.",
-              "lossy": true
+              "element": "var/universe[@clusion='I'] + var/notes[@type='cdl:relevant']",
+              "note": "The universe is prose formtransform generates from the expression and the referenced questions' labels (see ddiEncoding.universe); the typed note carries the XPath verbatim. A select_multiple's go on its varGrp[@type='multipleResp'] (the semi-open pair's on its varGrp[@type='other']), with the universe prose also on each binary var. Until every group has a varGrp (formtransform#152), the enclosing groups' conditions are ANDed into each variable's own.",
+              "lossy": false
             }
           },
           {
@@ -51,9 +51,9 @@
             "lsTsvColumn": "em_validation_q",
             "lsSyntax": "ExpressionScript (EM)",
             "ddi": {
-              "closestElement": "var/valrng",
-              "note": "DDI valrng can express simple numeric ranges only. Currently NOT emitted — constraints are dropped in DDI output.",
-              "lossy": true
+              "element": "var/notes[@type='cdl:constraint'] (+ var/valrng/range for a simple numeric range)",
+              "note": "The typed note carries the XPath verbatim. On an integer or decimal, a constraint that is only bounds on `.` (`. >= 1 and . <= 10`, `. > 0`) is also written as the standard valrng/range (min/max, minExclusive/maxExclusive).",
+              "lossy": false
             }
           },
           {
@@ -63,9 +63,21 @@
             "lsTsvColumn": "em_validation_q_tip",
             "lsSyntax": "plain text",
             "ddi": {
-              "closestElement": null,
-              "note": "No DDI counterpart; dropped.",
-              "lossy": true
+              "element": "var/notes[@type='cdl:constraint_message']",
+              "note": "The form's own text, one note per language (base untagged, others with xml:lang), like every other form text.",
+              "lossy": false
+            }
+          },
+          {
+            "concept": "required answer",
+            "xlsformColumn": "required",
+            "xlsformSyntax": "yes/no",
+            "lsTsvColumn": "mandatory",
+            "lsSyntax": "Y/N",
+            "ddi": {
+              "element": "var/notes[@type='cdl:required']",
+              "note": "Written only for a required question, with the text `yes`. It tells 'not asked' (universe) from 'refused' apart.",
+              "lossy": false
             }
           }
         ],
@@ -111,7 +123,79 @@
           "today",
           "now"
         ],
-        "unsupportedFunctionBehavior": "error — the transpiler throws on any XPath function outside supportedXPathFunctions"
+        "unsupportedFunctionBehavior": "error — the transpiler throws on any XPath function outside supportedXPathFunctions",
+        "ddiEncoding": {
+          "noteSubject": "xlsform-xpath",
+          "noteSubjectMeaning": "The note's text is an XLSForm XPath expression. `${name}` refers to a var/@name or varGrp/@name of the same codebook; `.` to the variable's own answer.",
+          "notes": {
+            "relevant": {
+              "type": "cdl:relevant",
+              "subject": "xlsform-xpath",
+              "on": [
+                "var",
+                "varGrp"
+              ]
+            },
+            "constraint": {
+              "type": "cdl:constraint",
+              "subject": "xlsform-xpath",
+              "on": [
+                "var",
+                "varGrp"
+              ]
+            },
+            "constraint_message": {
+              "type": "cdl:constraint_message",
+              "on": [
+                "var",
+                "varGrp"
+              ],
+              "localized": true
+            },
+            "required": {
+              "type": "cdl:required",
+              "on": [
+                "var",
+                "varGrp"
+              ],
+              "text": "yes"
+            }
+          },
+          "order": "The typed notes come last in var/varGrp, after any untyped note; universe comes after qstn and valrng, before catgry. At most one untyped note per language stays allowed (a citation).",
+          "universe": {
+            "clusion": "I",
+            "untaggedLanguage": "en",
+            "use": "The prose covers `=`, `!=`, `<`, `<=`, `>`, `>=`, `selected()`, `and`, `or`, `not()`, and a comparison with '' (answered / not answered). A condition using anything else gets no universe; its typed note still carries it. It is written in the codebook's base language and each other language with a template here (an untagged codebook uses untaggedLanguage), and only where every referenced label exists in that language. The connecting words are formtransform's metadata, not the form's text; labels and choice labels are the form's own.",
+            "templates": {
+              "en": {
+                "prefix": "Only if ",
+                "and": "and",
+                "or": "or",
+                "not": "not ({})",
+                "selected": "{q} includes {v}",
+                "answered": "{q} is answered",
+                "unanswered": "{q} is not answered",
+                "quotes": [
+                  "“",
+                  "”"
+                ]
+              },
+              "de": {
+                "prefix": "Nur wenn ",
+                "and": "und",
+                "or": "oder",
+                "not": "nicht ({})",
+                "selected": "{q} enthält {v}",
+                "answered": "{q} ist beantwortet",
+                "unanswered": "{q} ist nicht beantwortet",
+                "quotes": [
+                  "„",
+                  "“"
+                ]
+              }
+            }
+          }
+        }
       }
     }
   ]

registry/entities/select_multiple_other/ddi.xml

@@ -75,8 +75,10 @@
       <qstn responseDomainType="text">
         <qstnLit>Sonstiges (bitte angeben)</qstnLit>
       </qstn>
+      <universe clusion="I">Only if “Welche dieser Geräte besitzen Sie?” includes Sonstiges</universe>
       <concept>Sonstiges (bitte angeben)</concept>
       <varFormat type="character" schema="other"/>
+      <notes type="cdl:relevant" subject="xlsform-xpath">selected(${geraetebesitz}, 'other')</notes>
     </var>
   </dataDscr>
 </codeBook>

registry/entities/select_one_other/ddi.xml

@@ -52,8 +52,10 @@
       <qstn responseDomainType="text">
         <qstnLit>Sonstiges (bitte angeben)</qstnLit>
       </qstn>
+      <universe clusion="I">Only if “Wie sind Sie auf unser Angebot aufmerksam geworden?” = Sonstiges</universe>
       <concept>Sonstiges (bitte angeben)</concept>
       <varFormat type="character" schema="other"/>
+      <notes type="cdl:relevant" subject="xlsform-xpath">${aufmerksam} = 'other'</notes>
     </var>
   </dataDscr>
 </codeBook>

src/api.ts

@@ -162,5 +162,8 @@ export function lstsvToDdi(
   tsv: string,
   options: LstsvToDdiOptions = {},
 ): string {
-  return lstsvToDdiXml(tsv, options);
+  return lstsvToDdiXml(tsv, {
+    ...options,
+    onWarning: onceEach(options.onWarning),
+  });
 }

src/cli.ts

@@ -425,6 +425,8 @@ function cmdLstsv2ddi(argv: string[]): void {
       datasetFilename,
       prodDate: values['prod-date'] as string | undefined,
       submissions,
+      onWarning: (w) =>
+        process.stderr.write(`${PROG}: warning: ${w.message}\n`),
     });
     if (submissions) {
       csv = lstsvToDataCsv(tsv, submissions, {