Skip to content

feat(ddi)!: a codebook gives back its whole form; round trips through LimeSurvey (#160) #162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jstet merged 1 commit into from
Sep 27, 2026
Merged

feat(ddi)!: a codebook gives back its whole form; round trips through LimeSurvey (#160) #162

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ The migration runs in phases, each keeping every snapshot byte-identical:

### The way back from DDI (`ddi2xlsform`)

A DDI codebook describes a *dataset*, not an *instrument*. A codebook formtransform wrote carries the whole instrument too (#155), so `ddi2xlsform` (#154) turns it back into its form: `src/instrument/fromDdi.ts` reads the standard elements and the `cdl:` notes into the Instrument, and the XLSForm emitter both reverse paths share (`src/xlsform/fromInstrument.ts`) writes the sheets. Any other DDI converts as far as its standard elements go, with a warning for each missing field. The round trip is tested on the model, never on bytes; `src/pipelines/ddi2xlsform/README.md` lists the losses. A `ddi2lstsv` would be `ddi2xlsform` → `xlsform2lstsv` and earns no module of its own.
A DDI codebook describes a *dataset*, not an *instrument*. A codebook formtransform wrote carries the whole instrument too (#155), so `ddi2xlsform` (#154) turns it back into its form: `src/instrument/fromDdi.ts` reads the standard elements and the `cdl:` notes into the Instrument, and the XLSForm emitter both reverse paths share (`src/xlsform/fromInstrument.ts`) writes the sheets. Any other DDI converts as far as its standard elements go, with a warning for each missing field. The round trips are tested on the model, and where the target is text, on bytes: DDI → XLSForm → DDI and XLSForm → DDI → XLSForm → LimeSurvey give the same file (#160). `src/pipelines/ddi2xlsform/README.md` lists what stays lost. A `ddi2lstsv` would be `ddi2xlsform` → `xlsform2lstsv` and earns no module of its own.

What a CDL codebook carries:

Expand Down
8 changes: 5 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,15 +192,17 @@ records how each maps onto it.
`xlsform2lstsv` (deploy the survey), `xlsform2ddi` (document the dataset),
`lstsv2ddi`, `lstsv2xlsform` and `ddi2xlsform` (the reverse paths). All are lossy for some
types: nested groups flatten in LimeSurvey, choice codes over 5 chars truncate,
`select_multiple` becomes N binary variables, and the reverse paths cannot
recover a select's authored `list_name`.
`select_multiple` becomes N binary variables, and the LimeSurvey reverse paths
cannot recover a select's authored `list_name`. A CDL codebook holds the whole
form: XLSForm → DDI → XLSForm gives it back, and its codebook again.

**DDI goes back to XLSForm:** a CDL codebook carries the whole form. Skip
logic, validation and `required` are each a readable `<universe>` sentence
(a simple numeric range also `<valrng>`), plus the exact expression in a typed
note such as `<notes type="cdl:relevant" subject="xlsform-xpath">`
(`convention:logicMapping`). Groups, order, hints, defaults, appearances and
parameters are in standard DDI where it has a place and typed notes where not
parameters, list names, note rows and metadata rows, settings and language
names are in standard DDI where it has a place and typed notes where not
(`convention:ddiFields`). `ddiToXlsform` reads it back; any other DDI converts
as far as its standard elements go, with a warning per missing field
([`src/pipelines/ddi2xlsform/README.md`](src/pipelines/ddi2xlsform/README.md)).
Expand Down
5 changes: 3 additions & 2 deletions codegen/schematron.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,13 +208,14 @@ def generate_schematron(registry: dict[str, Any], output: Path) -> None:
{subject_rules}
</rule>
"""
# At most one note of each cdl: type per element and language (per subject on stdyDscr).
# At most one note of each cdl: type per element and language (on stdyDscr,
# where the subject names what a note is about: per subject too).
cdl_note_uniqueness = """\
<rule context="%P%var | %P%varGrp">
<assert test="every $t in distinct-values(%P%notes[starts-with(@type, 'cdl:')]/@type) satisfies every $l in distinct-values(%P%notes[@type = $t]/string(@xml:lang)) satisfies count(%P%notes[@type = $t][string(@xml:lang) = $l]) &lt;= 1"><value-of select="@name"/> has more than one note of one cdl: type in one language.</assert>
</rule>
<rule context="%P%stdyDscr">
<assert test="every $s in distinct-values(%P%notes[@type = 'cdl:setting']/@subject) satisfies count(%P%notes[@type = 'cdl:setting'][@subject = $s]) &lt;= 1">A setting has more than one cdl:setting note.</assert>
<assert test="every $t in distinct-values(%P%notes[starts-with(@type, 'cdl:')]/@type) satisfies every $s in distinct-values(%P%notes[@type = $t]/string(@subject)) satisfies every $l in distinct-values(%P%notes[@type = $t][string(@subject) = $s]/string(@xml:lang)) satisfies count(%P%notes[@type = $t][string(@subject) = $s][string(@xml:lang) = $l]) &lt;= 1">The study has more than one note of one cdl: type about one subject in one language.</assert>
</rule>
"""

Expand Down
22 changes: 18 additions & 4 deletions ddi-validation/schematron/ddi_custom_rules.sch
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,15 +216,29 @@

<pattern id="cdl_notes">
<rule context="ddi:notes[starts-with(@type, 'cdl:')]">
<assert test="@type = 'cdl:appearance' or @type = 'cdl:constraint' or @type = 'cdl:constraint_message' or @type = 'cdl:default' or @type = 'cdl:exclusive' or @type = 'cdl:hint' or @type = 'cdl:parameters' or @type = 'cdl:relevant' or @type = 'cdl:required' or @type = 'cdl:setting'">Note type "<value-of select="@type"/>" is not in the CDL vocabulary (cdl:appearance, cdl:constraint, cdl:constraint_message, cdl:default, cdl:exclusive, cdl:hint, cdl:parameters, cdl:relevant, cdl:required, cdl:setting).</assert>
<assert test="@type = 'cdl:appearance' or @type = 'cdl:constraint' or @type = 'cdl:constraint_message' or @type = 'cdl:default' or @type = 'cdl:exclusive' or @type = 'cdl:hint' or @type = 'cdl:language' or @type = 'cdl:list' or @type = 'cdl:note_names' or @type = 'cdl:or_other' or @type = 'cdl:other_label' or @type = 'cdl:parameters' or @type = 'cdl:position' or @type = 'cdl:relevant' or @type = 'cdl:required' or @type = 'cdl:row' or @type = 'cdl:row_appearance' or @type = 'cdl:row_hint' or @type = 'cdl:row_label' or @type = 'cdl:row_relevant' or @type = 'cdl:setting'">Note type "<value-of select="@type"/>" is not in the CDL vocabulary (cdl:appearance, cdl:constraint, cdl:constraint_message, cdl:default, cdl:exclusive, cdl:hint, cdl:language, cdl:list, cdl:note_names, cdl:or_other, cdl:other_label, cdl:parameters, cdl:position, cdl:relevant, cdl:required, cdl:row, cdl:row_appearance, cdl:row_hint, cdl:row_label, cdl:row_relevant, cdl:setting).</assert>
<assert test="not(@type = 'cdl:constraint') or @subject = 'xlsform-xpath'">A cdl:constraint note needs subject="xlsform-xpath": its text is an expression in that syntax.</assert>
<assert test="not(@type = 'cdl:language') or normalize-space(@subject) != ''">A cdl:language note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:position') or normalize-space(@subject) != ''">A cdl:position note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:relevant') or @subject = 'xlsform-xpath'">A cdl:relevant note needs subject="xlsform-xpath": its text is an expression in that syntax.</assert>
<assert test="not(@type = 'cdl:row') or normalize-space(@subject) != ''">A cdl:row note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:row_appearance') or normalize-space(@subject) != ''">A cdl:row_appearance note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:row_hint') or normalize-space(@subject) != ''">A cdl:row_hint note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:row_label') or normalize-space(@subject) != ''">A cdl:row_label note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:row_relevant') or normalize-space(@subject) != ''">A cdl:row_relevant note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:setting') or normalize-space(@subject) != ''">A cdl:setting note needs a subject: the name of what it holds.</assert>
</rule>
<rule context="notes[starts-with(@type, 'cdl:')]">
<assert test="@type = 'cdl:appearance' or @type = 'cdl:constraint' or @type = 'cdl:constraint_message' or @type = 'cdl:default' or @type = 'cdl:exclusive' or @type = 'cdl:hint' or @type = 'cdl:parameters' or @type = 'cdl:relevant' or @type = 'cdl:required' or @type = 'cdl:setting'">Note type "<value-of select="@type"/>" is not in the CDL vocabulary (cdl:appearance, cdl:constraint, cdl:constraint_message, cdl:default, cdl:exclusive, cdl:hint, cdl:parameters, cdl:relevant, cdl:required, cdl:setting).</assert>
<assert test="@type = 'cdl:appearance' or @type = 'cdl:constraint' or @type = 'cdl:constraint_message' or @type = 'cdl:default' or @type = 'cdl:exclusive' or @type = 'cdl:hint' or @type = 'cdl:language' or @type = 'cdl:list' or @type = 'cdl:note_names' or @type = 'cdl:or_other' or @type = 'cdl:other_label' or @type = 'cdl:parameters' or @type = 'cdl:position' or @type = 'cdl:relevant' or @type = 'cdl:required' or @type = 'cdl:row' or @type = 'cdl:row_appearance' or @type = 'cdl:row_hint' or @type = 'cdl:row_label' or @type = 'cdl:row_relevant' or @type = 'cdl:setting'">Note type "<value-of select="@type"/>" is not in the CDL vocabulary (cdl:appearance, cdl:constraint, cdl:constraint_message, cdl:default, cdl:exclusive, cdl:hint, cdl:language, cdl:list, cdl:note_names, cdl:or_other, cdl:other_label, cdl:parameters, cdl:position, cdl:relevant, cdl:required, cdl:row, cdl:row_appearance, cdl:row_hint, cdl:row_label, cdl:row_relevant, cdl:setting).</assert>
<assert test="not(@type = 'cdl:constraint') or @subject = 'xlsform-xpath'">A cdl:constraint note needs subject="xlsform-xpath": its text is an expression in that syntax.</assert>
<assert test="not(@type = 'cdl:language') or normalize-space(@subject) != ''">A cdl:language note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:position') or normalize-space(@subject) != ''">A cdl:position note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:relevant') or @subject = 'xlsform-xpath'">A cdl:relevant note needs subject="xlsform-xpath": its text is an expression in that syntax.</assert>
<assert test="not(@type = 'cdl:row') or normalize-space(@subject) != ''">A cdl:row note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:row_appearance') or normalize-space(@subject) != ''">A cdl:row_appearance note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:row_hint') or normalize-space(@subject) != ''">A cdl:row_hint note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:row_label') or normalize-space(@subject) != ''">A cdl:row_label note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:row_relevant') or normalize-space(@subject) != ''">A cdl:row_relevant note needs a subject: the name of what it holds.</assert>
<assert test="not(@type = 'cdl:setting') or normalize-space(@subject) != ''">A cdl:setting note needs a subject: the name of what it holds.</assert>
</rule>
</pattern>
Expand All @@ -234,13 +248,13 @@
<assert test="every $t in distinct-values(ddi:notes[starts-with(@type, 'cdl:')]/@type) satisfies every $l in distinct-values(ddi:notes[@type = $t]/string(@xml:lang)) satisfies count(ddi:notes[@type = $t][string(@xml:lang) = $l]) &lt;= 1"><value-of select="@name"/> has more than one note of one cdl: type in one language.</assert>
</rule>
<rule context="ddi:stdyDscr">
<assert test="every $s in distinct-values(ddi:notes[@type = 'cdl:setting']/@subject) satisfies count(ddi:notes[@type = 'cdl:setting'][@subject = $s]) &lt;= 1">A setting has more than one cdl:setting note.</assert>
<assert test="every $t in distinct-values(ddi:notes[starts-with(@type, 'cdl:')]/@type) satisfies every $s in distinct-values(ddi:notes[@type = $t]/string(@subject)) satisfies every $l in distinct-values(ddi:notes[@type = $t][string(@subject) = $s]/string(@xml:lang)) satisfies count(ddi:notes[@type = $t][string(@subject) = $s][string(@xml:lang) = $l]) &lt;= 1">The study has more than one note of one cdl: type about one subject in one language.</assert>
</rule>
<rule context="var | varGrp">
<assert test="every $t in distinct-values(notes[starts-with(@type, 'cdl:')]/@type) satisfies every $l in distinct-values(notes[@type = $t]/string(@xml:lang)) satisfies count(notes[@type = $t][string(@xml:lang) = $l]) &lt;= 1"><value-of select="@name"/> has more than one note of one cdl: type in one language.</assert>
</rule>
<rule context="stdyDscr">
<assert test="every $s in distinct-values(notes[@type = 'cdl:setting']/@subject) satisfies count(notes[@type = 'cdl:setting'][@subject = $s]) &lt;= 1">A setting has more than one cdl:setting note.</assert>
<assert test="every $t in distinct-values(notes[starts-with(@type, 'cdl:')]/@type) satisfies every $s in distinct-values(notes[@type = $t]/string(@subject)) satisfies every $l in distinct-values(notes[@type = $t][string(@subject) = $s]/string(@xml:lang)) satisfies count(notes[@type = $t][string(@subject) = $s][string(@xml:lang) = $l]) &lt;= 1">The study has more than one note of one cdl: type about one subject in one language.</assert>
</rule>
</pattern>

Expand Down
Loading
Loading