diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b888b86..afb9901 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -155,8 +155,9 @@ The migration runs in phases, each keeping every snapshot byte-identical: The canonical `Variable` (`src/ddi/types.ts`) is what survives an emit: -- **`relevant`, `constraint`, `constraint_message`, `required`** survive (#151). DDI Codebook 2.5 has no expression syntax, so each goes in twice: readable (`` prose, `` for a simple numeric range) and exact, in a typed ``. `convention:logicMapping` (`ddiEncoding`) defines the notes. Until every group has a `varGrp` (#152), a group's condition is ANDed into each variable's own. -- **no `default`, per-question `appearance`, `calculation`, plain group structure** yet (#152, #153). A question's `hint` and `guidance_hint` do survive, as `` and ``. +- **`relevant`, `constraint`, `constraint_message`, `required`** survive (#151). DDI Codebook 2.5 has no expression syntax, so each goes in twice: readable (`` prose, `` for a simple numeric range) and exact, in a typed ``. `convention:logicMapping` (`ddiEncoding`) defines the notes. A group's own condition is on its `varGrp`; a variable's `` states its groups' conditions too. +- **Groups and order** survive (#152): every group is a `` (a plain one `type="section"`), nested through `@varGrp`, and the ``s and data columns follow the survey. +- **no `default`, per-question `appearance`, `calculation`, group hint** yet (#153). A question's `hint` and `guidance_hint` do survive, as `` and ``. Compare `lstsv2xlsform`, which *is* implemented: a LimeSurvey structure TSV carries `relevance`, `em_validation_q`, `mandatory`, `default` and the `!`/`T` type overrides. It is a form definition in a different dialect, so reversing it is a translation problem. Reversing a codebook without those pieces is a *reconstruction* problem, and they cannot be inferred from it. diff --git a/README.md b/README.md index 39e8d09..e07b1a7 100644 --- a/README.md +++ b/README.md @@ -186,7 +186,7 @@ records how each maps onto it. **Supported directions:** four, one per module under `src/pipelines/` — `xlsform2lstsv` (deploy the survey), `xlsform2ddi` (document the dataset), `lstsv2ddi` and `lstsv2xlsform` (the reverse paths). All are lossy for some -types: plain/nested groups flatten, choice codes over 5 chars truncate, +types: nested groups flatten in LimeSurvey, choice codes over 5 chars truncate, `select_multiple` becomes N binary variables, and the reverse paths cannot recover a select's authored `list_name` or tell `integer` from `decimal`. diff --git a/ddi-validation/schematron/ddi_custom_rules.sch b/ddi-validation/schematron/ddi_custom_rules.sch index 8635861..077bea4 100644 --- a/ddi-validation/schematron/ddi_custom_rules.sch +++ b/ddi-validation/schematron/ddi_custom_rules.sch @@ -35,7 +35,7 @@ Variable Group is missing a name attribute. - Variable Group has type="". Only "grid", "multipleResp", or "other" are supported. + Variable Group has type="". Only "grid", "multipleResp", "other", or "section" are supported. Variable Group is missing a concept element. Variable Group uses labl — use concept instead. labl is only for catgry elements. @@ -60,7 +60,7 @@ Variable Group is missing a name attribute. - Variable Group has type="". Only "grid", "multipleResp", or "other" are supported. + Variable Group has type="". Only "grid", "multipleResp", "other", or "section" are supported. Variable Group is missing a concept element. Variable Group uses labl — use concept instead. labl is only for catgry elements. diff --git a/registry/conventions/logicMapping.jsonld b/registry/conventions/logicMapping.jsonld index 9e0410b..d5d3505 100644 --- a/registry/conventions/logicMapping.jsonld +++ b/registry/conventions/logicMapping.jsonld @@ -39,8 +39,8 @@ "lsTsvColumn": "relevance", "lsSyntax": "ExpressionScript (EM)", "ddi": { - "element": "var/universe[@clusion='I'] + var/notes[@type='cdl:relevant']", - "note": "The universe is prose formtransform generates from the expression and the referenced questions' labels (see ddiEncoding.universe); the typed note carries the XPath verbatim. A select_multiple's go on its varGrp[@type='multipleResp'] (the semi-open pair's on its varGrp[@type='other']), with the universe prose also on each binary var. Until every group has a varGrp (formtransform#152), the enclosing groups' conditions are ANDed into each variable's own.", + "element": "var/universe[@clusion='I'] + var/notes[@type='cdl:relevant']; a group's on its varGrp", + "note": "The universe is prose formtransform generates from the expression and the referenced questions' labels (see ddiEncoding.universe); the typed note carries the XPath verbatim. A select_multiple's go on its varGrp[@type='multipleResp'] (the semi-open pair's on its varGrp[@type='other']), with the universe prose also on each binary var. A group's own relevant is on its varGrp (type='section' or 'grid', formtransform#152). Each note carries its element's own condition; each universe states who is asked, i.e. the enclosing groups' conditions ANDed with its own.", "lossy": false } }, diff --git a/registry/root.jsonld b/registry/root.jsonld index 0385667..bd5f872 100644 --- a/registry/root.jsonld +++ b/registry/root.jsonld @@ -226,7 +226,7 @@ "supportsNesting": false }, "ddi": { - "preservation": "conditional", + "preservation": "full", "conditions": [ { "if": "appearance='table-list' OR name contains 'grid' OR label contains 'matrix'", @@ -234,9 +234,10 @@ }, { "if": "else", - "then": "DROPPED - variables flattened to top level" + "then": "varGrp[@type='section']" } - ] + ], + "note": "Every group is a varGrp (formtransform#152): a grid as type='grid', any other as the XSD's own type='section' (type='other' is the semi-open pair's, convention:other). @ID is VG_, @name the group path; a group lists its direct variables in @var and its direct child groups (sections, grids, a select_multiple's multipleResp, an other pair's varGrp) in @varGrp. The label is txt (per xml:lang) and concept; the group's own relevant is its universe and cdl:relevant note (convention:logicMapping). A group with no variable under it has no varGrp." }, "variants": [ "variant:grid" @@ -246,4 +247,4 @@ ] } ] -} \ No newline at end of file +} diff --git a/src/ddi/codebook.ts b/src/ddi/codebook.ts index 703aaca..e4ce493 100644 --- a/src/ddi/codebook.ts +++ b/src/ddi/codebook.ts @@ -7,6 +7,8 @@ * - `select_multiple` expands into `` + binary vars * - grid groups (`appearance="table-list"`) as `` * - the semi-open `_other` pattern as `` + * - every other group as ``, nested by `@varGrp` (#152) + * - ``s in survey order (#152) * - external code lists (`select_*_from_file`) as `` */ @@ -20,13 +22,15 @@ import { import { isGridAppearance } from '../conventions/grid.js'; import { XmlElement } from './xml.js'; import { classifyNotes } from './notes.js'; -import { Choice, Translations, Variable } from './types.js'; +import { Choice, DdiGroup, Translations, Variable } from './types.js'; import { joinTranslations, localizedChild, textsOf } from './translations.js'; import { addLogicNotes, + addRelevantNote, addUniverse, addValrng, logicContext, + universeOf, type LogicContext, } from './logic.js'; import { registeredVocabCodes } from '../conventions/fromFile.js'; @@ -179,7 +183,7 @@ function addVarElement(parent: XmlElement, spec: AddVarSpec): XmlElement { if (spec.logic) { addValrng(varEl, spec.logic.v); - addUniverse(varEl, spec.logic.v, spec.logic.ctx); + addUniverse(varEl, universeOf(spec.logic.v), spec.logic.ctx); } if (!vocab) { @@ -204,7 +208,7 @@ function addGroupLogic( ctx: LogicContext, notes?: { note: InlineNotes; name: string }, ): void { - addUniverse(grpEl, v, ctx); + addUniverse(grpEl, universeOf(v), ctx); if (notes) addGroupNote(grpEl, notes.note, notes.name); addLogicNotes(grpEl, v); } @@ -230,7 +234,7 @@ function addBinaryVar( localizedChild(qstn, 'preQTxt', questionLabel, textsOf(question, 'label')); localizedChild(qstn, 'qstnLit', choiceLabel, choice.translations); // The question's notes are on its varGrp; the prose also here, for readers. - addUniverse(varEl, question, ctx); + addUniverse(varEl, universeOf(question), ctx); for (const val of ['0', '1']) { varEl.child('catgry').textChild('catValu', val); @@ -387,12 +391,23 @@ export interface BuildDdiOptions { prodDate?: string; } +/** + * One question's ``s, in survey order (#152): a plain variable (`grid`: + * the grid group it is a member of), a `select_multiple`'s binaries, or a + * semi-open pair. + */ +export type EmitUnit = + | { kind: 'var'; v: Variable; grid?: string } + | { kind: 'multi'; v: Variable } + | { kind: 'other'; p: OtherPattern }; + /** Returned by {@link splitDataVars}: every data var bucketed by its emit role. */ export interface DataVarBuckets { otherPatterns: Map; gridGroups: Map; multiRespGroups: Map; - standaloneVars: Variable[]; + /** The questions in survey order, which the ``s and data columns follow. */ + units: EmitUnit[]; } /** Sort the flat data vars into the four emit roles `dataDscr` walks through. */ @@ -407,22 +422,28 @@ export function splitDataVars(dataVars: Variable[]): DataVarBuckets { const gridGroups = new Map(); const multiRespGroups = new Map(); - const standaloneVars: Variable[] = []; + const units: EmitUnit[] = []; for (const v of dataVars) { - if (baseNamesInOther.has(v.name) || otherVarNames.has(v.name)) continue; - if (v.type === 'select_multiple') { + // A semi-open pair is emitted where its select is; the companion with it. + if (otherVarNames.has(v.name)) continue; + const pattern = otherPatterns.get(v.name); + if (pattern && baseNamesInOther.has(v.name)) { + units.push({ kind: 'other', p: pattern }); + } else if (v.type === 'select_multiple') { multiRespGroups.set(v.name, v); + units.push({ kind: 'multi', v }); } else if (v.group && isGridGroup(dataVars, v.group)) { const members = gridGroups.get(v.group) ?? []; members.push(v); gridGroups.set(v.group, members); + units.push({ kind: 'var', v, grid: v.group }); } else { - standaloneVars.push(v); + units.push({ kind: 'var', v }); } } - return { otherPatterns, gridGroups, multiRespGroups, standaloneVars }; + return { otherPatterns, gridGroups, multiRespGroups, units }; } /** Emit `` (citation + orphan notes appended). */ @@ -486,6 +507,73 @@ function addGroupNote( if (note) localizedChild(grpEl, 'notes', note, notes.translations[name]); } +/** A group with what it directly contains, by `varGrp` ID reference. */ +interface GroupNode { + group: DdiGroup; + /** Its conditions and its enclosing groups', outermost first. */ + universe: string[]; + /** IDs of the plain ``s directly in it. */ + vars: string[]; + /** IDs of the ``s directly in it. */ + groups: string[]; +} + +/** The variable a unit is placed by. */ +function unitVar(unit: EmitUnit): Variable { + return unit.kind === 'other' ? unit.p.base : unit.v; +} + +/** + * Every group that has a variable under it, by path in survey order, with + * its direct members (#152). A grid member belongs to its grid's `varGrp`; + * a `select_multiple` and a semi-open pair are represented by theirs. + */ +function groupTree(units: EmitUnit[]): Map { + const tree = new Map(); + for (const unit of units) { + const chain = unitVar(unit).groups ?? []; + chain.forEach((group, i) => { + if (tree.has(group.path)) return; + tree.set(group.path, { + group, + universe: chain.slice(0, i + 1).map((g) => g.relevant), + vars: [], + groups: [], + }); + if (i > 0) + tree.get(chain[i - 1].path)?.groups.push(makeGrpId(group.path)); + }); + const inner = chain.length ? tree.get(chain[chain.length - 1].path) : null; + if (!inner) continue; + if (unit.kind === 'multi') inner.groups.push(makeGrpId(unit.v.name)); + else if (unit.kind === 'other') + inner.groups.push(makeGrpId(unit.p.base.name)); + else if (!unit.grid) inner.vars.push(makeVarId(unit.v.name)); + } + return tree; +} + +/** A plain group as `` (#152). */ +function addSection( + dataDscr: XmlElement, + node: GroupNode, + ctx: LogicContext, +): void { + const { group } = node; + const grpEl = dataDscr.child('varGrp', { + ID: makeGrpId(group.path), + name: group.path, + type: 'section', + ...(node.vars.length ? { var: node.vars.join(' ') } : {}), + ...(node.groups.length ? { varGrp: node.groups.join(' ') } : {}), + }); + const label = group.label || group.name; + localizedChild(grpEl, 'txt', label, group.translations); + grpEl.textChild('concept', label); + addUniverse(grpEl, node.universe, ctx); + addRelevantNote(grpEl, group.relevant); +} + /** Emit every `` element into `` (must come before ``). */ function addVarGroups( dataDscr: XmlElement, @@ -495,19 +583,28 @@ function addVarGroups( ctx: LogicContext, ): void { const { gridGroups, multiRespGroups, otherPatterns } = buckets; + const tree = groupTree(buckets.units); + + for (const node of tree.values()) { + if (!gridGroups.has(node.group.path)) addSection(dataDscr, node, ctx); + } for (const [groupName, members] of gridGroups) { const group = getGroupLabel(dataVars, groupName); + const node = tree.get(groupName); const grpEl = dataDscr.child('varGrp', { ID: makeGrpId(groupName), name: groupName, type: 'grid', var: members.map((m) => makeVarId(m.name)).join(' '), + ...(node?.groups.length ? { varGrp: node.groups.join(' ') } : {}), }); localizedChild(grpEl, 'txt', group.label, group.translations); grpEl.textChild('concept', group.label); + if (node) addUniverse(grpEl, node.universe, ctx); // A lead-in note belongs to the group: a member's preQTxt must equal txt. addGroupNote(grpEl, notes, members[0]?.name ?? ''); + if (node) addRelevantNote(grpEl, node.group.relevant); } for (const [smName, smVar] of multiRespGroups) { @@ -535,56 +632,63 @@ function addVars( buckets: DataVarBuckets, ctx: LogicContext, ): void { - const { gridGroups, multiRespGroups, standaloneVars, otherPatterns } = - buckets; - - for (const [groupName, members] of gridGroups) { - const group = getGroupLabel(dataVars, groupName); - for (const v of members) { + for (const unit of buckets.units) { + if (unit.kind === 'multi') { + for (const choice of unit.v.choices) { + addBinaryVar( + dataDscr, + `${unit.v.name}_${choice.name}`, + unit.v, + choice, + ctx, + ); + } + } else if (unit.kind === 'other') { + emitOtherPatternVars(dataDscr, unit.p, ctx); + } else if (unit.grid) { + const group = getGroupLabel(dataVars, unit.grid); // preQTxt must equal the group's txt (Schematron), so a member's own // hint has no slot; validateSubset warns `hint-dropped`. addVarElement(dataDscr, { - varId: makeVarId(v.name), - name: v.name, - label: v.label, - varType: v.type, - choices: v.choices, + varId: makeVarId(unit.v.name), + name: unit.v.name, + label: unit.v.label, + varType: unit.v.type, + choices: unit.v.choices, opts: { preQTxt: group.label, preQTxtTranslations: group.translations }, - guidanceHint: v.guidanceHint, - ...specTranslations(v, false), - logic: { v, ctx }, + guidanceHint: unit.v.guidanceHint, + ...specTranslations(unit.v, false), + logic: { v: unit.v, ctx }, }); + } else { + addStandaloneVar(dataDscr, unit.v, notes, ctx); } } +} - for (const [smName, smVar] of multiRespGroups) { - for (const choice of smVar.choices) { - addBinaryVar(dataDscr, `${smName}_${choice.name}`, smVar, choice, ctx); - } - } - - for (const p of otherPatterns.values()) { - emitOtherPatternVars(dataDscr, p, ctx); - } - - for (const v of standaloneVars) { - addVarElement(dataDscr, { - varId: makeVarId(v.name), - name: v.name, - label: v.label, - varType: v.type, - choices: v.choices, - opts: { - vocab: v.vocab, - preQTxt: notes.text[v.name] ?? '', - preQTxtTranslations: notes.translations[v.name], - }, - hint: v.hint, - guidanceHint: v.guidanceHint, - ...specTranslations(v), - logic: { v, ctx }, - }); - } +/** A variable outside any grid, pair or `select_multiple`. */ +function addStandaloneVar( + dataDscr: XmlElement, + v: Variable, + notes: InlineNotes, + ctx: LogicContext, +): void { + addVarElement(dataDscr, { + varId: makeVarId(v.name), + name: v.name, + label: v.label, + varType: v.type, + choices: v.choices, + opts: { + vocab: v.vocab, + preQTxt: notes.text[v.name] ?? '', + preQTxtTranslations: notes.translations[v.name], + }, + hint: v.hint, + guidanceHint: v.guidanceHint, + ...specTranslations(v), + logic: { v, ctx }, + }); } /** diff --git a/src/ddi/data.ts b/src/ddi/data.ts index 7db199d..cf14729 100644 --- a/src/ddi/data.ts +++ b/src/ddi/data.ts @@ -12,7 +12,7 @@ */ import { splitDataVars } from './codebook.js'; -import type { DataVarBuckets, OtherPattern } from './codebook.js'; +import type { OtherPattern } from './codebook.js'; import { classifyNotes } from './notes.js'; import { OTHER_CODE } from '../conventions/other.js'; import type { Variable } from './types.js'; @@ -61,29 +61,23 @@ function otherPatternColumns(p: OtherPattern): Column[] { } /** - * Ordered column plan for a variable list. - * - * The walk order is the one `addVars` in `ddi/codebook.ts` uses: grid-group - * members, then `select_multiple` binaries, then `_other` patterns, then - * standalone variables. `note` variables carry no data and are skipped. + * Ordered column plan for a variable list: the `` order `addVars` in + * `ddi/codebook.ts` uses, i.e. survey order (#152), a `select_multiple` as + * its binaries, a semi-open pair as its select then its text. `note` + * variables carry no data and are skipped. */ function columnPlan(variables: Variable[]): Column[] { const { dataVars } = classifyNotes(variables); - const buckets: DataVarBuckets = splitDataVars(dataVars); - const { gridGroups, multiRespGroups, otherPatterns, standaloneVars } = - buckets; - const cols: Column[] = []; - for (const members of gridGroups.values()) { - cols.push(...members.map((v) => single(v))); - } - for (const smVar of multiRespGroups.values()) { - cols.push(...smVar.choices.map((c) => binary(smVar, c.name))); - } - for (const p of otherPatterns.values()) { - cols.push(...otherPatternColumns(p)); + for (const unit of splitDataVars(dataVars).units) { + if (unit.kind === 'multi') { + cols.push(...unit.v.choices.map((c) => binary(unit.v, c.name))); + } else if (unit.kind === 'other') { + cols.push(...otherPatternColumns(unit.p)); + } else { + cols.push(single(unit.v)); + } } - cols.push(...standaloneVars.map((v) => single(v))); return cols; } diff --git a/src/ddi/fromInstrument.ts b/src/ddi/fromInstrument.ts index 2ed54f7..4ba94a0 100644 --- a/src/ddi/fromInstrument.ts +++ b/src/ddi/fromInstrument.ts @@ -28,7 +28,13 @@ import type { } from '../instrument/types.js'; import { warning, type WarningHandler } from '../diagnostics.js'; import { languageTagOf } from '../utils/languageUtils.js'; -import type { Choice, Translations, Variable, VariableTexts } from './types.js'; +import type { + Choice, + DdiGroup, + Translations, + Variable, + VariableTexts, +} from './types.js'; // Semi-open "other" convention: the `or_other` type shorthand (and a // LimeSurvey `other=Y`) is expanded into an `other` category plus a @@ -108,31 +114,24 @@ interface GroupContext { /** The group's label in every language, for its translations. */ labelText: Text; appearance: string; - /** The enclosing groups' `relevant` expressions, outermost first. */ - relevant: string[]; + /** The enclosing groups, outermost first (#152). */ + groups: DdiGroup[]; } -/** - * All of `conditions` as one XPath: each parenthesized, ANDed. Until every - * group has a `varGrp` (#152), a variable carries its groups' conditions. - */ -function allOf(conditions: string[]): string { - const parts = conditions.map((c) => c.trim()).filter(Boolean); - return parts.length === 1 - ? parts[0] - : parts.map((c) => `(${c})`).join(' and '); +/** The groups a variable is in, absent at the top level. */ +function groupsOf(ctx: GroupContext): Pick { + return ctx.groups.length ? { groups: ctx.groups } : {}; } -/** A question's logic fields, absent when empty (#151). */ +/** A question's own logic fields, absent when empty (#151). */ function logicOf( q: QuestionItem, - ctx: GroupContext, lang: string | undefined, ): Pick< Variable, 'relevant' | 'constraint' | 'constraintMessage' | 'required' > { - const relevant = allOf([...ctx.relevant, q.relevant]); + const relevant = q.relevant.trim(); const message = pick(q.constraintMessage, lang).trim(); return { ...(relevant ? { relevant } : {}), @@ -288,10 +287,7 @@ function pushCompanion( ): void { const name = q.name + OTHER_SUFFIX; if (state.authoredNames.has(name)) return; - const relevant = allOf([ - ...ctx.relevant, - otherCompanionRelevance(stdType, q.name), - ]); + const relevant = otherCompanionRelevance(stdType, q.name); const translations: Record = {}; const groupLabels = translationsOf(ctx.labelText, state.others) ?? {}; for (const [, tag] of state.others) { @@ -310,6 +306,7 @@ function pushCompanion( listName: '', vocab: '', choices: [], + ...groupsOf(ctx), relevant, ...(Object.keys(translations).length ? { translations } : {}), }); @@ -357,7 +354,8 @@ function pushQuestion( hint: pick(q.hint, state.lang).trim(), guidanceHint: pick(q.guidanceHint, state.lang).trim(), }), - ...logicOf(q, ctx, state.lang), + ...groupsOf(ctx), + ...logicOf(q, state.lang), ...(translations ? { translations } : {}), }); @@ -370,14 +368,25 @@ function project(items: Item[], ctx: GroupContext, state: ProjectState): void { pushQuestion(item, ctx, state); continue; } + const path = ctx.path ? `${ctx.path}/${item.name}` : item.name; + const label = pick(item.label, state.lang); + const translations = translationsOf(item.label, state.others); + const group: DdiGroup = { + name: item.name, + path, + label, + ...(translations ? { translations } : {}), + appearance: item.appearance, + relevant: item.relevant.trim(), + }; project( item.children, { - path: ctx.path ? `${ctx.path}/${item.name}` : item.name, - label: pick(item.label, state.lang), + path, + label, labelText: item.label, appearance: item.appearance, - relevant: [...ctx.relevant, item.relevant], + groups: [...ctx.groups, group], }, state, ); @@ -440,7 +449,7 @@ export function variablesFromInstrument( }; project( instrument.body, - { path: '', label: '', labelText: {}, appearance: '', relevant: [] }, + { path: '', label: '', labelText: {}, appearance: '', groups: [] }, state, ); return state.variables; diff --git a/src/ddi/logic.ts b/src/ddi/logic.ts index 74431d0..e1431c2 100644 --- a/src/ddi/logic.ts +++ b/src/ddi/logic.ts @@ -224,14 +224,37 @@ export function universeProse( return out; } -/** `` per language, when there's a condition. */ +/** + * All of `conditions` as one XPath: each parenthesized, ANDed; `''` when + * none. + */ +function allOf(conditions: string[]): string { + const parts = conditions.map((c) => c.trim()).filter(Boolean); + return parts.length === 1 + ? parts[0] + : parts.map((c) => `(${c})`).join(' and '); +} + +/** + * Who is asked: the enclosing groups' conditions and its own. The universe + * states all of them; each note only its element's own. + */ +export function universeOf(v: Variable): string[] { + return [...(v.groups ?? []).map((g) => g.relevant), v.relevant ?? '']; +} + +/** + * `` per language for `conditions` (outermost first), + * when there are any and the prose covers them all. + */ export function addUniverse( el: XmlElement, - v: Variable, + conditions: string[], ctx: LogicContext, ): void { - if (!v.relevant) return; - for (const [lang, text] of universeProse(v.relevant, ctx)) { + const relevant = allOf(conditions); + if (!relevant) return; + for (const [lang, text] of universeProse(relevant, ctx)) { el.textChild('universe', text, { clusion: ENCODING.universe.clusion, ...(lang ? { 'xml:lang': lang } : {}), @@ -239,6 +262,15 @@ export function addUniverse( } } +/** A `cdl:relevant` note for a group's own condition. */ +export function addRelevantNote(el: XmlElement, relevant: string): void { + if (!relevant) return; + el.textChild('notes', relevant, { + type: NOTES.relevant.type, + subject: ENCODING.noteSubject, + }); +} + /** Bounds on `.` alone, as `` attributes. */ export type Range = Partial< Record<'min' | 'minExclusive' | 'max' | 'maxExclusive', string> @@ -320,9 +352,7 @@ export function addValrng(el: XmlElement, v: Variable): void { */ export function addLogicNotes(el: XmlElement, v: Variable): void { const subject = ENCODING.noteSubject; - if (v.relevant) { - el.textChild('notes', v.relevant, { type: NOTES.relevant.type, subject }); - } + addRelevantNote(el, v.relevant ?? ''); if (v.constraint) { el.textChild('notes', v.constraint, { type: NOTES.constraint.type, diff --git a/src/ddi/types.ts b/src/ddi/types.ts index 9748828..ae82fc3 100644 --- a/src/ddi/types.ts +++ b/src/ddi/types.ts @@ -21,6 +21,24 @@ export interface Choice { translations?: Translations; } +/** + * A group the variable is in (#152): one object per group, shared by all its + * members, so the DDI can give every group a ``. + */ +export interface DdiGroup { + /** Its own name. */ + name: string; + /** Slash-joined path from the top level, the `varGrp/@name`. */ + path: string; + label: string; + /** `label` in the form's other languages. */ + translations?: Translations; + /** Lowercased `appearance`. */ + appearance: string; + /** Its own `relevant` (XPath); `''` when none. */ + relevant: string; +} + /** A variable's texts in one of the form's other languages. */ export interface VariableTexts { label?: string; @@ -55,8 +73,13 @@ export interface Variable { /** XLSForm `guidance_hint`, emitted as ``. */ guidanceHint?: string; /** - * XLSForm `relevant` (XPath), with the enclosing groups' conditions ANDed - * in: `` prose plus a `cdl:relevant` note (#151). + * The enclosing groups, outermost first (#152). Without it the codebook + * has no `section` groups. + */ + groups?: DdiGroup[]; + /** + * Its own XLSForm `relevant` (XPath): a `cdl:relevant` note, and with the + * groups' conditions the `` prose (#151). */ relevant?: string; /** XLSForm `constraint` (XPath): a `cdl:constraint` note, maybe ``. */ diff --git a/src/generated/conventions.json b/src/generated/conventions.json index ddff170..c0d86ec 100644 --- a/src/generated/conventions.json +++ b/src/generated/conventions.json @@ -234,8 +234,8 @@ "lsTsvColumn": "relevance", "lsSyntax": "ExpressionScript (EM)", "ddi": { - "element": "var/universe[@clusion='I'] + var/notes[@type='cdl:relevant']", - "note": "The universe is prose formtransform generates from the expression and the referenced questions' labels (see ddiEncoding.universe); the typed note carries the XPath verbatim. A select_multiple's go on its varGrp[@type='multipleResp'] (the semi-open pair's on its varGrp[@type='other']), with the universe prose also on each binary var. Until every group has a varGrp (formtransform#152), the enclosing groups' conditions are ANDed into each variable's own.", + "element": "var/universe[@clusion='I'] + var/notes[@type='cdl:relevant']; a group's on its varGrp", + "note": "The universe is prose formtransform generates from the expression and the referenced questions' labels (see ddiEncoding.universe); the typed note carries the XPath verbatim. A select_multiple's go on its varGrp[@type='multipleResp'] (the semi-open pair's on its varGrp[@type='other']), with the universe prose also on each binary var. A group's own relevant is on its varGrp (type='section' or 'grid', formtransform#152). Each note carries its element's own condition; each universe states who is asked, i.e. the enclosing groups' conditions ANDed with its own.", "lossy": false } }, diff --git a/src/generated/conventions.ts b/src/generated/conventions.ts index 60762af..e507ee1 100644 --- a/src/generated/conventions.ts +++ b/src/generated/conventions.ts @@ -243,8 +243,8 @@ const conventions = { "lsTsvColumn": "relevance", "lsSyntax": "ExpressionScript (EM)", "ddi": { - "element": "var/universe[@clusion='I'] + var/notes[@type='cdl:relevant']", - "note": "The universe is prose formtransform generates from the expression and the referenced questions' labels (see ddiEncoding.universe); the typed note carries the XPath verbatim. A select_multiple's go on its varGrp[@type='multipleResp'] (the semi-open pair's on its varGrp[@type='other']), with the universe prose also on each binary var. Until every group has a varGrp (formtransform#152), the enclosing groups' conditions are ANDed into each variable's own.", + "element": "var/universe[@clusion='I'] + var/notes[@type='cdl:relevant']; a group's on its varGrp", + "note": "The universe is prose formtransform generates from the expression and the referenced questions' labels (see ddiEncoding.universe); the typed note carries the XPath verbatim. A select_multiple's go on its varGrp[@type='multipleResp'] (the semi-open pair's on its varGrp[@type='other']), with the universe prose also on each binary var. A group's own relevant is on its varGrp (type='section' or 'grid', formtransform#152). Each note carries its element's own condition; each universe states who is asked, i.e. the enclosing groups' conditions ANDed with its own.", "lossy": false } }, diff --git a/src/pipelines/README.md b/src/pipelines/README.md index e95dfac..16976b4 100644 --- a/src/pipelines/README.md +++ b/src/pipelines/README.md @@ -24,9 +24,9 @@ emitter is: it takes `Variable[]` and raw response rows, so either DDI pipeline can feed it. Its one hard contract is **column order equals `` order**. The -column plan walks the same buckets `dataDscr` does — grid-group members, -`select_multiple` binaries, `_other` patterns, standalone variables — so every -header matches a `` in the XML, position for position, and schema↔data +column plan walks the same survey-ordered questions `dataDscr` does (#152) — +a `select_multiple` as its binaries, an `_other` pair as its select then its +text — so every header matches a `` in the XML, position for position, and schema↔data alignment stays a zip rather than a lookup. A `select_multiple` expands to one `0`/`1` column per choice; `note` variables get no column at all. @@ -67,10 +67,13 @@ since #151 its logic: typed note (``). `convention:logicMapping` ([`registry/conventions/logicMapping.jsonld`](../../registry/conventions/logicMapping.jsonld), - `ddiEncoding`) defines them. A group's condition is ANDed into each - variable's own until groups get a `varGrp` (#152). -- **still absent:** `default`, per-question `appearance`, `calculation`, plain - group structure, question order (#152, #153). + `ddiEncoding`) defines them. A group's own condition is on its `varGrp`; + a variable's `` states its groups' conditions too. +- **groups and order** (#152): every group is a `` (a plain one + `type="section"`), nested through `@varGrp`, and the ``s follow the + survey. +- **still absent:** `default`, per-question `appearance`, `calculation`, the + group hint, and a note row apart from the hint after it (#153). Compare `lstsv2xlsform`, which *is* implemented: a LimeSurvey structure TSV carries `relevance`, `em_validation_q`, `mandatory`, `default` and the `!`/`T` @@ -80,7 +83,7 @@ is a translation problem. Reversing a codebook without those pieces is a The failure mode matters more than the missing feature. A `ddi2xlsform` before the rest of #155 would emit a survey that looks correct and behaves wrongly: no -defaults, no appearances, flattened groups. Silently producing a broken instrument is worse +defaults, no appearances. Silently producing a broken instrument is worse than declining to produce one — the same reasoning behind `validateLstsvSubset` rejecting out-of-subset input rather than guessing at it. diff --git a/tests/fixtures/surveys/all_types_survey/ddi.xml b/tests/fixtures/surveys/all_types_survey/ddi.xml index a95dca6..e65ad8b 100644 --- a/tests/fixtures/surveys/all_types_survey/ddi.xml +++ b/tests/fixtures/surveys/all_types_survey/ddi.xml @@ -24,6 +24,30 @@ + + Persönliche Angaben + Persönliche Angaben + + + Ausführliches Feedback + Ausführliches Feedback + + + Selbsteinschätzung technischer Fähigkeiten + Selbsteinschätzung technischer Fähigkeiten + + + Programmierkenntnisse + Programmierkenntnisse + + + Weitere Fragen + Weitere Fragen + + + Anmeldedaten + Anmeldedaten + An welchen Tracks haben Sie teilgenommen? An welchen Tracks haben Sie teilgenommen? @@ -40,131 +64,6 @@ Welche Tracks würden Sie erneut besuchen? Welche Tracks würden Sie erneut besuchen? - - - An welchen Tracks haben Sie teilgenommen? - Frontend-Entwicklung - - - 0 - - - 1 - - An welchen Tracks haben Sie teilgenommen?: Frontend-Entwicklung - - - - - An welchen Tracks haben Sie teilgenommen? - Backend-Entwicklung - - - 0 - - - 1 - - An welchen Tracks haben Sie teilgenommen?: Backend-Entwicklung - - - - - An welchen Tracks haben Sie teilgenommen? - Data Science - - - 0 - - - 1 - - An welchen Tracks haben Sie teilgenommen?: Data Science - - - - - Welcher Track hat Ihnen am besten gefallen? - - - red - Frontend-Entwicklung - - - blue - Backend-Entwicklung - - - green - Data Science - - - other - Sonstiges: - - Welcher Track hat Ihnen am besten gefallen? - - - - - Sonstiges: - - Nur wenn „Welcher Track hat Ihnen am besten gefallen?“ = Sonstiges: - Sonstiges: - - ${q_sel1_other} = 'other' - - - - Welche Tracks würden Sie erneut besuchen? - Frontend-Entwicklung - - - 0 - - - 1 - - Welche Tracks würden Sie erneut besuchen?: Frontend-Entwicklung - - - - - Welche Tracks würden Sie erneut besuchen? - Backend-Entwicklung - - - 0 - - - 1 - - Welche Tracks würden Sie erneut besuchen?: Backend-Entwicklung - - - - - Welche Tracks würden Sie erneut besuchen? - Data Science - - - 0 - - - 1 - - Welche Tracks würden Sie erneut besuchen?: Data Science - - - - - Sonstiges: - - Nur wenn „Welche Tracks würden Sie erneut besuchen?“ enthält Sonstiges: - Sonstiges: - - selected(${q_selm_other}, 'other') - Wie lautet Ihre Teilnehmer-ID? @@ -236,6 +135,48 @@ Würden Sie diesen Workshop weiterempfehlen? + + + An welchen Tracks haben Sie teilgenommen? + Frontend-Entwicklung + + + 0 + + + 1 + + An welchen Tracks haben Sie teilgenommen?: Frontend-Entwicklung + + + + + An welchen Tracks haben Sie teilgenommen? + Backend-Entwicklung + + + 0 + + + 1 + + An welchen Tracks haben Sie teilgenommen?: Backend-Entwicklung + + + + + An welchen Tracks haben Sie teilgenommen? + Data Science + + + 0 + + + 1 + + An welchen Tracks haben Sie teilgenommen?: Data Science + + Beschreiben Sie bitte Ihre Erfahrungen beim Workshop @@ -353,6 +294,89 @@ SQL + + + Welcher Track hat Ihnen am besten gefallen? + + + red + Frontend-Entwicklung + + + blue + Backend-Entwicklung + + + green + Data Science + + + other + Sonstiges: + + Welcher Track hat Ihnen am besten gefallen? + + + + + Sonstiges: + + Nur wenn „Welcher Track hat Ihnen am besten gefallen?“ = Sonstiges: + Sonstiges: + + ${q_sel1_other} = 'other' + + + + Welche Tracks würden Sie erneut besuchen? + Frontend-Entwicklung + + + 0 + + + 1 + + Welche Tracks würden Sie erneut besuchen?: Frontend-Entwicklung + + + + + Welche Tracks würden Sie erneut besuchen? + Backend-Entwicklung + + + 0 + + + 1 + + Welche Tracks würden Sie erneut besuchen?: Backend-Entwicklung + + + + + Welche Tracks würden Sie erneut besuchen? + Data Science + + + 0 + + + 1 + + Welche Tracks würden Sie erneut besuchen?: Data Science + + + + + Sonstiges: + + Nur wenn „Welche Tracks würden Sie erneut besuchen?“ enthält Sonstiges: + Sonstiges: + + selected(${q_selm_other}, 'other') + Wie lautet Ihre E-Mail-Adresse? diff --git a/tests/fixtures/surveys/appearances_survey/ddi.xml b/tests/fixtures/surveys/appearances_survey/ddi.xml index c952b9b..9a64a4e 100644 --- a/tests/fixtures/surveys/appearances_survey/ddi.xml +++ b/tests/fixtures/surveys/appearances_survey/ddi.xml @@ -21,42 +21,18 @@ + + Über Sie + Über Sie + + + Wie oft nutzen Sie … + Wie oft nutzen Sie … + Wie wichtig ist Ihnen … Wie wichtig ist Ihnen … - - - Wie wichtig ist Ihnen … - … der Preis? - - - nie - Nie - - - oft - Oft - - … der Preis? - - - - - Wie wichtig ist Ihnen … - … der Takt? - - - nie - Nie - - - oft - Oft - - … der Takt? - - Wo wohnen Sie? @@ -128,6 +104,38 @@ Bahn + + + Wie wichtig ist Ihnen … + … der Preis? + + + nie + Nie + + + oft + Oft + + … der Preis? + + + + + Wie wichtig ist Ihnen … + … der Takt? + + + nie + Nie + + + oft + Oft + + … der Takt? + + Herkunftsland? diff --git a/tests/fixtures/surveys/bilingual_survey/ddi.xml b/tests/fixtures/surveys/bilingual_survey/ddi.xml index bdfbcb4..f836dea 100644 --- a/tests/fixtures/surveys/bilingual_survey/ddi.xml +++ b/tests/fixtures/surveys/bilingual_survey/ddi.xml @@ -24,6 +24,11 @@ + + Zur Person + About you + Zur Person + Wie sehr vertrauen Sie … How much do you trust … @@ -39,56 +44,41 @@ How did you hear about us? Wie haben Sie von uns erfahren? - - - Wie sehr vertrauen Sie … - How much do you trust … - … dem Parlament? - … parliament? - Skala vorlesen - Read the scale + + + Alle Angaben sind freiwillig. + +Aktuelle Tätigkeit + All answers are optional. + +Current job + Was ist Ihr Beruf? + What is your occupation? + Nicht vorlesen - - 1 - gar nicht - not at all - - - 2 - etwas - somewhat - - - 3 - sehr - very much - - … dem Parlament? - + Was ist Ihr Beruf? + - + - Wie sehr vertrauen Sie … - How much do you trust … - … der Polizei? - … the police? + Höchster Abschluss? + Highest degree? 1 - gar nicht - not at all + Kein Abschluss + No degree 2 - etwas - somewhat + Hauptschulabschluss 3 - sehr - very much + Abitur + A levels - … der Polizei? + Höchster Abschluss? @@ -157,41 +147,56 @@ ${quelle} = 'other' - - - Alle Angaben sind freiwillig. - -Aktuelle Tätigkeit - All answers are optional. - -Current job - Was ist Ihr Beruf? - What is your occupation? - Nicht vorlesen + + + Wie sehr vertrauen Sie … + How much do you trust … + … dem Parlament? + … parliament? + Skala vorlesen + Read the scale - Was ist Ihr Beruf? - + + 1 + gar nicht + not at all + + + 2 + etwas + somewhat + + + 3 + sehr + very much + + … dem Parlament? + - + - Höchster Abschluss? - Highest degree? + Wie sehr vertrauen Sie … + How much do you trust … + … der Polizei? + … the police? 1 - Kein Abschluss - No degree + gar nicht + not at all 2 - Hauptschulabschluss + etwas + somewhat 3 - Abitur - A levels + sehr + very much - Höchster Abschluss? + … der Polizei? diff --git a/tests/fixtures/surveys/complex_survey/ddi.xml b/tests/fixtures/surveys/complex_survey/ddi.xml index d3b33a1..84c3649 100644 --- a/tests/fixtures/surveys/complex_survey/ddi.xml +++ b/tests/fixtures/surveys/complex_survey/ddi.xml @@ -22,12 +22,60 @@ + + Demographics + Demographics + + + Preferences + Preferences + Only if “Age” ≥ 18 + ${age} >= 18 + What are your favorite colors? What are your favorite colors? Only if “Age” ≥ 18 - ${age} >= 18 + + + Full Name + + Full Name + + yes + + + + Age + + + + + Age + + . >= 18 and . <= 120 + Age must be between 18 and 120 + + + + Gender + + + male + Male + + + feml + Woman + + + othr + Other + + Gender + + What are your favorite colors? @@ -88,45 +136,6 @@ What are your favorite colors?: Yellow - - - Full Name - - Full Name - - yes - - - - Age - - - - - Age - - . >= 18 and . <= 120 - Age must be between 18 and 120 - - - - Gender - - - male - Male - - - feml - Woman - - - othr - Other - - Gender - - How satisfied are you? @@ -142,7 +151,6 @@ How satisfied are you? - ${age} >= 18 @@ -151,7 +159,7 @@ Only if “Age” ≥ 18 and “How satisfied are you?” = “unhappy” Any other comments? - (${age} >= 18) and (${satisfaction_level} = 'unhappy') + ${satisfaction_level} = 'unhappy' diff --git a/tests/fixtures/surveys/complex_xpath_survey/ddi.xml b/tests/fixtures/surveys/complex_xpath_survey/ddi.xml index d5b47f4..c9bac4c 100644 --- a/tests/fixtures/surveys/complex_xpath_survey/ddi.xml +++ b/tests/fixtures/surveys/complex_xpath_survey/ddi.xml @@ -22,6 +22,10 @@ + + Consent and Demographics + Consent and Demographics + Do you consent to participate? diff --git a/tests/fixtures/surveys/hints_survey/ddi.xml b/tests/fixtures/surveys/hints_survey/ddi.xml index c999b81..c52b661 100644 --- a/tests/fixtures/surveys/hints_survey/ddi.xml +++ b/tests/fixtures/surveys/hints_survey/ddi.xml @@ -26,34 +26,6 @@ Welche Medien nutzen Sie? Welche Medien nutzen Sie? - - - Welche Medien nutzen Sie? - Fernsehen - - - 0 - - - 1 - - Welche Medien nutzen Sie?: Fernsehen - - - - - Welche Medien nutzen Sie? - Radio - - - 0 - - - 1 - - Welche Medien nutzen Sie?: Radio - - Die folgenden Fragen betreffen Ihren Haushalt. @@ -89,5 +61,33 @@ Sie selbst mitgezählt. Sind Sie ehrenamtlich tätig? + + + Welche Medien nutzen Sie? + Fernsehen + + + 0 + + + 1 + + Welche Medien nutzen Sie?: Fernsehen + + + + + Welche Medien nutzen Sie? + Radio + + + 0 + + + 1 + + Welche Medien nutzen Sie?: Radio + + diff --git a/tests/fixtures/surveys/multilingual_survey/ddi.xml b/tests/fixtures/surveys/multilingual_survey/ddi.xml index 61f4a95..5f928c1 100644 --- a/tests/fixtures/surveys/multilingual_survey/ddi.xml +++ b/tests/fixtures/surveys/multilingual_survey/ddi.xml @@ -22,6 +22,11 @@ + + Demographics + Datos demográficos + Demographics + Enter your full name diff --git a/tests/fixtures/surveys/multipage_survey/ddi.xml b/tests/fixtures/surveys/multipage_survey/ddi.xml index 95120d5..808ec9d 100644 --- a/tests/fixtures/surveys/multipage_survey/ddi.xml +++ b/tests/fixtures/surveys/multipage_survey/ddi.xml @@ -23,6 +23,14 @@ + + Page One + Page One + + + Page Two + Page Two + Your name diff --git a/tests/fixtures/surveys/settings_survey/ddi.xml b/tests/fixtures/surveys/settings_survey/ddi.xml index 56408ec..d4b17b5 100644 --- a/tests/fixtures/surveys/settings_survey/ddi.xml +++ b/tests/fixtures/surveys/settings_survey/ddi.xml @@ -24,6 +24,10 @@ + + Main Questions + Main Questions + What is your _favorite_ color? What is your _favorite_ color? diff --git a/tests/fixtures/surveys/testA/ddi.xml b/tests/fixtures/surveys/testA/ddi.xml index 7754e68..0a91463 100644 --- a/tests/fixtures/surveys/testA/ddi.xml +++ b/tests/fixtures/surveys/testA/ddi.xml @@ -27,66 +27,30 @@ + + teilnahme + teilnahme + + + wirkung + wirkung + + + erfahrungen + erfahrungen + + + empfehlung + empfehlung + + + demografie + demografie + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) - - - Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) - Holzwerkstatt - - - 0 - - - 1 - - Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich): Holzwerkstatt - - - - - Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) - Metallwerkstatt - - - 0 - - - 1 - - Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich): Metallwerkstatt - - - - - Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) - Textilwerkstatt - - - 0 - - - 1 - - Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich): Textilwerkstatt - - - - - Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) - Medienwerkstatt/Digital - - - 0 - - - 1 - - Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich): Medienwerkstatt/Digital - - Wie lange kommst du schon zu TestWerk? @@ -175,6 +139,62 @@ yes + + + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) + Holzwerkstatt + + + 0 + + + 1 + + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich): Holzwerkstatt + + + + + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) + Metallwerkstatt + + + 0 + + + 1 + + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich): Metallwerkstatt + + + + + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) + Textilwerkstatt + + + 0 + + + 1 + + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich): Textilwerkstatt + + + + + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich) + Medienwerkstatt/Digital + + + 0 + + + 1 + + Welche Bereiche nutzt du überhaupt? (Mehrfachauswahl möglich): Medienwerkstatt/Digital + + Hattest du vorher schon handwerkliche Erfahrung? diff --git a/tests/fixtures/surveys/testB/ddi.xml b/tests/fixtures/surveys/testB/ddi.xml index ba2b30c..c43630e 100644 --- a/tests/fixtures/surveys/testB/ddi.xml +++ b/tests/fixtures/surveys/testB/ddi.xml @@ -28,6 +28,36 @@ + + Deine Fähigkeiten und Erfahrungen + Your skills and experiences + Deine Fähigkeiten und Erfahrungen + + + Tools + Tools + Tools + + + Techniken + Techniques + Techniken + + + Themen + Topics + Themen + + + Du und das Projekt + You and the project + Du und das Projekt + + + Über dich + About you + Über dich + Für welche Projekte möchtest Du dich bewerben? Which project(s) do you want to apply for? diff --git a/tests/ts/contract/lstsv2ddiRoundtrip.test.ts b/tests/ts/contract/lstsv2ddiRoundtrip.test.ts index 922eea6..3c013ff 100644 --- a/tests/ts/contract/lstsv2ddiRoundtrip.test.ts +++ b/tests/ts/contract/lstsv2ddiRoundtrip.test.ts @@ -33,6 +33,19 @@ function scrub(xml: string): string { ); } +/** + * LimeSurvey needs every question in a group, so xlsform2lstsv puts the + * top-level questions into one named `defaults.groupName` ("Questions"). The + * TSV can't tell it from an authored group, so lstsv2ddi keeps it as a + * `section` (#152); the XLSForm had none. + */ +function withoutWrapperGroup(xml: string): string { + return xml.replace( + /\n\s*]*>[\s\S]*?<\/varGrp>/, + '', + ); +} + interface Case { id: string; dir: string; @@ -65,6 +78,6 @@ describe('lstsv2ddi round-trip', () => { const committed = fs.readFileSync(path.join(dir, 'ddi.xml'), 'utf-8'); // xlsform2ddi's snapshots use the folder name as the study title. const observed = lstsvToDdiXml(tsv, { assetName: id }); - expect(scrub(observed)).toBe(scrub(committed)); + expect(withoutWrapperGroup(scrub(observed))).toBe(scrub(committed)); }); }); diff --git a/tests/ts/integration/cli2ddi.test.ts b/tests/ts/integration/cli2ddi.test.ts index 0cb67a4..14954d3 100644 --- a/tests/ts/integration/cli2ddi.test.ts +++ b/tests/ts/integration/cli2ddi.test.ts @@ -115,7 +115,7 @@ describe('xlsform2ddi --data', () => { expect(fileUri(xml)).toBe('data.csv'); expect(fileName(xml)).toBe('data.csv'); expect(csvHeader(csv)).toEqual(xmlVarNames(xml)); - expect(csv).toContain('1,0,1,Ada,36,feml,"likes, commas",2026-01-02'); + expect(csv).toContain('Ada,36,feml,1,0,1,"likes, commas",2026-01-02'); }); test('Kobo JSON with group paths; --dataset-filename drives the default path', () => { @@ -135,7 +135,7 @@ describe('xlsform2ddi --data', () => { expect(caseQnty(xml)).toBe('3'); expect(fileUri(xml)).toBe('abc.csv'); expect(csvHeader(csv)).toEqual(xmlVarNames(xml)); - expect(csv.split('\r\n')[1]).toMatch(/^0,1,0,Ada,/); + expect(csv.split('\r\n')[1]).toMatch(/^Ada,,,0,1,0,/); }); test('--data-out names the file and the recorded URI', () => { diff --git a/tests/ts/unit/ddi/logic.test.ts b/tests/ts/unit/ddi/logic.test.ts index 2b25747..18a0d2c 100644 --- a/tests/ts/unit/ddi/logic.test.ts +++ b/tests/ts/unit/ddi/logic.test.ts @@ -99,7 +99,7 @@ describe('relevant', () => { expect(v).toContain('string-length(${a}) > 3'); }); - test("a group's condition is ANDed into each member's", () => { + test("a group's condition is on its section; the universe states both", () => { const xml = ddi([ { type: 'select_one yn', name: 'dog', label: 'Dog?' }, { @@ -112,11 +112,23 @@ describe('relevant', () => { { type: 'text', name: 't', label: 'T', relevant: '${age} > 1' }, { type: 'end_group' }, ]); - expect(varXml(xml, 'age')).toContain( + const section = xml.slice( + xml.indexOf(''), + ); + expect(section).toContain('type="section" var="V_age V_t"'); + expect(section).toContain('Only if “Dog?” = Yes'); + expect(section).toContain( `subject="xlsform-xpath">\${dog} = 'yes'`, ); + // A member's note is its own condition only. + expect(varXml(xml, 'age')).not.toContain(''); + expect(varXml(xml, 't')).toContain( + `subject="xlsform-xpath">\${age} > 1`, + ); expect(varXml(xml, 't')).toContain( - `subject="xlsform-xpath">(\${dog} = 'yes') and (\${age} > 1)`, + 'Only if “Dog?” = Yes and “Age” > 1', ); }); diff --git a/tests/ts/unit/ddi/structure.test.ts b/tests/ts/unit/ddi/structure.test.ts new file mode 100644 index 0000000..378dcdf --- /dev/null +++ b/tests/ts/unit/ddi/structure.test.ts @@ -0,0 +1,131 @@ +/** + * The form's structure in the DDI (#152): every group a `` (a plain + * one `type="section"`), nested by `@varGrp`, and the ``s in survey + * order. + */ +import { describe, test, expect } from 'vitest'; + +import { buildDdiXml } from '../../../../src/pipelines/xlsform2ddi/index.js'; +import { getDdiColumnNames } from '../../../../src/ddi/data.js'; +import { extractVariables } from '../../../../src/pipelines/xlsform2ddi/variables.js'; + +const OPTS = { prodDate: '2020-01-01' }; + +const CHOICES = [ + { list_name: 'yn', name: 'yes', label: 'Yes' }, + { list_name: 'yn', name: 'no', label: 'No' }, +]; + +const SURVEY = [ + { type: 'text', name: 'first', label: 'First' }, + { type: 'begin_group', name: 'outer', label: 'Outer' }, + { type: 'select_multiple yn', name: 'm', label: 'M' }, + { type: 'begin_group', name: 'inner', label: 'Inner' }, + { type: 'integer', name: 'n', label: 'N' }, + { type: 'select_one yn or_other', name: 'o', label: 'O' }, + { type: 'end_group' }, + { + type: 'begin_group', + name: 'grid', + label: 'Grid', + appearance: 'table-list', + }, + { type: 'select_one yn', name: 'g1', label: 'G1' }, + { type: 'select_one yn', name: 'g2', label: 'G2' }, + { type: 'end_group' }, + { type: 'end_group' }, + { type: 'begin_group', name: 'empty', label: 'Only a note' }, + { type: 'note', name: 'hello', label: 'Hello' }, + { type: 'end_group' }, + { type: 'date', name: 'last', label: 'Last' }, +]; + +function varGrp(xml: string, id: string): string { + const start = xml.indexOf(`', start)); +} + +const xml = buildDdiXml(SURVEY, CHOICES, OPTS); + +describe('groups', () => { + test('a plain group is a section listing its direct members', () => { + expect(varGrp(xml, 'VG_outer')).toContain( + '', + ); + expect(varGrp(xml, 'VG_outer_inner')).toContain( + 'name="outer/inner" type="section" var="V_n" varGrp="VG_o">', + ); + expect(varGrp(xml, 'VG_outer_inner')).toContain('Inner'); + expect(varGrp(xml, 'VG_outer_inner')).toContain('Inner'); + }); + + test('a grid stays a grid, inside its section', () => { + expect(varGrp(xml, 'VG_outer_grid')).toContain( + 'type="grid" var="V_g1 V_g2"', + ); + }); + + test('a group without a variable under it has no varGrp', () => { + expect(xml).not.toContain('VG_empty'); + }); + + test('top-level questions are in no group', () => { + expect(xml).not.toMatch(/var="[^"]*V_first/); + }); + + test('a section label in every language', () => { + const bi = buildDdiXml( + [ + { + type: 'begin_group', + name: 'p', + 'label::Deutsch (de)': 'Person', + 'label::English (en)': 'Person (en)', + }, + { + type: 'text', + name: 'job', + 'label::Deutsch (de)': 'Beruf?', + 'label::English (en)': 'Job?', + }, + { type: 'end_group' }, + ], + [], + { ...OPTS, settings: { default_language: 'Deutsch (de)' } }, + ); + expect(varGrp(bi, 'VG_p')).toContain( + 'Person\n Person (en)', + ); + }); +}); + +describe('order', () => { + const names = [...xml.matchAll(/ m[1], + ); + const survey = [ + 'first', + 'm_yes', + 'm_no', + 'n', + 'o', + 'o_other', + 'g1', + 'g2', + 'last', + ]; + + test('the vars follow the survey', () => { + expect(names).toEqual(survey); + }); + + test('the data columns follow the vars', () => { + const variables = extractVariables( + SURVEY, + Object.fromEntries( + ['yn'].map((l) => [l, CHOICES.filter((c) => c.list_name === l)]), + ), + ); + expect(getDdiColumnNames(variables)).toEqual(survey); + }); +}); diff --git a/workers/schematron-worker/src/test/java/dev/correlaid/schematron/SchematronValidatorTest.java b/workers/schematron-worker/src/test/java/dev/correlaid/schematron/SchematronValidatorTest.java index 84ea9ff..dc42c1a 100644 --- a/workers/schematron-worker/src/test/java/dev/correlaid/schematron/SchematronValidatorTest.java +++ b/workers/schematron-worker/src/test/java/dev/correlaid/schematron/SchematronValidatorTest.java @@ -96,7 +96,7 @@ void badVarGrpTypeDetected() throws Exception { List errors = schValidator.validate(xml); assertFalse(errors.isEmpty(), "Expected errors for bad varGrp type"); assertTrue( - errors.stream().anyMatch(e -> e.message.contains("has type=\"section\"")), + errors.stream().anyMatch(e -> e.message.contains("has type=\"subject\"")), "Expected varGrp type error, got: " + formatErrors(errors) ); } diff --git a/workers/schematron-worker/src/test/resources/invalid_vargrp_bad_type.xml b/workers/schematron-worker/src/test/resources/invalid_vargrp_bad_type.xml index 669ca55..5ef1299 100644 --- a/workers/schematron-worker/src/test/resources/invalid_vargrp_bad_type.xml +++ b/workers/schematron-worker/src/test/resources/invalid_vargrp_bad_type.xml @@ -8,8 +8,8 @@ - - + + Section groups are not supported