Repository navigation
feat(dloc): extend the DSM converter to the layouts of the EBRAINS datasets - #147
Draft
ehennestad wants to merge 5 commits into
Draft
ehennestad wants to merge 5 commits into
ehennestad wants to merge 5 commits into
Conversation
A Dataset Structure Model names file patterns per layout level, so two data locations may both have a pattern called "original" (the same recording as Igor Pro in one location and NWB in the other, as in the EBRAINS datasets of Dorst et al. and Carannante et al.). A NANSEN variable name is unique within a project, and importing such a model failed when the second variable was inserted. dsm2DataLocationModel now renames every variable whose name occurs in more than one data location to <data location>_<name>, and reports each rename. Names used in one location are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Dataset Structure Model may identify a session by several fields, for example the experiment and the day number when day numbers restart in each experiment (the raw fMRI of the EBRAINS dataset of Murris et al.). The converter left the Session ID unset for such a model, and NANSEN then found no sessions to match. NANSEN can join the names of several folder levels before it applies one extraction rule. For a composite identity whose parts are each read from one level at or above the session level, the Session ID now joins the whole names of those levels with '_'. The levels of the session's ancestor entities are joined too: a model's identity is unique within its parents, a NANSEN Session ID within the project, and without the subject day 1 of two monkeys would be one session. A part given by a fixed rule is not in the path and is left out; both cases are reported. Single-field identities are mapped as before. convertMetadata receives the layout instead of its level names, and the level lookup it already did moved into levelIndexOf so the new function can use it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NANSEN resolves a variable's FileNameExpression with dir, so * is its
only wildcard. The converter refused every file pattern with a
character class, which dropped the only data file of some EBRAINS
datasets (Bos et al.: _{session_id}__\d{4}-\d{2}-\d{2}\.pkl$) and most
of the files of others (Blackstad et al.: ^TT\d+\.ntt$, ^CSC\d+\.ncs$).
A character class (\d, \w, \s, [...]) or a . with its quantifier now
becomes *, and the report says the variable was approximated. An
approximation that keeps no literal text but dots would match every
file in the folder and is still refused. Tokens naming the subject's
identity become * like the session's, since the files of a session
belong to one subject (Kanter et al.: ^{subject_id}_{session_id}\.nwb$).
A cardinality 'many' pattern is reported, because a NANSEN variable
holds one file and NANSEN uses the first match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DSM converter dropped every layout level below the session level, so a dataset whose files sit in a folder of the session, such as the runs in each scanning day's funct folder in murris-2021, got no variables. A NANSEN variable can name a subfolder of the session folder. Folder levels with a fixed name below the session now become the Subfolder of the variables made from the file patterns reached through them. The first level that is neither a fixed folder nor a level of files, and the levels below it, are still dropped and reported. Where a session has several files of a level, the report says that NANSEN uses the first. With levels below the session, one data location can have two file patterns of the same name, in the session folder and in a subfolder. Such a name is now also prefixed with the variable's subfolder (funct_sidecar), where before both became <location>_<name> and the second insert into the variable model failed. Names still equal after that, from two levels in the same folder, are numbered. An extension of several parts, such as \.nii\.gz$, now gives the file type .nii.gz, and listFileAdapters removes only the leading dot of an extension, so such a type can match an adapter that lists nii.gz. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nansen.load and nansen.save choose an adapter with detectFileAdapterForFilepath, which took the extension from fileparts (.gz for run.nii.gz) and compared it with ismember, so an adapter that lists nii.gz was never found. The DSM converter, which asks listFileAdapters, already found it. An adapter now supports a file whose name ends with one of its SupportedFileTypes. Where several adapters support the file, one with the longest matching type is used, so nii.gz wins over gz. Case is ignored, as in listFileAdapters. The adapter list can be passed in, so the choice can be tested without a project. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ehennestad
added this pull request to stack #122
September 19, 2026 16:41
ehennestad
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mechanism
The DSM converter maps each file pattern of the session level to one NANSEN variable, named by the pattern. A NANSEN variable name is unique within a project, and NANSEN reads a session ID from one folder level with one rule.
Problem
Converting the DSM configs of 21 EBRAINS datasets hit four limits. A pattern name used in several data locations failed at the second insert into the variable model. A session identified by two levels (
<experiment>/<day>) got no session ID. A pattern with a character class (\d+) has no wildcard equivalent and was dropped. Every level below the session level was dropped, so a dataset with its files in a fixed folder of the session (<day>/funct/*.nii.gz) got no variables. Separately,nansen.loadandnansen.savedid not find an adapter for an extension of several parts such as.nii.gz.Changes
<location>_<name>. A name used more than once within one data location, in the session folder and in a subfolder, also gets the subfolder (funct_sidecar); names still equal after that are numbered. Each rename is reported.identifierRefsgets a Session ID rule that joins the folder names of the levels its parts are read from (SubfolderLevelarray,Separator '_'), including the levels of its ancestor entities. A fixed part is left out. A part that cannot be read leaves the ID unset, with one report that gives the reason.convertFilePatternToWildcardturns\d,\w,\s,[...]and unescaped.with their quantifier into*and flags the result as approximate; a result that would match every file is refused. The report names each approximated pattern, and wherecardinality: manymeans NANSEN uses the first file.Subfolderof the variables made from the file patterns reached through them; the first level that is neither a fixed folder nor a level of files ends the walk, and it and the levels below it are reported. An extension of several parts (\.nii\.gz$) gives the file type.nii.gz, andlistFileAdaptersremoves only the leading dot.detectFileAdapterForFilepathmatches the end of the file name against each adapter'sSupportedFileTypesand prefers the longest match (nii.gzovergz), ignoring case aslistFileAdaptersdoes. The adapter list can be passed in.Dsm2DataLocationModelTest,Dsm2DataLocationFilesBelowSessionTest(including the same name in the session folder and a subfolder, and nested fixed folders),ImportDatasetStructureModelTest,DetectFileAdapterForFilepathTest; fixturesscanning-days.json,runs-below-session.json.🤖 Generated with Claude Code