Repository navigation
feat: session and subject types, file adapters for the EBRAINS dataset formats, and a source for bucket files - #2
Draft
ehennestad wants to merge 18 commits into
Draft
ehennestad wants to merge 18 commits into
ehennestad wants to merge 18 commits into
Conversation
Introduces mustBeDataSharingCollabName to validate Data Sharing Collab names by prefix and UUID, and mustBeValidUUID to ensure identifiers match the RFC 4122 UUID format.
Introduces fetchCollabInfo.m to retrieve metadata for a Data Sharing Collab using the ebrains.collaboratory.api.Collab client.
Replace with uiSelectDataSharingCollabInfo as this function has an almost identical implementation
…nfo" to "nansen.module.sharebrain.internal.uiSelectDatasetCollab".
These functions will be part of the EBRAINS Services Toolbox
Replaces returning an empty array with an error when the specified file does not exist, improving error handling and making missing files explicit.
…iles nansen.module.sharebrain.type.Session and .Subject extend the NANSEN types with the values that sharebrain.importKGSubjects takes from the EBRAINS Knowledge Graph: the subject's identifiers in the Knowledge Graph, and the age, weight and attributes of its subject state on each session. They were in the sharebrain package of the SHAREbrain repository, which env sharebrain does not add to the path; tables of these classes do not open without them. A dataset cloned from an EBRAINS bucket holds empty files. The file .ebrains-bucket.json at the root of the clone names the bucket (writeBucketInfo, findBucketInfo). EbrainsBucketSource implements nansen.dataio.RemoteFileSource: it counts an empty file below such a folder as stored online and downloads it with ebrains.bucket.downloadFile, using the path relative to the clone as the object name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DSM converter of NANSEN assigns a variable the first file adapter of the project that supports its file extension. With this module in a project, .abf files get ABF, .nwb files NWB, and .txt, .csv and .tsv files DelimitedText. ABF reads the file with abfload (Harald Hentschke and Forrest Collman, BSD licence, in the private folder) and returns the samples with the sampling rate, channel names and units, and the header. NWB returns the NwbFile from nwbRead. DelimitedText returns a table from readtable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
abfload stopped with "internal variables 'lSynchArraynnn' are zero or negative" for ABF 1 files recorded in a fixed-length mode without a synch array, such as the files of garad-2022. It reads the sweeps one after the other whether or not there is a synch array, and takes only the sweep length and the sweep start times from it. Without the array, the sweep length is now the number of data points divided by episodes and channels, which abfload already requires to divide evenly, and the start times are left unknown. For 170518_1a.ABF of garad-2022 the file size equals the data offset plus two bytes per sample, and the ten 700 ms sweeps range from -71 mV to +38 mV. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JSON returns the decoded content. NIfTI returns the image and header from niftiread and niftiinfo, for .nii and .nii.gz. NeoPickle (.pkl), NeoNix (.nio, .nix) and IgorPro (.ibw, .pxp) read formats with no MATLAB reader through the Python package Neo. readWithNeo runs neo_to_mat.py with the Python that pyenv names, as a separate process because h5py, which nixio loads, can conflict with the HDF5 library of MATLAB. The script writes the block to a .mat file: segments with analog and irregular signals, spike trains, events and epochs, times in seconds, rates in Hz and units as text. Lists, such as the segments and the labels of events, are cell arrays, also when they are empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
XML returns the content read with readstruct: elements become fields, and an element's attributes become fields ending in Attribute. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The derived traces of several HBP datasets (.dat, e.g. in kozlov-2019-projection-neurons) start with 149 lines that begin with # and hold the protocol, stimulus and units, followed by two columns of numbers. readtable took parts of the header for column names. DelimitedText now reads lines that start with # as comments and joins the comment lines at the top of a file in the table's Properties.Description. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mechanism
NANSEN finds a module's file adapters in its
+fileadapterfolder and gives an imported variable the first adapter that supports its file type. A dataset cloned from an EBRAINS bucket holds an empty file for each object in the bucket.Changes
nansen.module.sharebrain.type.Sessionand.Subjectextend the NANSEN types with the Knowledge Graph subject identifiers and, per session, the subject state's age, age category, weight and attributes.ABF(vendoredabfload, with its BSD licence inprivate/; returnsData,SamplingRate,ChannelNames,ChannelUnits,Header),NWB(nwbRead),DelimitedText(readtablewith#lines as comments; the#header of a file is kept inProperties.Description),JSON,XML(readstruct),NIfTI(niftiinfo/niftiread,.niiand.nii.gz), andNeoPickle,NeoNixandIgorPro. The last three runprivate/neo_to_mat.pywith the Python thatpyenvnames, as a separate process because nixio loads h5py, and load the.matfile it writes: segments with signals, spike trains, events and epochs, times in seconds, rates in Hz, units as text, and lists as cell arrays also when they are empty.abfloadpatch: an ABF 1 file recorded in a fixed-length mode without a synch array (every file of garad-2022) is read with the sweep length taken from the number of points, episodes and channels; the start times of the sweeps are left unknown. A file without episodes gives an error.dataio.EbrainsBucketSourceimplementsnansen.dataio.RemoteFileSource: an empty file below a folder with.ebrains-bucket.jsonis online-only and is downloaded withebrains.bucket.downloadFile, with the path relative to the clone as the object name.ebrains.writeBucketInfoandfindBucketInfowrite and find that file (constants.BucketInfoFilename).EbrainsBucketSourceTest,FileAdapterTest(DelimitedText, JSON, XML, NIfTI, NeoPickle, NeoNix, NWB; the Neo tests write their files with Neo in Python). ABF and IgorPro have no test, because MATLAB cannot write either format.Depends on VervaekeLab/NANSEN#148, which adds
nansen.dataio.RemoteFileSource.🤖 Generated with Claude Code