Repository navigation
feat(session): download data files that are stored online - #148
Draft
ehennestad wants to merge 1 commit into
Draft
ehennestad wants to merge 1 commit into
ehennestad wants to merge 1 commit into
Conversation
A data location can hold placeholders for files that are stored online, such as the empty files of a cloned EBRAINS bucket. A project now names a subclass of nansen.dataio.RemoteFileSource with Project.setRemoteFileSource. The source tells which files are placeholders and downloads them. Session.loadData checks each file before loading it. When the file is a placeholder, it downloads the file if the project preference AutoDownloadRemoteFiles is true (Project.setAutoDownloadRemoteFiles), and otherwise raises NANSEN:Session:FileIsOnlineOnly, which names Session.downloadDataFile. Session.downloadDataFile downloads the file of one variable. A project without a source, and a session object used where no user session is active, such as in a parallel worker, load files as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ehennestad
added this pull request to stack #122
September 19, 2026 16:41
ehennestad
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mechanism
Session.loadDataresolves a variable's file and reads it with the variable's file adapter. A data location can hold empty placeholder files for a dataset that is stored online, such as a cloned EBRAINS bucket.Problem
Loading a placeholder failed with "file is empty (0 bytes)", and NANSEN had no way to fetch the file.
Changes
nansen.dataio.RemoteFileSourcewithisOnlineOnly(filePath)anddownload(filePath). A project names its subclass withProject.setRemoteFileSource(className), which checks that the class inherits fromRemoteFileSource.loadDataasks the source before the empty-file check. WithProject.setAutoDownloadRemoteFiles(true)it downloads the file; otherwise it raisesNANSEN:Session:FileIsOnlineOnly, which namesSession.downloadDataFile. A project without a source, and a session object used where no user session is active (for example in a parallel worker), load files as before.Session.downloadDataFile(varName)downloads the file of one variable.RemoteFileDownloadTestwithRemoteFileSourceFake, andLoadDataWithoutUserSessionTest.🤖 Generated with Claude Code