Repository navigation
Fix silent correctness bugs; improve SIF build tooling and job provenance - #9
Merged
Merged
Conversation
Replace random.randint + np.random.seed/random.seed with np.random.default_rng().integers() for seed generation. Callers needing reproducibility should use the stored seed with np.random.default_rng(metadata["seed"]). Ported from sgcode/dev PR #40.
pd.read_csv represents empty cells as NaN (float). Without sanitization, optional None params (e.g. degree bounds) arrive as truthy floats and silently corrupt downstream logic. Added _clean() helper that maps NaN → None. Ported from sgcode/dev PR #42.
build_sif.py: - Add fast path when apptainer is available locally (docker --load + apptainer build docker-daemon://) — skips intermediate OCI archive - Add _ssh_flag() that only passes --ssh default when agent has keys - Add BuildKit layer cache in the OCI fallback path - Replace importlib_metadata (third-party) with stdlib importlib.metadata - Remove tqdm; use simple labelled print() instead - Remove importlib-metadata dev dependency job_utils.py: - capture_job_metadata() now records SLURM_JOB_ID, SLURM_JOB_PARTITION, SLURM_CPUS_PER_TASK, SLURM_MEM_PER_CPU, SLURM_NODELIST, SLURM_TIMELIMIT, SLURM_ARRAY_TASK_ID (all None on local runs) - capture_run_metadata() reads container path from APPTAINER_CONTAINER / SINGULARITY_CONTAINER env vars instead of manual CONTAINER_DIGEST; computes SHA-256 digest of the SIF file; records container_image name
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
values from empty CSV cells silently corrupting Snakemake rule params
build_sif.py: fast local-apptainer path, SSH safety check, BuildKit cache, removetqdmand third-partyimportlib_metadatajob_utils.pywith full SLURM metadata capture and automatic container provenanceResolves #7.
Changes
Fix global RNG state in
sweeper.pyConfig.default_metadatawas callingrandom.seed(seed)andnp.random.seed(seed)as a sideeffect of generating a config. This mutated global state on every
Configinstantiation, causingnon-obvious test interference and making independent RNG streams impossible. Replaced with
np.random.default_rng().integers(2**32)for seed generation (no global state). Callers needingreproducibility should use
np.random.default_rng(metadata["seed"])locally.NaN sanitization in Snakefile
pd.read_csvrepresents empty CSV cells asNaN(float). Without sanitization, any optionalparameter left blank in the sweep CSV arrives at the Snakemake rule as a truthy float and silently
passes incorrect values downstream. Added
_clean()helper that mapsNaN → Nonewhen buildingparams_by_path.build_sif.pyrefactoredapptainer/singularityis installed locally and building for the hostplatform, uses
docker buildx build --load+apptainer build docker-daemon://instead ofwriting an intermediate OCI archive — substantially faster for native builds.
_ssh_flag()only passes--ssh defaultwhenssh-add -lconfirms keys areloaded; previously it was always passed, causing failures on machines without an SSH agent.
--cache-from/--cache-toto avoid rebuildingunchanged layers.
importlib_metadatawith stdlibimportlib.metadata;removed
importlib-metadataandtqdmfrom dev dependencies.SLURM metadata and container provenance in
job_utils.pycapture_job_metadata()now records SLURM env vars (SLURM_JOB_ID,SLURM_JOB_PARTITION,SLURM_CPUS_PER_TASK,SLURM_MEM_PER_CPU,SLURM_NODELIST,SLURM_TIMELIMIT,SLURM_ARRAY_TASK_ID) — previously none of this was captured, making it impossible to correlateresults with cluster allocation details after the fact. All fields are
Noneon local runs.capture_run_metadata()now reads the container path fromAPPTAINER_CONTAINER/SINGULARITY_CONTAINER(set automatically when running inside Apptainer) rather than the manually-set
CONTAINER_DIGESTenv var, and computes a SHA-256 digest of the SIF file for exact reproducibility.
Version
Bumped to 0.4.0.