Skip to content

Unify experiment layout; reorganize hpc/config/; add plots and sweeper improvements - #11

Closed
juliendrapeau wants to merge 1 commit into
devfrom
sync
Closed

juliendrapeau wants to merge 1 commit into
devfrom
sync

Conversation

@juliendrapeau

@juliendrapeau juliendrapeau commented May 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Consolidate data/, results/, and sweeps/ under experiments/<stage>/<exp>/ to keep all per-experiment artifacts co-located.
  • Reorganize hpc/ config files into hpc/config/; replace hardcoded CLUSTER_ROOTS dict with a clusters.yaml file that users copy from a template.
  • Add workflow/plots/ matplotlib style infrastructure (common utilities + 4 mplstyles: default, one-column, two-column, presentation).
  • Add use_hash option and duplicate-path detection to ConfigSet.slugify_all.

Changes

experiments/ directory structure

data/, results/, and sweeps/ are replaced by a unified experiments/ tree:

experiments/<stage>/<exp>/
  sweeps/          ← sweep scripts and summary.csv (tracked example)
  data/            ← per-job outputs (gitignored)
  results/         ← aggregated CSVs (gitignored)

Updated Snakefile, rules, workflow_example.yaml, docs, and all template snippets. The old data/ and results/ root directories remain gitignored for backward compat.

hpc/config/ reorganization

hpc/config_example.yaml and hpc/submit_example.yaml move to hpc/config/. A new hpc/config/clusters_example.yaml replaces the hardcoded CLUSTER_ROOTS dict in lifecycle.py: users copy it to hpc/config/clusters.yaml (gitignored) and fill in their SSH host aliases and scratch paths. The --profile hpc flag in run_snakemake.sh is updated to --profile hpc/config.

workflow/plots/ matplotlib style infrastructure

New workflow/plots/ package with common.py (use_style, read_results_csv, style constants) and four mplstyles: default, one_column, two_column, and presentation.

ConfigSet.slugify_all improvements

  • use_hash=True uses each config's content hash as the directory name instead of a human-readable slug — guarantees uniqueness when slugs would collide.
  • After assigning paths, duplicate detection raises ValueError listing all collisions.
  • Both options are threaded through generate_and_save.

Version

Bumped to 0.5.0.

@github-actions

Copy link
Copy Markdown

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
59 55 93% 90% 🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: 2cc818f by action🐍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant