Capture build and run metadata automatically across the modeling workflow

[chaseshyu]

Problem

The metadata that answers "what produced this result?" exists, but none of
it survives the workflow, and nothing captures it automatically:

• Build identity (revision, options, dependency versions, toolchain) lives
  only in snapshot.diff — per checkout, overwritten by every make, never
  attached to a binary or a run. A 2D opt=0 run and a 3D optimized run share
  a hostname; a g++ and an nvc++ build of the same source do not agree bit for
  bit — a wall-clock without the identity of that run's binary is not
  comparable.
• Run identity is printed once to stdout and scrolls away; host detail
  beyond the CPU name (cores, hybrid split, RAM) is never probed, and the
  thread count a run actually got is never measured —
  omp_get_max_threads() is only an upper bound under OMP_DYNAMIC.
• Outputs are anonymous: a .save frame or a run directory handed over on
  its own names neither its build nor its host.

The result: comparing timings, reproducing a run, or trusting an old
checkpoint all rest on human bookkeeping that nobody does.

Proposal

Capture metadata automatically at the three points of the workflow where it
is born — each artifact self-identifying, one source of truth per fact. Each
item lists its full field inventory (from the preserved implementations);
comments on what to add or drop are the point of this issue.

1. Build time — a build.snapshot block embedded in the executable,
   read without running the binary (strings <exe> | grep '^build\.snapshot\.')
   or printed at run start via DES_BUILD_SNAPSHOT=1. One
   topic : key=value line per group:

   group	fields
   code	rev (git describe against version tags), branch, dirty (file/±line counts only), origin (credential-stripped), state_utc (when this build's source/configuration state last changed -- deliberately not the link time)
   options	ndims + every make knob that is on or carries a value (opt openmp openacc hdf5 usemmg useexo use_gospl nprof gprof nofma snapshot_diff), gpu_cc on ACC builds
   output	container (vtkhdf | des-binary), layout (BINARY_FILE_REVISION)
   submodules	knn-bvh rev (ACC builds), nanoflann, mmg
   libs	hdf5, openmp (spec date), libcxx/libstdcxx, boost — each with its provider label (brew conda user submodule toolchain system nvhpc)
   toolchain	compiler (nvc++-aware), target (arch-os)
   defines / cxxflags / ldflags / warnings	the path-free flag words, importance-ordered
   host	os, builder (user@host), build_utc (link time, from the executable's mtime via the manifest)

   make snapshot_diff=1 additionally embeds the uncommitted code changes
   (the snapshot.diff summary) — off by default for privacy, and a binary
   carrying them names snapshot_diff=1 on its options line:

   section	content
   summary	the HEAD commit (git show -s) and git status, untracked files included
   not checked-in	git diff — the uncommitted modification, indented three spaces
   not in origin	commit subjects only (git log --oneline origin..)
   fallback	one plain line when git or the repository is unavailable

2. Run start — <modelname>.manifest in the output directory, written
   unconditionally by every run: one plain text file carrying the build and
   the initial runtime state together. Being text is the point: cat
   answers it, no strings or h5dump needed. Truncated on a fresh run,
   appended on restart, so a run resumed on another machine keeps both
   records. Fields:

   group	fields
   run	modelname, restarting (+ restart_from_model/frame), run_start_utc
   build	a quote of the executable's embedded build.snapshot block
   host	os, runner (user@host, the run-side counterpart of builder), cpu_model, physical/logical/perf/eff_cores, mem_total_gb, mem_free_at_start_gb (a snapshot of what other processes left, not comparable across runs)
   device	kernel (CPU | GPU), name, device (N of M), driver, mem_total_gb, mem_free_at_start_gb
   threads	omp_threads (measured — omp_get_max_threads() is only an upper bound), omp_max_threads, omp_num_procs, omp_dynamic, omp_wait_policy_eff
   env	only the variables that are set: OMP_NUM_THREADS, OMP_PROC_BIND, OMP_PLACES, KMP_BLOCKTIME, CUDA_VISIBLE_DEVICES, ACC_DEVICE_TYPE, NVCOMPILER_ACC_DEVICE_TYPE

3. Output writing — root-group attributes on every .save/.chkpt
.vtkhdf frame, so a single frame names both its origin and how the
   run was doing when it was written:

   group	fields
   identity	beside the existing ndims/revision: the build's code and host groups and the run's host and device groups, as tabled above — a lone frame then explains itself without .manifest
   status at write time	host memory in use; device memory in use on GPU runs
   date	write_utc (when this frame was written)

Controls

• make snapshot_diff=1 (build time): embed the uncommitted code changes.
  Off by default for privacy; a binary carrying them says so on its options
  line.
• DES_BUILD_SNAPSHOT=1 (run time): print the embedded block as [Snapshot]
  lines, first on stdout. Off by default; an environment variable rather
  than a config parameter because it acts before the config file is read.
  The code-changes payload never prints — strings is its read path.

Design notes

• Build facts come from compiler macros plus a make-generated header,
  content-compared ignoring its own timestamp: the clock alone never rewrites
  it, so state_utc stamps the last state change and a no-change make
  relinks nothing.
• Run facts are measured, not assumed: the thread count in a throwaway
  parallel region; the device from init_offload_device(), the one function
  that selects.
• No filesystem paths in any artifact; dependency providers (brew, conda,
  submodule, toolchain) are classified labels, never prefixes.
• Degrade, never fail: without git the embedded block says so in one line;
  unknown values are unknown/-1 sentinels consumers must reject; a failed
  .manifest open warns and returns — metadata must not abort a run.

Alternatives

• Fold into .info.
• Fold into snapshot.diff,
• Console-only reporting.

Done when

Every artifact of a modeling run — the binary, the log, the run directory,
each output frame — names its build and run identity with no user action,
and README documents the three read paths.

Questions for reviewers

1. Field set — anything to add or drop in the tables above?
2. Is snapshot_diff worth having at all? dirty and state_utc
   already say that the build was modified; embedding what changed is
   opt-in and self-labeled, but is the payoff worth the machinery?

[tan2]

I haven't read the proposal in detail, but I like most of it.

However, I would hope my use case is covered in this proposal. I want my model result to be reproducible, ie., with the metadata and input cfg file, I can rerun the model and get (more or less) the same results, so that I don't need to archive/backup the model results.

To achieve this, the snapshot of the source code used to build the binary must contain:

• the diff of any uncommitted local modifications
• the diff of unpublished commits
• the compiler name, version, and flags
• all environmental variables that might affect the build

This proposal also records information about the runtime environment, which is good. I would prefer that the metadata be stored in text files rather than binary files.

[chaseshyu]

The design can just keep snapshot.diff as it is, which wouldn't change the workflow for the users using snapshot.diff now. The major feature will be making executable files carry metadata in text format, which can be taken out by strings <exe> | grep '^build\.snapshot\.' or some command like this without executing it. However, the most controversial question is if embedding uncommitted code changes into the executable. To address this, I think it would be great to include a privacy control option, so users can decide whether they want to include those changes or not.