Skip to content

Fix time check - #16

Merged
xylar merged 3 commits into
ismip:mainfrom
xylar:fix-time-check
Jul 26, 2026
Merged

Fix time check#16
xylar merged 3 commits into
ismip:mainfrom
xylar:fix-time-check

Conversation

@xylar

@xylar xylar commented Jul 26, 2026

Copy link
Copy Markdown
Member

Partially addresses #12 but with an open question.

What was wrong

The time checks inferred a run's length from its endpoints and its cadence from its first interval, so nothing looked at what happened in between. A ctrl file whose time axis holds 2015, 2016, 2299, 2300 starts in the right year, ends in the right year, and has a 365-day first interval — it passed every time check while being 282 time steps short of the 286 required. Decimated axes, mid-run gaps and dropped segments all passed the same way. I confirmed this against main rather than inferring it: the old checker reports 0 errors on that file, the new one reports 2.

The snapshot (x,y,z,t, i.e. litemp) check had the matching gap. It validated only the nominal years a file happened to contain, and built its set of valid years partly out of the file's own last time step, so whatever a file ended on was correct by construction and nothing ever looked for snapshots that should have been present and were not. A litemp file holding a single snapshot at an arbitrary year reported 0 time errors on main; it now reports 1.

What changed

Reconstruct the expected axis and compare. The nominal years come from experiments_ismip7.csv and are encoded with the variable's ST/FL convention, giving the exact list of timestamps the file should hold. One comparison then subsumes the step count, the cadence, the first and last timestamps and the duration — and it can say which years are missing rather than only that something is off. Messages collapse consecutive runs, so a 99-year hole reads as 2101-2199 rather than as ninety-nine separate findings.

Filename year range moves into the time check. _check_naming now only parses the range and checks it is well formed, handing the parsed years to _check_time, which decodes the file anyway. Previously both functions decoded the dataset separately and each carried its own copy of the ST/FL year offset. Errors about the year range therefore now count under Time rather than Naming in the summary — expected, and the main reason the golden log moved.

Required snapshot sets for x,y,z,t. The required set is the run's final year, the century marks falling inside the run, and — for historical only — the run's first year. historical is the one experiment whose start year the modeller chooses, so it is the one whose initial state is not recorded anywhere else; a projection's initial state is the historical run's final state, already reported as historical's last-year snapshot. Missing snapshots are now errors.

A file is checked as far as it can be checked. _run_variable_checks used to abandon a file the moment _check_naming reported anything, so one unrecognised ESM name meant the numerical, spatial, time and attribute checks never ran — the report read "1 error(s)", the modeller fixed the name, re-ran, and only then learned about everything else. That also hid exactly the time errors this PR adds. Only a missing x/y dimension now stops the rest. An unrecognised region costs just the three checks that index by it (value range, grid extent/resolution, crs), each of which now says it is skipping and why; previously _check_spatial quietly fell back to the GrIS extents and checked the file against the wrong ice sheet's criteria.

One decision that needs @hgoelzer's confirmation

The repo documents two different sets of required snapshot years, differing only over 2000. ISMIP7_variable_request.csv lists "(1900 if in historical), initial state, 2014, 2100, 2200, and 2300"; the README, checker.py and generate.py all instead used {1900, 2000, 2100, 2200, 2300}. Since the check now errors on missing snapshots, the required set has to be right in a way it did not before.

This PR follows the data request and does not require 2000, but tolerates it when present — the generator no longer writes it, and a file that carries it is not failed for a year still in dispute. That keeps files written to the earlier README passing. See #12.

What reviewers will notice

The golden log shrinks by 81 lines: four verbose time lines per file collapse into one, e.g. Time step: 365 days / Experiment lasts 2 years / First time stamp correctly set to… / Last time stamp correctly set to… become Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK. Error counts on the reference dataset are unchanged at zero.

Error counts on broken files will rise, sometimes sharply, because files that used to abort after their naming block now run the remaining four blocks. Those errors were always present; the checker was not reaching them.

TIME_STEP_MIN_DAYS and TIME_STEP_MAX_DAYS are gone — the 365-vs-366 latitude they encoded comes free from comparing real calendar dates. generate.py imports CENTURY_SNAPSHOT_YEARS from checker.py rather than keeping a second copy, so the generator and the checker cannot drift apart again.

Tests

51 tests, up from 39. New coverage for the axes that used to pass: hollowed out between correct endpoints, decimated, shifted by a year, duplicated, disagreeing with the file name, and written with the wrong ST/FL convention; plus litemp missing a required snapshot, carrying an unrequested year, and carrying the tolerated 2000. A full-length 286-year ctrl fixture is added — nothing exercised the real ISMIP7 projection length before. Two tests assert the decoupling directly: a file with a bad ESM name still reports its time errors, and a file with an unrecognised region still gets everything that does not depend on the region.

Known gap

litemp is non-mandatory and the golden-log fixture does not pass include_non_mandatory, so the snapshot path has no golden-log coverage. The dedicated tests cover it. Adding it to the fixture would be worth doing but would add several hundred lines to the reference file, so I left it out of this PR.

xylar and others added 3 commits July 26, 2026 09:03
The time checks inferred a run's length from its endpoints and its cadence
from its first interval, so neither could see what happened in between.  A
ctrl file holding 2015, 2016, 2299, 2300 starts in the right year, ends in
the right year and has a 365-day first interval: it passed every time check
while being 282 time steps short.  Decimated axes, mid-run gaps and dropped
segments all passed the same way.

Reconstruct the axis the file should have had instead -- the nominal years
from experiments_ismip7.csv, encoded per the variable's ST/FL convention --
and compare it with the axis the file has.  One comparison subsumes the step
count, the cadence, the first and last timestamps and the duration, and it
can say which years are missing rather than only that something is wrong.

This also collapses the two near-identical start/end branches (the -1
duration case and the fixed-duration case) that between them repeated the
same two timestamp-window comparisons.

The filename year range moves with it.  _check_naming now only parses the
range and checks it is well formed, handing the parsed years to _check_time,
which decodes the file anyway; previously both decoded it separately and
each carried its own copy of the ST/FL year offset.  Errors about the range
therefore count under Time rather than Naming.

Two constants go: TIME_STEP_MIN_DAYS and TIME_STEP_MAX_DAYS encoded the
365-vs-366 latitude that comparing real calendar dates gives for free.

Adds a full-length 286-year ctrl fixture, which nothing exercised before,
and tests for the axes that used to pass: hollowed out between correct
endpoints, decimated, shifted by a year, duplicated, disagreeing with the
file name, and written with the wrong ST/FL convention.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The snapshot check validated only the nominal years a litemp file happened to
contain, and built the set of valid years partly out of the file's own last
time step, so whatever a file ended on was correct by construction and nothing
looked for snapshots that should have been there and were not.  A litemp file
holding one snapshot at an arbitrary year passed.

Check against a required set instead: the run's final year, the century marks
falling inside the run, and -- for historical only -- the run's first year.
Historical is the one experiment whose start year the modeller chooses, so it
is the one whose initial state is not recorded anywhere else; a projection's
initial state is the historical run's final state, already reported as
historical's last-year snapshot.  Missing snapshots are now errors, which is
what issue ismip#12 asks for.

The required set follows the data request, which lists 1900, the initial
state, 2014, 2100, 2200 and 2300 -- and not 2000, which this checker, the
generator and the README have all required until now.  hgoelzer is away until
mid-August, so 2000 is dropped from the requirement but tolerated when present:
the generator no longer writes it, and a file that carries it is not failed for
a year that is still in dispute.  See ISSUE_12_COMMENT.md.

The generator imports CENTURY_SNAPSHOT_YEARS from the checker rather than
keeping a second copy, so the two cannot drift apart again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_run_variable_checks abandoned a file the moment _check_naming reported
anything, so a single unrecognised ESM name meant the numerical, spatial,
time and attribute checks never ran.  The file's report read "1 error(s)",
the modeller fixed the name, re-ran, and only then learned about the twelve
real problems that had been there all along.  For a tool people run before
submitting, that turns one pass into as many passes as the file has
independent problems -- and it hid exactly the time errors this branch adds.

Most of what _check_naming reports says nothing about whether a file can be
read: an ISM member id, an ESM name, a forcing member id, a set counter and a
malformed year range are all reported and moved past.  Only a missing x or y
dimension really stops everything, and NamingResult.can_continue now says so
explicitly rather than the caller inferring it from an error count.

An unrecognised region used to be fatal as well, because three checks index
by it -- and _check_spatial quietly fell back to the GrIS extents, checking
the file against another ice sheet's criteria.  Each now says it is skipping
and why, so a file with a typo'd region field still gets its time axis
checked, which is the case this branch cares about most.

Guards ds[ivar].attrs["units"], which raised KeyError rather than reporting a
missing attribute.  It was unreachable for most files before, since a file
that got that far was usually well-formed; more files reaching _check_numerical
makes it worth fixing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@xylar
xylar merged commit dd65c5b into ismip:main Jul 26, 2026
4 checks passed
@xylar
xylar deleted the fix-time-check branch July 26, 2026 07:11
xylar added a commit that referenced this pull request Jul 26, 2026
The log has promised warnings since it was written and never delivered
any: two lines were hardcoded, "No warnings." under every file's footer
and "0 warning(s) detected." in the synthesis.  With no warning severity
in the counting path, anything worth mentioning at all had to be reported
as an error -- which is why the extra litemp snapshots of PR #16 are
errors today.

Add warning() alongside error(), counted separately by category, and make
the four places that report counts tell the truth: the per-file footer,
the synthesis block, the console summaries and run_checker's summary
dict.  Warnings are kept clear of the verdict throughout: a file with
warnings and no errors still reads "No errors. Good job !", and the
console phrases its warning count so it cannot be mistaken for a failure.

No check emits a warning yet, so every count is still zero.  The
synthesis block does gain the per-category warning breakdown that mirrors
the error one, which is the only change in the reference log.

Unit tests for the reporter itself: the line formats, the split between
severities and categories, and the roll-up from a file's sub-total to the
run's total.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xylar added a commit that referenced this pull request Jul 26, 2026
PR #16 left this gap on the record: litemp is the data request's only
x,y,z,t variable, and so the only one whose sparse snapshot axis
_check_time looks at, but it is not mandatory and the golden fixture did
not ask for non-mandatory variables.  The whole snapshot branch was
therefore checked only by assertions, never by a stored log.  The static
x,y refgeoid was in the same position.

Parametrise the fixture and add a second reference generated with
include_non_mandatory=True.  The existing reference keeps its meaning and
is unchanged; the new one covers 42 files rather than 27.

Beyond closing the gap, this gives the severity changes that follow a
visible diff in a committed log rather than only a number in an
assertion.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xylar added a commit that referenced this pull request Jul 26, 2026
Two findings settle together here.

An x,y,z,t file carrying a snapshot the experiment does not call for was
made an error in PR #16, and that was a stopgap: no warning severity
existed, so the choice was between failing the file and saying nothing.
It is a warning.  The data request specifies these years as a *minimum*
set, so over-delivering 3D temperature is not non-compliance -- but a
year nobody asked for is usually a sign of a mistake, so it is worth
naming.  The asymmetry with the annual axis, where an extra year stays an
error, is deliberate: that axis is pinned end to end by
experiments_ismip7.csv, so an extra year there means the file does not
match the experiment it names.

A snapshot at 2000 then needs no special case.  It was accepted in
silence, which told a modeller following the old README nothing at all;
reported as an unrequested snapshot it gets the honest answer -- it is
here, it was not asked for, it is not held against you -- and
TOLERATED_SNAPSHOT_YEARS goes.

Neither reference log changes; the generator writes only required
snapshots.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant