Fix time check - #16
Merged
Merged
Conversation
The time checks inferred a run's length from its endpoints and its cadence from its first interval, so neither could see what happened in between. A ctrl file holding 2015, 2016, 2299, 2300 starts in the right year, ends in the right year and has a 365-day first interval: it passed every time check while being 282 time steps short. Decimated axes, mid-run gaps and dropped segments all passed the same way. Reconstruct the axis the file should have had instead -- the nominal years from experiments_ismip7.csv, encoded per the variable's ST/FL convention -- and compare it with the axis the file has. One comparison subsumes the step count, the cadence, the first and last timestamps and the duration, and it can say which years are missing rather than only that something is wrong. This also collapses the two near-identical start/end branches (the -1 duration case and the fixed-duration case) that between them repeated the same two timestamp-window comparisons. The filename year range moves with it. _check_naming now only parses the range and checks it is well formed, handing the parsed years to _check_time, which decodes the file anyway; previously both decoded it separately and each carried its own copy of the ST/FL year offset. Errors about the range therefore count under Time rather than Naming. Two constants go: TIME_STEP_MIN_DAYS and TIME_STEP_MAX_DAYS encoded the 365-vs-366 latitude that comparing real calendar dates gives for free. Adds a full-length 286-year ctrl fixture, which nothing exercised before, and tests for the axes that used to pass: hollowed out between correct endpoints, decimated, shifted by a year, duplicated, disagreeing with the file name, and written with the wrong ST/FL convention. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The snapshot check validated only the nominal years a litemp file happened to contain, and built the set of valid years partly out of the file's own last time step, so whatever a file ended on was correct by construction and nothing looked for snapshots that should have been there and were not. A litemp file holding one snapshot at an arbitrary year passed. Check against a required set instead: the run's final year, the century marks falling inside the run, and -- for historical only -- the run's first year. Historical is the one experiment whose start year the modeller chooses, so it is the one whose initial state is not recorded anywhere else; a projection's initial state is the historical run's final state, already reported as historical's last-year snapshot. Missing snapshots are now errors, which is what issue ismip#12 asks for. The required set follows the data request, which lists 1900, the initial state, 2014, 2100, 2200 and 2300 -- and not 2000, which this checker, the generator and the README have all required until now. hgoelzer is away until mid-August, so 2000 is dropped from the requirement but tolerated when present: the generator no longer writes it, and a file that carries it is not failed for a year that is still in dispute. See ISSUE_12_COMMENT.md. The generator imports CENTURY_SNAPSHOT_YEARS from the checker rather than keeping a second copy, so the two cannot drift apart again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_run_variable_checks abandoned a file the moment _check_naming reported anything, so a single unrecognised ESM name meant the numerical, spatial, time and attribute checks never ran. The file's report read "1 error(s)", the modeller fixed the name, re-ran, and only then learned about the twelve real problems that had been there all along. For a tool people run before submitting, that turns one pass into as many passes as the file has independent problems -- and it hid exactly the time errors this branch adds. Most of what _check_naming reports says nothing about whether a file can be read: an ISM member id, an ESM name, a forcing member id, a set counter and a malformed year range are all reported and moved past. Only a missing x or y dimension really stops everything, and NamingResult.can_continue now says so explicitly rather than the caller inferring it from an error count. An unrecognised region used to be fatal as well, because three checks index by it -- and _check_spatial quietly fell back to the GrIS extents, checking the file against another ice sheet's criteria. Each now says it is skipping and why, so a file with a typo'd region field still gets its time axis checked, which is the case this branch cares about most. Guards ds[ivar].attrs["units"], which raised KeyError rather than reporting a missing attribute. It was unreachable for most files before, since a file that got that far was usually well-formed; more files reaching _check_numerical makes it worth fixing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xylar
added a commit
that referenced
this pull request
Jul 26, 2026
The log has promised warnings since it was written and never delivered any: two lines were hardcoded, "No warnings." under every file's footer and "0 warning(s) detected." in the synthesis. With no warning severity in the counting path, anything worth mentioning at all had to be reported as an error -- which is why the extra litemp snapshots of PR #16 are errors today. Add warning() alongside error(), counted separately by category, and make the four places that report counts tell the truth: the per-file footer, the synthesis block, the console summaries and run_checker's summary dict. Warnings are kept clear of the verdict throughout: a file with warnings and no errors still reads "No errors. Good job !", and the console phrases its warning count so it cannot be mistaken for a failure. No check emits a warning yet, so every count is still zero. The synthesis block does gain the per-category warning breakdown that mirrors the error one, which is the only change in the reference log. Unit tests for the reporter itself: the line formats, the split between severities and categories, and the roll-up from a file's sub-total to the run's total. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xylar
added a commit
that referenced
this pull request
Jul 26, 2026
PR #16 left this gap on the record: litemp is the data request's only x,y,z,t variable, and so the only one whose sparse snapshot axis _check_time looks at, but it is not mandatory and the golden fixture did not ask for non-mandatory variables. The whole snapshot branch was therefore checked only by assertions, never by a stored log. The static x,y refgeoid was in the same position. Parametrise the fixture and add a second reference generated with include_non_mandatory=True. The existing reference keeps its meaning and is unchanged; the new one covers 42 files rather than 27. Beyond closing the gap, this gives the severity changes that follow a visible diff in a committed log rather than only a number in an assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xylar
added a commit
that referenced
this pull request
Jul 26, 2026
Two findings settle together here. An x,y,z,t file carrying a snapshot the experiment does not call for was made an error in PR #16, and that was a stopgap: no warning severity existed, so the choice was between failing the file and saying nothing. It is a warning. The data request specifies these years as a *minimum* set, so over-delivering 3D temperature is not non-compliance -- but a year nobody asked for is usually a sign of a mistake, so it is worth naming. The asymmetry with the annual axis, where an extra year stays an error, is deliberate: that axis is pinned end to end by experiments_ismip7.csv, so an extra year there means the file does not match the experiment it names. A snapshot at 2000 then needs no special case. It was accepted in silence, which told a modeller following the old README nothing at all; reported as an unrequested snapshot it gets the honest answer -- it is here, it was not asked for, it is not held against you -- and TOLERATED_SNAPSHOT_YEARS goes. Neither reference log changes; the generator writes only required snapshots. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partially addresses #12 but with an open question.
What was wrong
The time checks inferred a run's length from its endpoints and its cadence from its first interval, so nothing looked at what happened in between. A
ctrlfile whose time axis holds 2015, 2016, 2299, 2300 starts in the right year, ends in the right year, and has a 365-day first interval — it passed every time check while being 282 time steps short of the 286 required. Decimated axes, mid-run gaps and dropped segments all passed the same way. I confirmed this againstmainrather than inferring it: the old checker reports 0 errors on that file, the new one reports 2.The snapshot (
x,y,z,t, i.e.litemp) check had the matching gap. It validated only the nominal years a file happened to contain, and built its set of valid years partly out of the file's own last time step, so whatever a file ended on was correct by construction and nothing ever looked for snapshots that should have been present and were not. Alitempfile holding a single snapshot at an arbitrary year reported 0 time errors onmain; it now reports 1.What changed
Reconstruct the expected axis and compare. The nominal years come from
experiments_ismip7.csvand are encoded with the variable's ST/FL convention, giving the exact list of timestamps the file should hold. One comparison then subsumes the step count, the cadence, the first and last timestamps and the duration — and it can say which years are missing rather than only that something is off. Messages collapse consecutive runs, so a 99-year hole reads as2101-2199rather than as ninety-nine separate findings.Filename year range moves into the time check.
_check_namingnow only parses the range and checks it is well formed, handing the parsed years to_check_time, which decodes the file anyway. Previously both functions decoded the dataset separately and each carried its own copy of the ST/FL year offset. Errors about the year range therefore now count under Time rather than Naming in the summary — expected, and the main reason the golden log moved.Required snapshot sets for
x,y,z,t. The required set is the run's final year, the century marks falling inside the run, and — forhistoricalonly — the run's first year.historicalis the one experiment whose start year the modeller chooses, so it is the one whose initial state is not recorded anywhere else; a projection's initial state is the historical run's final state, already reported as historical's last-year snapshot. Missing snapshots are now errors.A file is checked as far as it can be checked.
_run_variable_checksused to abandon a file the moment_check_namingreported anything, so one unrecognised ESM name meant the numerical, spatial, time and attribute checks never ran — the report read "1 error(s)", the modeller fixed the name, re-ran, and only then learned about everything else. That also hid exactly the time errors this PR adds. Only a missingx/ydimension now stops the rest. An unrecognised region costs just the three checks that index by it (value range, grid extent/resolution,crs), each of which now says it is skipping and why; previously_check_spatialquietly fell back to the GrIS extents and checked the file against the wrong ice sheet's criteria.One decision that needs @hgoelzer's confirmation
The repo documents two different sets of required snapshot years, differing only over 2000.
ISMIP7_variable_request.csvlists "(1900 if in historical), initial state, 2014, 2100, 2200, and 2300"; the README,checker.pyandgenerate.pyall instead used{1900, 2000, 2100, 2200, 2300}. Since the check now errors on missing snapshots, the required set has to be right in a way it did not before.This PR follows the data request and does not require 2000, but tolerates it when present — the generator no longer writes it, and a file that carries it is not failed for a year still in dispute. That keeps files written to the earlier README passing. See #12.
What reviewers will notice
The golden log shrinks by 81 lines: four verbose time lines per file collapse into one, e.g.
Time step: 365 days/Experiment lasts 2 years/First time stamp correctly set to…/Last time stamp correctly set to…becomeTime axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK. Error counts on the reference dataset are unchanged at zero.Error counts on broken files will rise, sometimes sharply, because files that used to abort after their naming block now run the remaining four blocks. Those errors were always present; the checker was not reaching them.
TIME_STEP_MIN_DAYSandTIME_STEP_MAX_DAYSare gone — the 365-vs-366 latitude they encoded comes free from comparing real calendar dates.generate.pyimportsCENTURY_SNAPSHOT_YEARSfromchecker.pyrather than keeping a second copy, so the generator and the checker cannot drift apart again.Tests
51 tests, up from 39. New coverage for the axes that used to pass: hollowed out between correct endpoints, decimated, shifted by a year, duplicated, disagreeing with the file name, and written with the wrong ST/FL convention; plus
litempmissing a required snapshot, carrying an unrequested year, and carrying the tolerated 2000. A full-length 286-yearctrlfixture is added — nothing exercised the real ISMIP7 projection length before. Two tests assert the decoupling directly: a file with a bad ESM name still reports its time errors, and a file with an unrecognised region still gets everything that does not depend on the region.Known gap
litempis non-mandatory and the golden-log fixture does not passinclude_non_mandatory, so the snapshot path has no golden-log coverage. The dedicated tests cover it. Adding it to the fixture would be worth doing but would add several hundred lines to the reference file, so I left it out of this PR.