Fix wrong variable names - #15
Merged
Merged
Conversation
The checks were driven by whatever a file happened to contain, filtered by
the data request:
for ivar in file_variables:
if ivar in ismip_var:
...
So a file whose variable had been misspelled inside -- `lim_....nc` holding
`limm` -- matched nothing, the loop body never ran, and the file was reported
as "No errors. Good job !" without a single numerical, spatial, time or
attribute check having been performed on it. That is issue ismip#11.
The same loop had a quieter failure. A `lim_....nc` holding `limnsw` was
checked against limnsw's row of the data request rather than lim's, so its
units, value range, fill value and time axis were all verified against the
wrong criteria and passed. The one error it did produce -- a standard_name
mismatch -- was incidental, and would not have appeared at all had the two
variables shared a standard_name or had the request left it blank.
The file name states which variable of the request a file carries, so it is
the file name, not the file's contents, that decides what to check and which
row to check it against. `_check_file_variables` then requires the file to
actually contain that variable, and says so with the near-miss spelling when
there is one, which is what turns silence into an error. A file that fails
this has nothing left to check, so the remaining checks are skipped for it,
as they already are for the other naming errors that make a file
uncheckable.
Files that are correct are unaffected: the same variable is checked against
the same row as before, and the golden reference log gains only the new
per-file OK line.
Verified: 33 tests, green. Two are new and both fail before this change,
reporting zero errors where they now require one -- `lim_....nc` holding
`limm`, and holding `limnsw`. The second also requires that the incidental
standard_name error is gone, which is the evidence that the swapped-in
variable is no longer what gets checked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ISMIP7 convention is one requested variable per file, and until now nothing enforced it: a file could carry any number of extra variables and the checker would not mention them. With the preceding commit the extras are not even incidentally checked any more, since the checks now read the variable the file name promises, so without this they would pass through entirely unseen. What counts as extra needs saying carefully, because CF lets the requested variable bring companions that plainly belong in the file. The bounds of a coordinate are the case that matters in practice -- `time_bounds` is written into every FL file by the request's own encoding rules -- and a variable may also point at a container variable through `grid_mapping` or name auxiliary coordinates through `coordinates`. `_allowed_file_variables` admits exactly those, found by following the attributes that refer to them rather than by matching names, so a group that calls its bounds variable something else is not punished for it. Both `.attrs` and `.encoding` are consulted, since xarray moves CF attributes between the two as it decodes. An extra variable does not make a file uncheckable, so unlike a missing variable it is reported and the file's remaining checks still run. Verified: 34 tests, green. The new one adds a `mask` variable to a scalar file and requires both the error and that `lim` is still checked afterwards. The allow-list is guarded by the tests that were already there: seven of the synthetic scalar files and thirteen of the golden-log files carry `time_bounds`, and all of them must stay at zero errors, which they do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The data request gives each variable a shape in its `Dim` column, and until now nothing compared it against the file. A variable could be correctly named and still be the wrong thing -- a two-dimensional `lithk` that had lost its y axis, say -- and the checker would run its numerical, spatial and time tests over it without complaint, because those read the file's coordinates and never look at the variable's own dimensions. `REQUESTED_DIMENSIONS` is the translation between the two spellings, and the four entries in it are every `Dim` the request uses. They differ in more than notation: the request writes dimensions innermost-first and calls the time dimension `t`, while the files write them outermost-first in the conventional CF order and call it `time`. Only the set is compared here; order is the following commit. A `Dim` the table does not know is reported as unchecked rather than guessed at or passed over silently, so adding a form to the request cannot quietly disable this check. The x/y coordinate test already in `_check_naming` is left where it is. It asks a different question -- whether the dataset has x and y coordinates at all -- and it returns early, so the two cannot both fire on one file. Verified: 36 tests, green. Two of the new ones are spatial, which needed a baseline the suite did not have: an x,y,t dataset, generated with non-mandatory variables included because refgeoid (x,y) and litemp (x,y,z,t) are the only variables of their shape in the request and two of the four forms would otherwise go unexercised. It passes with zero errors and logs all three spatial forms; the scalar baseline covers t. The third new test rebuilds a lithk file with its y dimension dropped and requires exactly one error, which is also what shows the rebuild leaves every other check seeing what it saw before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CF and the ISMIP7 request both prescribe the T-Z-Y-X order, and a file that departs from it passes every check this tool has. The spatial tests read the x and y coordinates, the numerical tests take a minimum and a maximum, and neither has any reason to look at how the variable's axes are arranged, so a `lithk` written as (time, x, y) is reported as correct today. That is a gap rather than a refinement: the file is not compliant, and nothing says so. Order is compared only once the set of dimensions matches, so a variable that is wrong in both respects is reported as having the wrong dimensions rather than as being transposed, which is the more useful of the two messages and avoids reporting one fault twice. Verified: 37 tests, green. The new one rebuilds a lithk file as (time, x, y) and requires exactly one error, naming it as an order problem. The golden reference log is untouched, as it must be: a file that passes already logged its dimensions line in the preceding commit, and this adds no output of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The variable field of a file name was looked up in the data request, and a
file that did not match was passed over without a word:
if considered_variable in ismip_var:
...
So `bogusvar_GrIS_....nc` was reported as "No errors. Good job !". For a
mandatory variable there was at least the indirect signal that the variable
it should have been was missing; for a non-mandatory one there was no signal
at all.
The obvious `else` would be wrong, because `ismip_var` is the request as
narrowed by `--variable-list`, and the README has modelers check a directory
once for its scalars and once for its x,y,t files. Under that reading every
`lithk_....nc` becomes an error during the scalars pass, which is not a
statement the checker is entitled to make: those files are not being checked,
not failing. So `_load_criteria` now also returns the request before it is
narrowed, and the error is reserved for a name that is in neither.
Verified: 39 tests, green. One requires the error, along with the
missing-mandatory-variable error that was previously the only sign of it.
The other is the guard for the distinction above: an x,y,t directory checked
with `--variable-list ismip7_scalars` reports nothing new, which is what
fails if the narrowed request is used for this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The naming category has grown from a claim about a file's name to a claim about its contents, and the summary at the top is what modelers read to know what the tool will hold them to. The line about what the test suite mutates was likewise a checklist of failures the suite no longer stops at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The loop meant to list the naming problems at the top of the log was:
for i in range(iline, len(report_naming_issues)):
contents.insert(i, " - " + report_naming_issues[i - 24] + "\n")
`iline` is around 24 by the time it runs, so `range(24, len(issues))` is
empty for any realistic number of issues and the loop was a no-op: the
"Naming tests errors report:" header was written and nothing followed it.
Had a run ever produced more than 24 issues it would have started listing
them from the wrong index instead. So the summary a modeler reads first has
never carried the naming problems it announces.
That was harmless while naming issues were rare. The preceding commits make
them common -- a misspelled variable inside a file, a file name that names no
variable of the request -- so the report is now worth having. The header is
also keyed to the list rather than to the naming error count, since several
naming errors are logged per-file without an entry here and would otherwise
print a header over nothing.
Verified: 39 tests, green, one of which now requires the missing-variable
message to appear in the synthesis and not only in the detailed results. The
golden reference log is unchanged, since it has no naming errors to report.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Check the variables inside a file, not just the file name
Fixes #11.
The problem
A file could be named correctly, contain the wrong variable, and be reported as
No errors. Good job !. The checks were driven by whatever a file happened to contain, filtered by the data request —for ivar in file_variables: if ivar in ismip_var:— so if nothing in the file was a requested name, the loop body never ran and the file accrued zero errors without a single numerical, spatial, time or attribute check having been performed on it.Reproduced against
mainbefore making any change, by mutating one file of the seeded synthetic scalar dataset per case:lim_….ncholds a variable namedlimmlim_….ncholdslimnswinstead oflimstandard_namemismatch, after units, ranges, fill value and time had all been checked against limnsw's row of the requestbogusvar_….nclimmissing"Case A is the issue as filed. Case B is the same blind spot doing something worse than staying silent: it checked the file against the wrong criteria and passed it, and would have passed silently too had the two variables shared a
standard_nameor had the request left it blank. Case C had no signal whatsoever for a non-mandatory variable.The fix
The file name states which variable of the data request a file carries, so it is the file name, not the file's contents, that now decides what gets checked and which row of the request to check it against. On top of that the naming category gained four checks, all of them reported as naming errors and listed in the synthesis at the top of the log.
The file contains the variable its name promises. Reported with the near-miss spelling where there is one, so the log says
'limm' may be a misspelling of 'lim'rather than leaving the reader to spot it. A file that fails this has nothing left to check, so its remaining checks are skipped, as they already were for the other naming errors that make a file uncheckable.The file holds nothing else. One requested variable per file is the convention, and nothing enforced it. What counts as extra needed care, because CF lets the requested variable bring companions that plainly belong in the file:
_allowed_file_variablesadmits the bounds of a coordinate (which is howtime_boundsreaches every FL file), the container variable agrid_mappingpoints at, and any auxiliary coordinates named throughcoordinates. They are found by following the attributes that refer to them rather than by matching names, so a group that calls its bounds variable something else is not punished for it.The variable has the dimensions the request asks for, in the conventional order.
REQUESTED_DIMENSIONStranslates the request'sDimcolumn into the dimensions of a file — the two differ in more than notation, since the request writes them innermost-first and calls the time dimensiont, while the files write them outermost-first in CF order and call ittime. A transposed variable passes every other check this tool has, because the spatial tests read the x and y coordinates and the numerical tests take a minimum and a maximum, and neither has any reason to look at how a variable's axes are arranged.The variable field of the file name is a variable of the request. The obvious
elsewould have been wrong here:ismip_varis the request as narrowed by--variable-list, and the README has modelers check a directory once for its scalars and once for its x,y,t files, under which reading everylithk_….ncbecomes an error during the scalars pass. Those files are not being checked, not failing, so the error is reserved for a name that is in neither the narrowed nor the full request, and there is a test guarding the distinction.Also in here
_insert_synthesisannounced a "Naming tests errors report:" and then listed nothing, because its loop wasfor i in range(iline, len(report_naming_issues))withilinealready around 24 — an empty range for any realistic number of issues, and the wrong index had there ever been more than 24. It was harmless while naming issues were rare; these commits make them common, so the report is now worth having and now works.Commits
Seven, each one independently reviewable, each leaving the suite green and the golden log consistent. Only the first is a prerequisite for the others, and only the dimension-order commit depends on the dimension-set commit.
Testing
39 tests, green, up from 31. Eight are new, and each of the new checks has a test that fails without its commit. The suite gained an x,y,t baseline, which it did not have before — dimension order needs a variable with more than one dimension — generated with non-mandatory variables included, because
refgeoid(x,y) andlitemp(x,y,z,t) are the only variables of their shape in the request and two of the four dimension forms would otherwise go unexercised. It passes with zero errors.The two dimension tests rebuild a
lithkfile with its shape changed, since netCDF cannot reshape a variable in place; every other variable, every attribute and the unlimited dimension are copied across unchanged, and each test requiring exactly one error is what shows the rebuild leaves every other check seeing what it saw before.The golden reference log gains three lines per file and loses none — the three new OK lines, checked one commit at a time so that each diff was purely additive and of a single form. The commits that should not have touched it did not.
Compatibility
Files that are correct are unaffected: the same variable is checked against the same row of the request as before, and passing files gain only the new OK lines in their log. Every new error is a file that was previously being reported as clean while being wrong, or in case C not reported at all.