Skip to content

Fix wrong variable names - #15

Merged
xylar merged 7 commits into
ismip:mainfrom
xylar:fix-wrong-variable-names
Jul 25, 2026
Merged

Fix wrong variable names#15
xylar merged 7 commits into
ismip:mainfrom
xylar:fix-wrong-variable-names

Conversation

@xylar

@xylar xylar commented Jul 25, 2026

Copy link
Copy Markdown
Member

Check the variables inside a file, not just the file name

Fixes #11.

The problem

A file could be named correctly, contain the wrong variable, and be reported as No errors. Good job !. The checks were driven by whatever a file happened to contain, filtered by the data request — for ivar in file_variables: if ivar in ismip_var: — so if nothing in the file was a requested name, the loop body never ran and the file accrued zero errors without a single numerical, spatial, time or attribute check having been performed on it.

Reproduced against main before making any change, by mutating one file of the seeded synthetic scalar dataset per case:

Case Mutation Before After
A lim_….nc holds a variable named limm 0 errors, "No errors. Good job !" 1 naming error
B lim_….nc holds limnsw instead of lim 1 error, but incidental — a standard_name mismatch, after units, ranges, fill value and time had all been checked against limnsw's row of the request 1 naming error, and the incidental one is gone
C file renamed bogusvar_….nc file itself reports "No errors"; only the indirect "mandatory variable lim missing" 1 naming error plus the missing-variable error

Case A is the issue as filed. Case B is the same blind spot doing something worse than staying silent: it checked the file against the wrong criteria and passed it, and would have passed silently too had the two variables shared a standard_name or had the request left it blank. Case C had no signal whatsoever for a non-mandatory variable.

The fix

The file name states which variable of the data request a file carries, so it is the file name, not the file's contents, that now decides what gets checked and which row of the request to check it against. On top of that the naming category gained four checks, all of them reported as naming errors and listed in the synthesis at the top of the log.

The file contains the variable its name promises. Reported with the near-miss spelling where there is one, so the log says 'limm' may be a misspelling of 'lim' rather than leaving the reader to spot it. A file that fails this has nothing left to check, so its remaining checks are skipped, as they already were for the other naming errors that make a file uncheckable.

The file holds nothing else. One requested variable per file is the convention, and nothing enforced it. What counts as extra needed care, because CF lets the requested variable bring companions that plainly belong in the file: _allowed_file_variables admits the bounds of a coordinate (which is how time_bounds reaches every FL file), the container variable a grid_mapping points at, and any auxiliary coordinates named through coordinates. They are found by following the attributes that refer to them rather than by matching names, so a group that calls its bounds variable something else is not punished for it.

The variable has the dimensions the request asks for, in the conventional order. REQUESTED_DIMENSIONS translates the request's Dim column into the dimensions of a file — the two differ in more than notation, since the request writes them innermost-first and calls the time dimension t, while the files write them outermost-first in CF order and call it time. A transposed variable passes every other check this tool has, because the spatial tests read the x and y coordinates and the numerical tests take a minimum and a maximum, and neither has any reason to look at how a variable's axes are arranged.

The variable field of the file name is a variable of the request. The obvious else would have been wrong here: ismip_var is the request as narrowed by --variable-list, and the README has modelers check a directory once for its scalars and once for its x,y,t files, under which reading every lithk_….nc becomes an error during the scalars pass. Those files are not being checked, not failing, so the error is reserved for a name that is in neither the narrowed nor the full request, and there is a test guarding the distinction.

Also in here

_insert_synthesis announced a "Naming tests errors report:" and then listed nothing, because its loop was for i in range(iline, len(report_naming_issues)) with iline already around 24 — an empty range for any realistic number of issues, and the wrong index had there ever been more than 24. It was harmless while naming issues were rare; these commits make them common, so the report is now worth having and now works.

Commits

Seven, each one independently reviewable, each leaving the suite green and the golden log consistent. Only the first is a prerequisite for the others, and only the dimension-order commit depends on the dimension-set commit.

  1. Check that a file contains the variable its name promises — the presence check together with the switch to filename-driven checking. Deliberately one commit and not two: separating them would leave an intermediate state where the checks are driven by a variable that may not exist.
  2. Report unexpected variables in a file
  3. Check variable dimensions against the data request — the set of dimensions
  4. Report a variable whose dimensions are in the wrong order
  5. Report a file name whose variable is not in the data request
  6. Document the new variable checks in the README
  7. Fix the naming-issues report in the log synthesis — a pre-existing bug, last in the series so it is easy to drop or move to its own issue

Testing

39 tests, green, up from 31. Eight are new, and each of the new checks has a test that fails without its commit. The suite gained an x,y,t baseline, which it did not have before — dimension order needs a variable with more than one dimension — generated with non-mandatory variables included, because refgeoid (x,y) and litemp (x,y,z,t) are the only variables of their shape in the request and two of the four dimension forms would otherwise go unexercised. It passes with zero errors.

The two dimension tests rebuild a lithk file with its shape changed, since netCDF cannot reshape a variable in place; every other variable, every attribute and the unlimited dimension are copied across unchanged, and each test requiring exactly one error is what shows the rebuild leaves every other check seeing what it saw before.

The golden reference log gains three lines per file and loses none — the three new OK lines, checked one commit at a time so that each diff was purely additive and of a single form. The commits that should not have touched it did not.

Compatibility

Files that are correct are unaffected: the same variable is checked against the same row of the request as before, and passing files gain only the new OK lines in their log. Every new error is a file that was previously being reported as clean while being wrong, or in case C not reported at all.

xylar and others added 7 commits July 25, 2026 21:17
The checks were driven by whatever a file happened to contain, filtered by
the data request:

    for ivar in file_variables:
        if ivar in ismip_var:
            ...

So a file whose variable had been misspelled inside -- `lim_....nc` holding
`limm` -- matched nothing, the loop body never ran, and the file was reported
as "No errors. Good job !" without a single numerical, spatial, time or
attribute check having been performed on it.  That is issue ismip#11.

The same loop had a quieter failure.  A `lim_....nc` holding `limnsw` was
checked against limnsw's row of the data request rather than lim's, so its
units, value range, fill value and time axis were all verified against the
wrong criteria and passed.  The one error it did produce -- a standard_name
mismatch -- was incidental, and would not have appeared at all had the two
variables shared a standard_name or had the request left it blank.

The file name states which variable of the request a file carries, so it is
the file name, not the file's contents, that decides what to check and which
row to check it against.  `_check_file_variables` then requires the file to
actually contain that variable, and says so with the near-miss spelling when
there is one, which is what turns silence into an error.  A file that fails
this has nothing left to check, so the remaining checks are skipped for it,
as they already are for the other naming errors that make a file
uncheckable.

Files that are correct are unaffected: the same variable is checked against
the same row as before, and the golden reference log gains only the new
per-file OK line.

Verified: 33 tests, green.  Two are new and both fail before this change,
reporting zero errors where they now require one -- `lim_....nc` holding
`limm`, and holding `limnsw`.  The second also requires that the incidental
standard_name error is gone, which is the evidence that the swapped-in
variable is no longer what gets checked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ISMIP7 convention is one requested variable per file, and until now
nothing enforced it: a file could carry any number of extra variables and
the checker would not mention them.  With the preceding commit the extras
are not even incidentally checked any more, since the checks now read the
variable the file name promises, so without this they would pass through
entirely unseen.

What counts as extra needs saying carefully, because CF lets the requested
variable bring companions that plainly belong in the file.  The bounds of a
coordinate are the case that matters in practice -- `time_bounds` is written
into every FL file by the request's own encoding rules -- and a variable may
also point at a container variable through `grid_mapping` or name auxiliary
coordinates through `coordinates`.  `_allowed_file_variables` admits exactly
those, found by following the attributes that refer to them rather than by
matching names, so a group that calls its bounds variable something else is
not punished for it.  Both `.attrs` and `.encoding` are consulted, since
xarray moves CF attributes between the two as it decodes.

An extra variable does not make a file uncheckable, so unlike a missing
variable it is reported and the file's remaining checks still run.

Verified: 34 tests, green.  The new one adds a `mask` variable to a scalar
file and requires both the error and that `lim` is still checked afterwards.
The allow-list is guarded by the tests that were already there: seven of the
synthetic scalar files and thirteen of the golden-log files carry
`time_bounds`, and all of them must stay at zero errors, which they do.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The data request gives each variable a shape in its `Dim` column, and until
now nothing compared it against the file.  A variable could be correctly
named and still be the wrong thing -- a two-dimensional `lithk` that had lost
its y axis, say -- and the checker would run its numerical, spatial and time
tests over it without complaint, because those read the file's coordinates
and never look at the variable's own dimensions.

`REQUESTED_DIMENSIONS` is the translation between the two spellings, and the
four entries in it are every `Dim` the request uses.  They differ in more
than notation: the request writes dimensions innermost-first and calls the
time dimension `t`, while the files write them outermost-first in the
conventional CF order and call it `time`.  Only the set is compared here;
order is the following commit.  A `Dim` the table does not know is reported
as unchecked rather than guessed at or passed over silently, so adding a form
to the request cannot quietly disable this check.

The x/y coordinate test already in `_check_naming` is left where it is.  It
asks a different question -- whether the dataset has x and y coordinates at
all -- and it returns early, so the two cannot both fire on one file.

Verified: 36 tests, green.  Two of the new ones are spatial, which needed a
baseline the suite did not have: an x,y,t dataset, generated with
non-mandatory variables included because refgeoid (x,y) and litemp (x,y,z,t)
are the only variables of their shape in the request and two of the four
forms would otherwise go unexercised.  It passes with zero errors and logs
all three spatial forms; the scalar baseline covers t.  The third new test
rebuilds a lithk file with its y dimension dropped and requires exactly one
error, which is also what shows the rebuild leaves every other check seeing
what it saw before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CF and the ISMIP7 request both prescribe the T-Z-Y-X order, and a file that
departs from it passes every check this tool has.  The spatial tests read the
x and y coordinates, the numerical tests take a minimum and a maximum, and
neither has any reason to look at how the variable's axes are arranged, so a
`lithk` written as (time, x, y) is reported as correct today.  That is a gap
rather than a refinement: the file is not compliant, and nothing says so.

Order is compared only once the set of dimensions matches, so a variable that
is wrong in both respects is reported as having the wrong dimensions rather
than as being transposed, which is the more useful of the two messages and
avoids reporting one fault twice.

Verified: 37 tests, green.  The new one rebuilds a lithk file as
(time, x, y) and requires exactly one error, naming it as an order problem.
The golden reference log is untouched, as it must be: a file that passes
already logged its dimensions line in the preceding commit, and this adds no
output of its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The variable field of a file name was looked up in the data request, and a
file that did not match was passed over without a word:

    if considered_variable in ismip_var:
        ...

So `bogusvar_GrIS_....nc` was reported as "No errors. Good job !".  For a
mandatory variable there was at least the indirect signal that the variable
it should have been was missing; for a non-mandatory one there was no signal
at all.

The obvious `else` would be wrong, because `ismip_var` is the request as
narrowed by `--variable-list`, and the README has modelers check a directory
once for its scalars and once for its x,y,t files.  Under that reading every
`lithk_....nc` becomes an error during the scalars pass, which is not a
statement the checker is entitled to make: those files are not being checked,
not failing.  So `_load_criteria` now also returns the request before it is
narrowed, and the error is reserved for a name that is in neither.

Verified: 39 tests, green.  One requires the error, along with the
missing-mandatory-variable error that was previously the only sign of it.
The other is the guard for the distinction above: an x,y,t directory checked
with `--variable-list ismip7_scalars` reports nothing new, which is what
fails if the narrowed request is used for this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The naming category has grown from a claim about a file's name to a claim
about its contents, and the summary at the top is what modelers read to know
what the tool will hold them to.  The line about what the test suite mutates
was likewise a checklist of failures the suite no longer stops at.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The loop meant to list the naming problems at the top of the log was:

    for i in range(iline, len(report_naming_issues)):
        contents.insert(i, "  - " + report_naming_issues[i - 24] + "\n")

`iline` is around 24 by the time it runs, so `range(24, len(issues))` is
empty for any realistic number of issues and the loop was a no-op: the
"Naming tests errors report:" header was written and nothing followed it.
Had a run ever produced more than 24 issues it would have started listing
them from the wrong index instead.  So the summary a modeler reads first has
never carried the naming problems it announces.

That was harmless while naming issues were rare.  The preceding commits make
them common -- a misspelled variable inside a file, a file name that names no
variable of the request -- so the report is now worth having.  The header is
also keyed to the list rather than to the naming error count, since several
naming errors are logged per-file without an entry here and would otherwise
print a header over nothing.

Verified: 39 tests, green, one of which now requires the missing-variable
message to appear in the synthesis and not only in the detailed results.  The
golden reference log is unchanged, since it has no naming errors to report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@xylar xylar added the bug Something isn't working label Jul 25, 2026
@xylar xylar self-assigned this Jul 25, 2026
@xylar
xylar merged commit c104eea into ismip:main Jul 25, 2026
4 checks passed
@xylar
xylar deleted the fix-wrong-variable-names branch July 25, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check for wrong variable names in netcdf files

1 participant