diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index 7158315..1476e84 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -63,4 +63,8 @@ jobs: # package: a data file or entry point that did not make it into the # wheel then fails here instead of passing against the source tree. working-directory: ${{ runner.temp }} - run: pytest -v "$GITHUB_WORKSPACE/tests/test_compliance_checker.py" + # The whole directory, not one file: test_golden_log.py exists to catch + # a dependency release inside the supported ranges changing the log, and + # that is exactly what running it here at both the floor and the latest + # environment is for. + run: pytest -v "$GITHUB_WORKSPACE/tests" diff --git a/README.md b/README.md index 283f171..f45263f 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,11 @@ Checks ISMIP7 NetCDF simulation datasets for compliance with the [ISMIP7 data request conventions](https://www.ismip.org/). The following categories are validated for every file: -1. **Naming** — variable name, region field, ISM member id (`mNNN`), ESM name (CMIP6/CMIP7 registry), forcing member id (`fNNN`), set counter (`[C|E|P]NNN`), and year range (well formed `YYYY-YYYY`; what the range *means* is checked under **Time**). Inside the file: the variable the file name names is the one the file contains, with the dimensions the data request asks for, in the conventional `(time, z, y, x)` order, and the file holds nothing else beyond its coordinates and any bounds or grid-mapping variables. +1. **Naming** — variable name, region field, ISM member id (`mNNN`), ESM name (CMIP6/CMIP7 registry), forcing member id (`fNNN`), set counter (`[C|E|P]NNN`), and year range (well formed `YYYY-YYYY`; what the range *means* is checked under **Time**). Inside the file: the variable the file name names is the one the file contains, with the dimensions the data request asks for, in the conventional `(time, z, y, x)` order, and the file holds nothing else beyond its coordinates and the companion variables CF lets them name (`bounds`, `grid_mapping`, `coordinates`, `cell_measures`, `ancillary_variables`) — anything further is a warning. 2. **Numerical** — units match the data request, in any UDUNITS spelling (`m2`, `m^2` and `m**2` are all accepted, as are `kg m-2 s-1`, `kg.m-2.s-1` and `kg/m2/s`); all values lie within the allowed min/max range for the relevant region; array is not entirely fill values. 3. **Spatial** *(xyt variables only)* — grid corners lie within the expected AIS or GrIS extents; resolution is one of the allowed values; x and y cell size are equal. 4. **Time** — time dimension is present, unlimited, and monotonically increasing; the file name's year range is one the experiment allows; and the time axis is **exactly** the axis the experiment calls for. For `x,y,t` and `t` variables that means every nominal year from `experiments_ismip7.csv`, each carrying the timestamp its ST/FL convention prescribes; for `x,y,z,t` variables it means the required set of sparse snapshots (see [Time encoding](#time-encoding)). -5. **Attributes** — required global and coordinate attributes are present and have correct values; `standard_name` matches data request; `_FillValue` equals the NetCDF4 default for the variable's dtype; variable and time are float32; `scale_factor` and `add_offset` are not allowed. +5. **Attributes** — required global and coordinate attributes are present and have correct values; `standard_name` matches data request; `_FillValue` equals the NetCDF4 default for the variable's dtype; the variable is float32, and so is the time coordinate (a warning — one number per record cannot inflate a file); `scale_factor` and `add_offset` are not allowed. Every file is checked as far as it can be. A naming problem stops the other checks only where it leaves them nothing to read — a missing `x` or `y` dimension, or a file that does not contain the variable its name promises. Everything else (a mistyped ESM name, a malformed year range, an unrecognised region) is reported and the file is checked on, so one run tells you everything that is wrong rather than only the first thing. An unrecognised region costs just the checks that depend on it: value range, grid extent and resolution, and `crs`. @@ -14,6 +14,49 @@ Compliance criteria are defined in `isschecker/data/ISMIP7_variable_request.csv` --- +## Errors and warnings + +Findings come at two severities, and the difference is worth stating precisely, because a check that reports at the wrong one either fails a submission that is fine or waves through one that is not. + +**ERROR** — the file, as written, is unusable for the intended analysis, departs from the protocol in a way that changes the science, or fails the data-hygiene requirements this archive is committing to. That last clause is deliberate: the output will be served to the broader community for analysis for years, so uniformity of encoding is a product requirement rather than a stylistic preference, and "a reader could cope with it" is not grounds for a warning. + +**WARNING** — the file is usable, the science is unaffected, and nothing downstream has to work around it, but it departs from what the data request asked for in a way you should look at and may reasonably have intended. + +Three consequences follow, and they are what make a warning safe to leave alone: + +- Warnings never enter the error count and never change a file's verdict. A file with warnings and no errors is compliant, and the log says so: `No errors. Good job !`, followed by the number of warnings to review. +- Warnings never affect the exit status. Errors do. +- A check whose failure means the checker could not read something is always an error. A warning never stops any later check from running. + +The synthesis block at the top of the log counts both severities, broken down by the same categories. + +### Variables your model does not represent + +An experiment that carries no files for a non-mandatory variable gets one warning naming all of them. This is expected if your model does not represent those variables — GIA is the obvious case — and it is listed only so that a variable lost from a submission does not pass unnoticed. It is scoped to the `--variable-list` you selected, so a run over `ismip7_scalars` says nothing about the `x,y,t` variables it was never asked to look at. + +To make that warning go quiet, put an optional `not_modelled.txt` in the `--source-path` directory: one variable name per line, blank lines ignored and `#` starting a comment, so you can record alongside each name why the variable is absent. + +``` +# ISMIP7: variables this model does not represent. +dlithkdt # no GIA in this configuration +litemp +``` + +Two rules keep the file from becoming a way to hide problems, and both are errors rather than silent no-ops: + +- A **mandatory** variable named in it is still a missing-mandatory error, and the claim itself is reported as a further error. The list is a statement about optional variables; a submission cannot opt out of the data request with it. +- A name that is **not in the data request** at all is an error. It is either a typo or a misunderstanding, and both are better said plainly than left to be inferred from a warning that did not go away. + +Whatever the file declares is echoed into the log, so the archived record of a run shows what was claimed rather than merely that a warning did not appear. If the file is absent, nothing changes. + +### Value ranges + +Some of the `min_value_*` / `max_value_*` bounds in the data request "are dependent on the forcing, input data and model implementation" ([issue #10](https://github.com/ismip/ISM_SimulationChecker/issues/10)), and some are not, so the severity of an out-of-range value is a per-variable question. It is answered by the `range_severity` column of `ISMIP7_variable_request.csv`, which holds `error` or `warning` for each variable row. Anything the column does not say — a blank cell, an unrecognised value, or the column being absent altogether — means `error`. + +Every shipped row is currently `error`, so this changes nothing today. Moving a variable to `warning` is a one-cell change to the data request that needs no reasoning about the checker. + +--- + ## Time encoding ISMIP7 uses the standard (Gregorian) CF calendar with time recorded as **days since 1850-01-01 00:00:00**. The encoding convention differs by variable type: @@ -40,9 +83,11 @@ The filename year range (`YYYY-YYYY`) always refers to the **nominal simulation Together, a `historical` run and a projection provide snapshots at 1900, 2014, 2100, 2200 and 2300, plus the first year of the historical run. The first year is required only for `historical`, whose start year the modeller chooses; a projection's initial state is the historical run's final state, already reported as historical's last-year snapshot. -The checker reports **missing** required snapshots as well as snapshots the experiment does not call for. The filename year range for `litemp` reflects the full simulation year range (e.g. `2015-2300`), not the first/last snapshot year, and the annual cadence checks do not apply. +A **missing** required snapshot is an error. A snapshot the experiment does not call for is a **warning**: the data request specifies these years as a minimum set, so over-delivering 3D temperature is not non-compliance, but a year nobody asked for is usually a sign that something was written by mistake. (The annual time axis is treated the other way round — an extra year there is an error — because that axis is pinned end to end by `experiments_ismip7.csv`, so an extra year means the file does not match the experiment it names.) -> **A snapshot at 2000 is accepted but not required.** Earlier versions of this README, the checker and the generator all required one; `ISMIP7_variable_request.csv` does not ask for one. The data request is being followed until [issue #12](https://github.com/ismip/ISM_SimulationChecker/issues/12) settles it, and 2000 is tolerated in the meantime so that files written to the earlier guidance still pass. +The filename year range for `litemp` reflects the full simulation year range (e.g. `2015-2300`), not the first/last snapshot year, and the annual cadence checks do not apply. + +> **A snapshot at 2000 is not required.** Earlier versions of this README, the checker and the generator all required one; `ISMIP7_variable_request.csv` does not ask for one. A file carrying one is reported as an unrequested snapshot — that is, warned about and not failed — until [issue #12](https://github.com/ismip/ISM_SimulationChecker/issues/12) settles it. Files written to the earlier guidance still pass. Reference lookup tables are available in the companion repository [`ismip7-time-encoding`](https://github.com/ismip/ismip7-time-encoding). @@ -113,6 +158,10 @@ ismip7-compliance-checker --source-path ./Models/GrIS/ISMIP7/SYNTH1/CORE/C001 -- `experiments_ismip7.csv` defines the allowed nominal year ranges and durations for each experiment. The checker derives the expected FL and ST timestamps from these year values at runtime (see [Time encoding](#time-encoding)). +### Exit status + +The checker exits **non-zero** when it found errors, or when it could not check anything at all — the `--source-path` does not exist, or holds no `.nc` files. It exits **zero** when the submission is compliant, including when there are warnings to review; see [Errors and warnings](#errors-and-warnings). Both `ismip7-compliance-checker` and `python -m isschecker` behave the same way, so either can be used in a script. + --- ## Generating synthetic test files diff --git a/isschecker/__main__.py b/isschecker/__main__.py index 10b1b5b..64f4308 100644 --- a/isschecker/__main__.py +++ b/isschecker/__main__.py @@ -3,4 +3,6 @@ from isschecker import main if __name__ == "__main__": - main() + # The console script gets this for free from setuptools; without it here, + # `python -m isschecker` would exit 0 on a failing check. + raise SystemExit(main()) diff --git a/isschecker/checker.py b/isschecker/checker.py index 747d434..76090ba 100644 --- a/isschecker/checker.py +++ b/isschecker/checker.py @@ -4,8 +4,9 @@ # 1. Naming (_check_naming, _check_file_variables) # - Variable field of the filename is a variable of the data request. # - The variable named in the filename is present in the file. -# - The file holds no variables beyond that one, its coordinates, and any -# bounds or grid-mapping variables they refer to. +# - The file holds no variables beyond that one, its coordinates, and the +# companion variables CF lets them name: bounds, grid mapping, cell +# measures and ancillary variables. (warning) # - The variable's dimensions are the ones the data request asks for, in the # conventional (time, z, y, x) order. # - Region field in filename matches the region inferred from the grid (AIS/GrIS). @@ -22,7 +23,9 @@ # 2. Numerical (_check_numerical) # - Variable units match the data request (any UDUNITS spelling of the # requested unit is accepted: 'm2', 'm^2' and 'm**2' are all the same). -# - All values lie within the allowed min/max range for the relevant region. +# - All values lie within the allowed min/max range for the relevant region, +# at the severity that variable's 'range_severity' names (error unless the +# data request says otherwise). # - Array is not entirely fill/missing values. # # 3. Spatial (_check_spatial) [xyt variables only] @@ -40,9 +43,11 @@ # cadence reasoning cannot see. # - For x,y,z,t snapshot variables the axis is instead checked against the # required set of snapshot nominal years: the run's last year, the century -# marks inside it, and (for historical only) the run's first year. Both -# missing and unasked-for snapshots are reported. A snapshot at 2000 is -# tolerated but not required -- see issue #12. +# marks inside it, and (for historical only) the run's first year. A +# missing snapshot is an error; an unasked-for one is a warning, since the +# request specifies the snapshots as a minimum set. That includes a +# snapshot at 2000, which the data request does not ask for -- see +# issue #12. # # 5. Attributes (_check_attributes) # - Global attributes present: group, model, contact_name, contact_email, crs @@ -51,7 +56,8 @@ # - Variable standard_name matches data request (if specified). # - _FillValue must be present and equal the default netCDF4 fill value for the variable's dtype. # If missing_value is also present, it must equal _FillValue. -# - Main variable and time coordinate are single-precision float (float32 / f4). +# - Main variable is single-precision float (float32 / f4); so is the time +# coordinate (warning -- one number per record cannot inflate a file). # - scale_factor and add_offset are not allowed on the main variable. # # A file is checked as far as it can be. A naming problem stops the other @@ -60,6 +66,39 @@ # else (a mistyped ESM name, a malformed year range, an unrecognised region) is # reported and the file is checked on, so that one run tells a modeller # everything that is wrong rather than only the first thing. +# +# +# Errors and warnings +# ------------------- +# +# One rule, stated here once so that a new check is classified by citing it +# rather than by arguing the case again: +# +# ERROR -- the file, as written, is unusable for the intended analysis, +# departs from the protocol in a way that changes the science, or +# fails the data-hygiene requirements this archive is committing +# to. That last clause is deliberate: the output will be served +# to the broader community for analysis for years, so uniformity +# of encoding is a product requirement rather than a stylistic +# preference, and "a reader could cope with it" is not grounds for +# a warning. +# +# WARNING -- the file is usable, the science is unaffected, and nothing +# downstream has to work around it, but it departs from what the +# request asked for in a way the modeller should look at and may +# reasonably have intended. +# +# Three corollaries keep warnings from quietly becoming errors: +# +# - Warnings never enter the error count and never change a file's verdict. +# A file with warnings and no errors is compliant, and is told so. +# - Warnings never affect the exit status. Errors do. +# - A check whose failure means the checker could not read something stays an +# error. Warnings never suppress later checks. +# +# An optional not_modelled.txt in the source directory silences the warning +# about non-mandatory variables an experiment carries no files for, and nothing +# else -- see _read_not_modelled. import datetime @@ -181,16 +220,140 @@ def _read_data_csv(filename: str, **kwargs) -> pd.DataFrame: } -def main() -> None: +class Reporter: + """Writes the log and counts what it wrote, by severity and category. + + Every finding has to reach two places at once: the line a modeller reads in + the log, and the counter the synthesis block at the top of that log adds up. + Keeping those in step by hand meant every check returning a count and every + caller adding it on, sixty call sites of bookkeeping running in parallel with + sixty writes. Here the write is the count: a check says what it found, at + the severity it means, and the arithmetic follows from that alone. + + Reporters nest. :meth:`category` scopes findings to one of the checker's + reporting categories, and :meth:`child` opens a sub-total -- one file's + worth, say -- that still rolls up into its parent, so a file's own footer and + the run-wide synthesis count the same events rather than counting separately + and hoping to agree. + """ + + def __init__( + self, + log_file, + parent: "Reporter | None" = None, + category: str = "", + qualifier: str | None = None, + bullet: str = " - ", + ): + self.log_file = log_file + self._parent = parent + self._category = category + # _check_attributes has always labelled its findings 'ERROR + # (attributes)'. The qualifier reproduces that without every other + # check growing a label of its own. + self._qualifier = qualifier + # File-level findings are bulleted list items under the file's heading; + # experiment-level ones are written flush left. + self._bullet = bullet + self.errors: dict[str, int] = {} + self.warnings: dict[str, int] = {} + + def category( + self, name: str, qualifier: str | None = None, bullet: str = " - " + ) -> "Reporter": + """A reporter that counts what it writes under reporting category `name`.""" + return Reporter( + self.log_file, + parent=self, + category=name, + qualifier=qualifier, + bullet=bullet, + ) + + def child(self) -> "Reporter": + """A reporter with its own sub-totals that still roll up into this one.""" + return Reporter( + self.log_file, + parent=self, + category=self._category, + qualifier=self._qualifier, + bullet=self._bullet, + ) + + @property + def total_errors(self) -> int: + return sum(self.errors.values()) + + @property + def total_warnings(self) -> int: + return sum(self.warnings.values()) + + def error_count(self, category: str) -> int: + return self.errors.get(category, 0) + + def warning_count(self, category: str) -> int: + return self.warnings.get(category, 0) + + def write(self, text: str) -> None: + """Write to the log without reporting a finding: headings and footers.""" + self.log_file.write(text) + + def error(self, message: str, count: int = 1) -> None: + """Report a file that is unusable, or that departs from the protocol. + + `count` is for the findings that read better as one line than as + several -- four mandatory variables missing from an experiment, say -- + but that are still four findings and not one. + """ + self._report("ERROR", message, count) + + def warning(self, message: str, count: int = 1) -> None: + """Report a departure from the request that leaves the file usable. + + Warnings do not enter the error count, do not change a file's verdict + and do not affect the exit status; they say 'look at this', not 'fix + this'. See the policy note at the top of this module. + """ + self._report("WARNING", message, count) + + def ok(self, message: str) -> None: + """Record a check that passed.""" + self.write(f"{self._bullet}{message}\n") + + def note(self, message: str) -> None: + """Record a check that did not run: not applicable, or nothing to read.""" + self.write(f"{self._bullet}{message}\n") + + def _report(self, label: str, message: str, count: int) -> None: + prefix = label if self._qualifier is None else f"{label} ({self._qualifier})" + self.write(f"{self._bullet}{prefix}: {message}\n") + self._count(label, self._category, count) + + def _count(self, label: str, category: str, count: int) -> None: + counter = self.errors if label == "ERROR" else self.warnings + counter[category] = counter.get(category, 0) + count + if self._parent is not None: + self._parent._count(label, category, count) + + +def main() -> int: + """Run the checker and return the process exit status. + + Non-zero when there are errors, or when the run could not check anything at + all; zero when there are only warnings, which is the whole point of having + them. + """ args = _parse_args() source_path = args.source_path variable_list = args.variable_list - run_checker( + summary = run_checker( source_path=source_path, variable_list=variable_list, ) + return 1 if summary["fatal"] or summary["total_errors"] > 0 else 0 + def run_checker( source_path: str, @@ -199,9 +362,13 @@ def run_checker( ): version = _describe_version() if version is None else version experiments_ismip7 = _load_experiments_csv() - ismip_meta, ismip_var, mandatory_variables, all_request_variables = _load_criteria( - variable_list - ) + ( + ismip_meta, + ismip_var, + mandatory_variables, + all_request_variables, + all_mandatory_variables, + ) = _load_criteria(variable_list) summary = _run_compliance_checker( source_path=source_path, @@ -210,6 +377,7 @@ def run_checker( ismip_var=ismip_var, mandatory_variables=mandatory_variables, all_request_variables=all_request_variables, + all_mandatory_variables=all_mandatory_variables, experiments=experiments_ismip7, criteria_file=VARIABLE_REQUEST_CSV, ) @@ -283,6 +451,25 @@ def _parse_args() -> argparse.Namespace: return parser.parse_args() +def _range_severity(value) -> str: + """How a value outside a variable's min/max range should be reported. + + Issue #10 is that some of these bounds "are dependent on the forcing, input + data and model implementation", so failing a run on them is too strong -- + but which ones is a per-variable question that the checker cannot answer + from its side. So the severity lives in the data request, one cell per + variable row, and switching a variable is a data-only diff that needs no + reasoning about the checker. + + Anything the column does not say is an error, which is what makes the + column safe to add before it is filled in: a blank cell, a missing column + and an unrecognised value all mean what the checker did before. + """ + if value is not None and pd.notna(value) and str(value).strip().lower() == "warning": + return "warning" + return "error" + + def _load_criteria(variable_list: str): try: df = _read_data_csv(VARIABLE_REQUEST_CSV) @@ -301,6 +488,15 @@ def _load_criteria(variable_list: str): # rather than reported: checking the scalars of a directory says nothing # about the x,y,t files sitting beside them. all_request_variables = set(df["Variable Name"]) + # Which of them are mandatory, before the narrowing too: not_modelled.txt is + # a statement about the submission rather than about one run of the checker, + # so a mandatory variable named in it has to be caught whichever list the + # run happens to have selected. + all_mandatory_variables = set( + df.loc[ + df["Mandatory (yes/no)"].astype(str).str.lower() == "yes", "Variable Name" + ] + ) if variable_list == "ismip7_xyt": df = df[df["Dim"].isin(["x,y,t", "x,y,z,t", "x,y"])] @@ -316,6 +512,7 @@ def _load_criteria(variable_list: str): "mandatory": 1 if str(row["Mandatory (yes/no)"]).lower() == "yes" else 0, "standard_name": str(row["standard_name"]) if pd.notna(row["standard_name"]) else None, "type": str(row["Type"]) if pd.notna(row["Type"]) else "", + "range_severity": _range_severity(row.get("range_severity")), } for col in df.columns: lc = str(col).lower() @@ -329,7 +526,13 @@ def _load_criteria(variable_list: str): ismip_var = [d["variable"] for d in ismip_meta] ismip_mandatory_var = [d["variable"] for d in ismip_meta if d["mandatory"] == 1] - return ismip_meta, ismip_var, ismip_mandatory_var, all_request_variables + return ( + ismip_meta, + ismip_var, + ismip_mandatory_var, + all_request_variables, + all_mandatory_variables, + ) def _load_experiments_csv(filename: str = EXPERIMENTS_ISMIP7_CSV_FILENAME): @@ -536,12 +739,6 @@ def _describe_convention(var_type: str) -> str: # naming a literal. CENTURY_SNAPSHOT_YEARS = frozenset({1900, 2100, 2200, 2300}) -# Accepted if a file carries it, never required. The README, this checker and -# the generator all used to require a snapshot at 2000; the data request does -# not ask for one. Tolerating it means a group that followed the README as -# published is not failed for a year that is still in dispute -- see issue #12. -TOLERATED_SNAPSHOT_YEARS = frozenset({2000}) - def _required_snapshot_years(exp: dict, run_years: list[int]) -> set[int]: """The snapshot nominal years an x,y,z,t file must carry. @@ -568,6 +765,7 @@ def _run_compliance_checker( ismip_var, mandatory_variables, all_request_variables, + all_mandatory_variables, experiments, criteria_file, ): @@ -590,29 +788,18 @@ def _run_compliance_checker( return _empty_summary() summary = _process_experiments( - log_file=f, + reporter=Reporter(f), source_path=source_path, experiment_groups=experiment_groups, mandatory_variables=mandatory_variables, all_request_variables=all_request_variables, + all_mandatory_variables=all_mandatory_variables, experiments=experiments, ismip_var=ismip_var, ismip_meta=ismip_meta, ) - _insert_synthesis( - source_path=source_path, - exp_counter=summary["exp_counter"], - file_counter=summary["file_counter"], - total_errors=summary["total_errors"], - total_file_errors=summary["total_file_errors"], - total_naming_errors=summary["total_naming_errors"], - total_num_errors=summary["total_num_errors"], - total_spatial_errors=summary["total_spatial_errors"], - total_time_errors=summary["total_time_errors"], - total_attr_errors=summary["total_attr_errors"], - report_naming_issues=summary["report_naming_issues"], - ) + _insert_synthesis(source_path=source_path, summary=summary) return summary except TypeError as err: @@ -624,7 +811,15 @@ def _run_compliance_checker( def _empty_summary() -> dict: + """The summary of a run that could not check anything. + + Its error counts are all zero, because nothing was checked -- which on an + error-based exit status would otherwise mean success. `fatal` is what tells + the two apart, and it is set here rather than at the call sites because + every path that reaches this function is a path where the checker gave up. + """ return { + "fatal": True, "exp_counter": 0, "file_counter": 0, "total_errors": 0, @@ -634,6 +829,13 @@ def _empty_summary() -> dict: "total_time_errors": 0, "total_attr_errors": 0, "total_file_errors": 0, + "total_warnings": 0, + "total_naming_warnings": 0, + "total_num_warnings": 0, + "total_spatial_warnings": 0, + "total_time_warnings": 0, + "total_attr_warnings": 0, + "total_file_warnings": 0, "report_naming_issues": [], } @@ -652,34 +854,45 @@ def _group_files_by_experiment(source_path: str) -> dict: def _process_experiments( - log_file, + reporter: Reporter, source_path: str, experiment_groups: dict, mandatory_variables, all_request_variables, + all_mandatory_variables, experiments, ismip_var, ismip_meta, ): - total_naming_errors = 0 - total_num_errors = 0 - total_spatial_errors = 0 - total_time_errors = 0 - total_attr_errors = 0 - total_file_errors = 0 report_naming_issues = [] + # A declaration is about the submission as a whole, so it is read and + # reported once, before the first experiment. + declared = _read_not_modelled(source_path) + not_modelled = ( + set() + if declared is None + else _report_not_modelled( + reporter.category("file", bullet=""), + declared, + all_mandatory_variables, + all_request_variables, + ) + ) + file_counter = 0 exp_counter = 0 for experiment_name, exp_files in experiment_groups.items(): exp_counter += 1 - exp_summary = _process_single_experiment( - log_file=log_file, + exp_reporter = reporter.child() + file_counter += _process_single_experiment( + reporter=exp_reporter, source_path=source_path, experiment_name=experiment_name, exp_files=exp_files, mandatory_variables=mandatory_variables, + not_modelled=not_modelled, all_request_variables=all_request_variables, experiments=experiments, ismip_var=ismip_var, @@ -687,95 +900,222 @@ def _process_experiments( report_naming_issues=report_naming_issues, ) - file_counter += exp_summary["file_counter"] - total_naming_errors += exp_summary["exp_naming_errors"] - total_num_errors += exp_summary["exp_num_errors"] - total_spatial_errors += exp_summary["exp_spatial_errors"] - total_time_errors += exp_summary["exp_time_errors"] - total_attr_errors += exp_summary["exp_attr_errors"] - total_file_errors += exp_summary["exp_file_errors"] - _print_experiment_summary( - experiment_name=exp_summary["experiment_name"], - exp_errors=exp_summary["exp_errors"], + experiment_name=experiment_name, + exp_errors=exp_reporter.total_errors, + exp_warnings=exp_reporter.total_warnings, ) - total_errors = ( - total_naming_errors - + total_num_errors - + total_spatial_errors - + total_time_errors - + total_attr_errors - + total_file_errors + _print_total_summary( + source_path=source_path, + total_errors=reporter.total_errors, + total_warnings=reporter.total_warnings, ) - _print_total_summary(source_path=source_path, total_errors=total_errors) return { + "fatal": False, "exp_counter": exp_counter, "file_counter": file_counter, - "total_errors": total_errors, - "total_naming_errors": total_naming_errors, - "total_num_errors": total_num_errors, - "total_spatial_errors": total_spatial_errors, - "total_time_errors": total_time_errors, - "total_attr_errors": total_attr_errors, - "total_file_errors": total_file_errors, + "total_errors": reporter.total_errors, + "total_naming_errors": reporter.error_count("naming"), + "total_num_errors": reporter.error_count("num"), + "total_spatial_errors": reporter.error_count("spatial"), + "total_time_errors": reporter.error_count("time"), + "total_attr_errors": reporter.error_count("attr"), + "total_file_errors": reporter.error_count("file"), + "total_warnings": reporter.total_warnings, + "total_naming_warnings": reporter.warning_count("naming"), + "total_num_warnings": reporter.warning_count("num"), + "total_spatial_warnings": reporter.warning_count("spatial"), + "total_time_warnings": reporter.warning_count("time"), + "total_attr_warnings": reporter.warning_count("attr"), + "total_file_warnings": reporter.warning_count("file"), "report_naming_issues": report_naming_issues, } +NOT_MODELLED_FILENAME = "not_modelled.txt" + + +def _read_not_modelled(source_path: str) -> list[str] | None: + """The variable names declared in not_modelled.txt, or None if there is none. + + Groups already submit a README explaining what they have submitted and why + some variables are absent, so the checker has no need to ask for that list + again -- and an explanatory list does not belong on a command line, where it + would be long and would have to be retyped for every run. What is worth + offering is a way to make the 'not submitted' warning go quiet once. + + One name per line; blank lines are ignored and '#' starts a comment, so a + group can say alongside each name why the variable is absent. An absent + file changes nothing. + """ + path = os.path.join(source_path, NOT_MODELLED_FILENAME) + if not os.path.isfile(path): + return None + + declared = [] + with open(path, "r") as f: + for line in f: + name = line.split("#", 1)[0].strip() + if name: + declared.append(name) + return declared + + +def _report_not_modelled( + reporter: Reporter, + declared: list[str], + all_mandatory_variables, + all_request_variables, +) -> set[str]: + """Echo the declaration, fault what it should not contain, return what it silences. + + Two rules keep the file from becoming a way to hide problems. A mandatory + variable named in it is still a missing-mandatory error, and the claim + itself is reported as a further error rather than honoured: the list is a + statement about optional variables, and a group cannot opt out of the data + request with it. A name that is not in the data request at all is an error + too -- it is either a typo or a misunderstanding, and both are better said + plainly than left to be inferred from a warning that did not go away, which + would protect nothing while looking like protection. + + What was declared is echoed either way, so the archived record of a run + shows what was claimed rather than merely that a warning did not appear. + """ + reporter.write( + f"Declared not modelled ({NOT_MODELLED_FILENAME}): {declared}\n" + ) + + suppressed = set() + for name in declared: + if name in all_mandatory_variables: + reporter.error( + f"{NOT_MODELLED_FILENAME} lists '{name}', which the data request" + f" makes mandatory. A submission cannot opt out of a mandatory" + f" variable, so the declaration is not honoured and the missing" + f" files are still reported." + ) + elif name not in all_request_variables: + message = ( + f"{NOT_MODELLED_FILENAME} lists '{name}', which is not a" + f" variable of the data request {VARIABLE_REQUEST_CSV}." + ) + # Sorted, so that two equally close names do not resolve by set + # iteration order and give one log on one run and another on the next. + near_misses = difflib.get_close_matches( + name, sorted(all_request_variables), n=1 + ) + if near_misses: + message += f" The closest requested name is '{near_misses[0]}'." + reporter.error(message) + else: + suppressed.add(name) + + reporter.write(" \n") + return suppressed + + +def _report_variables_not_submitted( + reporter: Reporter, experiment_name: str, not_submitted: list[str] +) -> None: + """Name the non-mandatory variables an experiment carries no files for. + + Nothing reported this before, so a group that meant to submit litemp and + lost it in a script got no signal whatsoever. Everything about how it is + said is chosen so that it cannot be read as an accusation: not every model + supports every non-mandatory variable -- GIA is the obvious case -- and a + deliberate omission is the common case, not the exception. + + Hence one line per experiment naming all of them, rather than one warning + per variable, which would make a model with a narrow scope look far worse + than one with a single dropped file; hence the wording, which says 'not + submitted' rather than 'missing'; and hence its absence from the trailing + naming-issues report, which is the part of the log that reads as a list of + faults. + + There is no suppression when an experiment submitted nothing optional at + all. For a full submission that is precisely the case worth naming, and + treating it as self-evidently deliberate would silence the warning for the + group most likely to have lost something. + """ + if not not_submitted: + return + reporter.warning( + "experiment " + + experiment_name + + " carries no files for the non-mandatory variable(s): " + + str(not_submitted) + + ". This is expected if your model does not represent them; it is" + + " listed only so that a variable lost from a submission does not pass" + + " unnoticed." + ) + + def _process_single_experiment( - log_file, + reporter: Reporter, source_path: str, experiment_name: str, exp_files: list, mandatory_variables, + not_modelled, all_request_variables, experiments, ismip_var, ismip_meta, report_naming_issues, -): - exp_naming_errors = 0 - exp_num_errors = 0 - exp_spatial_errors = 0 - exp_time_errors = 0 - exp_attr_errors = 0 - exp_file_errors = 0 - - temp_mandatory_var = list(mandatory_variables) - for i in exp_files: - variable = i.split("_")[ISMIP7_FILENAME_VAR_IDX] - if variable in temp_mandatory_var: - temp_mandatory_var.remove(variable) +) -> int: + """Check one experiment's files; return how many of them there were.""" + # Experiment-level findings are about the submission rather than about any + # one file, so they are written flush left rather than as list items under + # a file's heading. + presence_reporter = reporter.category("file", bullet="") + naming_reporter = reporter.category("naming", bullet="") + + submitted = {i.split("_")[ISMIP7_FILENAME_VAR_IDX] for i in exp_files} + temp_mandatory_var = [v for v in mandatory_variables if v not in submitted] + # Scoped to the selected --variable-list, exactly as the mandatory check is: + # a run over ismip7_scalars says nothing about the x,y,t variables it was + # never asked to look at. + # A declaration in not_modelled.txt silences the warning below, and nothing + # else: it cannot reach the mandatory-variable check above it. + not_submitted = [ + v + for v in ismip_var + if v not in mandatory_variables + and v not in submitted + and v not in not_modelled + ] file_counter = 0 if experiment_name in [dic["experiment"] for dic in experiments]: - log_file.write("\n ") - log_file.write("**********************************************************\n") - log_file.write(" ** Experiment: " + experiment_name + " \n ") - log_file.write("**********************************************************\n") - log_file.write("\n ") + reporter.write("\n ") + reporter.write("**********************************************************\n") + reporter.write(" ** Experiment: " + experiment_name + " \n ") + reporter.write("**********************************************************\n") + reporter.write("\n ") if not temp_mandatory_var: - log_file.write( + reporter.write( "Mandatory variables Test: " + experiment_name + " : all mandatory variables exist. \n" ) else: - log_file.write( - "ERROR: In experiment " + presence_reporter.error( + "In experiment " + experiment_name + ", these mandatory variable(s) is (are) missing: " - + str(temp_mandatory_var) - + "\n" + + str(temp_mandatory_var), + count=len(temp_mandatory_var), ) - exp_file_errors += len(temp_mandatory_var) + _report_variables_not_submitted( + presence_reporter, experiment_name, not_submitted + ) for file in tqdm(exp_files): file_counter += 1 - file_summary = _process_single_file( - log_file=log_file, + _process_single_file( + reporter=reporter, source_path=source_path, file=file, experiment_name=experiment_name, @@ -786,55 +1126,30 @@ def _process_single_experiment( report_naming_issues=report_naming_issues, ) - exp_naming_errors += file_summary["var_naming_errors"] - exp_num_errors += file_summary["var_num_errors"] - exp_spatial_errors += file_summary["var_spatial_errors"] - exp_time_errors += file_summary["var_time_errors"] - exp_attr_errors += file_summary["var_attr_errors"] - else: - log_file.write("\n ") - log_file.write("**********************************************************\n") - log_file.write(" ** Experiment: " + experiment_name + " \n ") - log_file.write("**********************************************************\n") - log_file.write("\n ") - log_file.write( - "ERROR: The compliance check is ignored for experiment " + reporter.write("\n ") + reporter.write("**********************************************************\n") + reporter.write(" ** Experiment: " + experiment_name + " \n ") + reporter.write("**********************************************************\n") + reporter.write("\n ") + naming_reporter.error( + "The compliance check is ignored for experiment " + experiment_name + " as it is not in " + str([exp["experiment"] for exp in experiments]) - + ". \n" + + ". " ) - exp_naming_errors += 1 report_naming_issues.append( "Compliance check ignored : experiment " + experiment_name + " not in the experiments list." ) - exp_errors = ( - exp_time_errors - + exp_spatial_errors - + exp_num_errors - + exp_naming_errors - + exp_attr_errors - + exp_file_errors - ) - return { - "file_counter": file_counter, - "experiment_name": experiment_name, - "exp_errors": exp_errors, - "exp_naming_errors": exp_naming_errors, - "exp_num_errors": exp_num_errors, - "exp_spatial_errors": exp_spatial_errors, - "exp_time_errors": exp_time_errors, - "exp_attr_errors": exp_attr_errors, - "exp_file_errors": exp_file_errors, - } + return file_counter def _process_single_file( - log_file, + reporter: Reporter, source_path: str, file: str, experiment_name: str, @@ -843,12 +1158,11 @@ def _process_single_file( all_request_variables, experiments, report_naming_issues, -): - var_naming_errors = 0 - var_num_errors = 0 - var_spatial_errors = 0 - var_time_errors = 0 - var_attr_errors = 0 +) -> None: + # A sub-total of this file's findings, for the footer below; it still rolls + # up into the experiment and the run. + file_reporter = reporter.child() + naming_reporter = file_reporter.category("naming") file_name = os.path.basename(file) file_name_split = file_name.split("_") @@ -860,49 +1174,35 @@ def _process_single_file( ds = xr.open_dataset(os.path.join(source_path, file), decode_times=False) except (ValueError, TypeError) as e: - log_file.write(" - ERROR: Cannot open " + file_name + ": " + str(e) + "\n") - var_naming_errors += 1 - return { - "var_naming_errors": var_naming_errors, - "var_num_errors": var_num_errors, - "var_spatial_errors": var_spatial_errors, - "var_time_errors": var_time_errors, - "var_attr_errors": var_attr_errors, - } + naming_reporter.error("Cannot open " + file_name + ": " + str(e)) + return file_variables = list(ds.data_vars) if len(file_name_split) != ISMIP7_FILENAME_PARTS: - log_file.write( - " - ERROR: the file name " + naming_reporter.error( + "the file name " + file_name + " does not follow the naming convention (expected " + str(ISMIP7_FILENAME_PARTS) - + " underscore-separated fields).\n" + + " underscore-separated fields)." ) report_naming_issues.append( "Compliance check ignored: file " + file_name + " does not follow the naming convention." ) - var_naming_errors += 1 - return { - "var_naming_errors": var_naming_errors, - "var_num_errors": var_num_errors, - "var_spatial_errors": var_spatial_errors, - "var_time_errors": var_time_errors, - "var_attr_errors": var_attr_errors, - } + return experiment_varname = file_name_split[ISMIP7_FILENAME_EXPERIMENT_IDX] if experiment_varname != experiment_name: - log_file.write( - " - ERROR: in the file name " + naming_reporter.error( + "in the file name " + file_name + ", the experiment name (" + experiment_varname + ") does not match the expected experiment: " + experiment_name - + ".\n" + + "." ) report_naming_issues.append( "Compliance check ignored: in the file name " @@ -913,71 +1213,63 @@ def _process_single_file( + experiment_name + ".\n" ) - var_naming_errors += 1 - return { - "var_naming_errors": var_naming_errors, - "var_num_errors": var_num_errors, - "var_spatial_errors": var_spatial_errors, - "var_time_errors": var_time_errors, - "var_attr_errors": var_attr_errors, - } + return if considered_variable not in all_request_variables: - log_file.write(" \n") - log_file.write("Experiment: " + experiment_name + " - File: " + file_name + "\n") - log_file.write(" \n") - log_file.write("NAMING Tests \n") + file_reporter.write(" \n") + file_reporter.write( + "Experiment: " + experiment_name + " - File: " + file_name + "\n" + ) + file_reporter.write(" \n") + file_reporter.write("NAMING Tests \n") message = ( f"'{considered_variable}' (field {ISMIP7_FILENAME_VAR_IDX} of the" f" file name) is not a variable in the data request" f" {VARIABLE_REQUEST_CSV}." ) near_misses = difflib.get_close_matches( - considered_variable, all_request_variables, n=1 + considered_variable, sorted(all_request_variables), n=1 ) if near_misses: message += f" The closest requested name is '{near_misses[0]}'." - log_file.write(f" - ERROR: {message}\n") + naming_reporter.error(message) report_naming_issues.append(f"Compliance check ignored: {message}") - var_naming_errors += 1 elif considered_variable in ismip_var: - var_naming_errors, var_num_errors, var_spatial_errors, var_time_errors, var_attr_errors = ( - _run_variable_checks( - log_file=log_file, - ds=ds, - file_name=file_name, - considered_variable=considered_variable, - experiment_name=experiment_name, - file_variables=file_variables, - region=region, - ismip_var=ismip_var, - ismip_meta=ismip_meta, - experiments=experiments, - report_naming_issues=report_naming_issues, - ) + _run_variable_checks( + reporter=file_reporter, + ds=ds, + file_name=file_name, + considered_variable=considered_variable, + experiment_name=experiment_name, + file_variables=file_variables, + region=region, + ismip_var=ismip_var, + ismip_meta=ismip_meta, + experiments=experiments, + report_naming_issues=report_naming_issues, ) - var_errors = var_naming_errors + var_num_errors + var_spatial_errors + var_time_errors + var_attr_errors + var_errors = file_reporter.total_errors + var_warnings = file_reporter.total_warnings - log_file.write("\n") - log_file.write("----------------------------------------------------------\n") - log_file.write( + file_reporter.write("\n") + file_reporter.write("----------------------------------------------------------\n") + file_reporter.write( experiment_name + " - " + considered_variable + " - File:" + file_name + "\n" ) if var_errors > 0: - log_file.write(str(var_errors) + " error(s). Please review before sharing.\n") + file_reporter.write( + str(var_errors) + " error(s). Please review before sharing.\n" + ) else: - log_file.write("No errors. Good job !\n") - log_file.write("No warnings.\n") - log_file.write("----------------------------------------------------------\n") - - return { - "var_naming_errors": var_naming_errors, - "var_num_errors": var_num_errors, - "var_spatial_errors": var_spatial_errors, - "var_time_errors": var_time_errors, - "var_attr_errors": var_attr_errors, - } + # Warnings never change the verdict: a file with warnings and no errors + # is compliant, and is told so. + file_reporter.write("No errors. Good job !\n") + if var_warnings > 0: + file_reporter.write(str(var_warnings) + " warning(s). Please review.\n") + else: + file_reporter.write("No warnings.\n") + file_reporter.write("----------------------------------------------------------\n") class NamingResult(NamedTuple): @@ -993,29 +1285,27 @@ class NamingResult(NamedTuple): the later checks with nothing to read clear it. """ - errors: int filename_years: tuple[int, int] | None can_continue: bool = True def _check_naming( - log_file, + reporter: Reporter, file_name: str, region: str, dim: set, isscalar: bool, report_naming_issues: list, ) -> NamingResult: - errors = 0 filename_years = None - log_file.write("NAMING Tests \n") + reporter.write("NAMING Tests \n") if not isscalar and not {"x", "y"}.issubset(dim): - log_file.write( - " - ERROR: Compliance check ignored: x or y in the mandatory dimensions (x,y,t) is missing.\n" + reporter.error( + "Compliance check ignored: x or y in the mandatory dimensions (x,y,t) is missing." ) - log_file.write( + reporter.write( " Only " + str(list(dim)) + " has been detected.\n" ) report_naming_issues.append( @@ -1025,80 +1315,73 @@ def _check_naming( # Without x and y there is no grid to check and no spatial variable to # read: this is one of the few naming problems that really does stop # everything else. - return NamingResult(errors + 1, filename_years, can_continue=False) + return NamingResult(filename_years, can_continue=False) if region not in ["AIS", "GrIS"]: - log_file.write( - " - ERROR: Region " + reporter.error( + "Region " + region + " not recognized. It should be AIS or GrIS. The checks that depend" + " on the region (value range, grid extent and resolution, crs) are" - + " skipped for this file; the rest still run.\n" + + " skipped for this file; the rest still run." ) report_naming_issues.append( "Region-dependent checks skipped: region (AIS/GrIS) not identified in the file " + file_name + " due to wrong naming." ) - errors += 1 parts = file_name.split("_") if len(parts) == ISMIP7_FILENAME_PARTS: ism_member = parts[ISMIP7_FILENAME_ISM_MEMBER_IDX] if not re.fullmatch(r"m\d{3}", ism_member): - log_file.write( - f" - ERROR: ISM member id '{ism_member}' (field {ISMIP7_FILENAME_ISM_MEMBER_IDX}) does not match expected format mNNN (e.g. m001).\n" + reporter.error( + f"ISM member id '{ism_member}' (field {ISMIP7_FILENAME_ISM_MEMBER_IDX}) does not match expected format mNNN (e.g. m001)." ) - errors += 1 esm_name = parts[ISMIP7_FILENAME_ESM_IDX] if esm_name not in VALID_ESM_NAMES: - log_file.write( - f" - ERROR: ESM name '{esm_name}' (field {ISMIP7_FILENAME_ESM_IDX}) is not a recognised CMIP6/CMIP7 model name.\n" + reporter.error( + f"ESM name '{esm_name}' (field {ISMIP7_FILENAME_ESM_IDX}) is not a recognised CMIP6/CMIP7 model name." ) - errors += 1 forcing_member = parts[ISMIP7_FILENAME_FORCING_MEMBER_IDX] if not re.fullmatch(r"f\d{3}", forcing_member): - log_file.write( - f" - ERROR: forcing member id '{forcing_member}' (field {ISMIP7_FILENAME_FORCING_MEMBER_IDX}) does not match expected format fNNN (e.g. f001).\n" + reporter.error( + f"forcing member id '{forcing_member}' (field {ISMIP7_FILENAME_FORCING_MEMBER_IDX}) does not match expected format fNNN (e.g. f001)." ) - errors += 1 set_counter = parts[ISMIP7_FILENAME_SET_COUNTER_IDX] if not re.fullmatch(r"[CEP]\d{3}", set_counter): - log_file.write( - f" - ERROR: set counter '{set_counter}' (field {ISMIP7_FILENAME_SET_COUNTER_IDX}) does not match expected format [C|E|P]NNN (e.g. C001, E041, P132).\n" + reporter.error( + f"set counter '{set_counter}' (field {ISMIP7_FILENAME_SET_COUNTER_IDX}) does not match expected format [C|E|P]NNN (e.g. C001, E041, P132)." ) - errors += 1 is_static = not ({"time", "t"} & dim) year_range_field = parts[ISMIP7_FILENAME_YEAR_RANGE_IDX].removesuffix(".nc") if is_static: - log_file.write(f" - Filename year range: N/A (static spatial variable)\n") + reporter.note("Filename year range: N/A (static spatial variable)") elif not (year_range_match := re.fullmatch(r"(\d{4})-(\d{4})", year_range_field)): - log_file.write( - f" - ERROR: year range '{year_range_field}' (field {ISMIP7_FILENAME_YEAR_RANGE_IDX}) does not match expected format YYYY-YYYY (e.g. 2015-2300).\n" + reporter.error( + f"year range '{year_range_field}' (field {ISMIP7_FILENAME_YEAR_RANGE_IDX}) does not match expected format YYYY-YYYY (e.g. 2015-2300)." ) - errors += 1 else: fn_start_year = int(year_range_match.group(1)) fn_end_year = int(year_range_match.group(2)) if fn_start_year > fn_end_year: - log_file.write( - f" - ERROR: year range '{year_range_field}': start year {fn_start_year} is after end year {fn_end_year}.\n" + reporter.error( + f"year range '{year_range_field}': start year {fn_start_year} is after end year {fn_end_year}." ) - errors += 1 else: # What the range means -- whether the experiment allows it, and # whether the time axis delivers it -- is for _check_time, which # decodes the file anyway. filename_years = (fn_start_year, fn_end_year) - log_file.write( - f" - Filename year range {fn_start_year}-{fn_end_year} is well formed: OK\n" + reporter.ok( + f"Filename year range {fn_start_year}-{fn_end_year} is well formed: OK" ) - return NamingResult(errors, filename_years) + return NamingResult(filename_years) # The data request writes a variable's dimensions innermost-first and calls the @@ -1114,20 +1397,20 @@ def _check_naming( def _check_variable_dimensions( - log_file, + reporter: Reporter, ds, considered_variable: str, requested_dim: str, -) -> int: +) -> None: """Check a variable's dimensions against the 'Dim' column of the request.""" expected = REQUESTED_DIMENSIONS.get(requested_dim) if expected is None: - log_file.write( - f" - Variable '{considered_variable}' dimensions: not checked (the" + reporter.note( + f"Variable '{considered_variable}' dimensions: not checked (the" f" data request gives Dim '{requested_dim}', which this checker does" - f" not know).\n" + f" not know)." ) - return 0 + return # 't' is an accepted spelling of the time dimension throughout the checker. actual = tuple( @@ -1135,26 +1418,25 @@ def _check_variable_dimensions( ) if set(actual) != set(expected): - log_file.write( - f" - ERROR: variable '{considered_variable}' has dimensions" + reporter.error( + f"variable '{considered_variable}' has dimensions" f" {actual}; the data request asks for {requested_dim}, that is" - f" {expected}.\n" + f" {expected}." ) - return 1 + return if actual != expected: - log_file.write( - f" - ERROR: variable '{considered_variable}' has dimensions" + reporter.error( + f"variable '{considered_variable}' has dimensions" f" {actual}; the data request asks for {requested_dim} in the" - f" conventional order {expected}.\n" + f" conventional order {expected}." ) - return 1 + return - log_file.write( - f" - Variable '{considered_variable}' dimensions ({', '.join(actual)})" - f" match the requested {requested_dim}: OK\n" + reporter.ok( + f"Variable '{considered_variable}' dimensions ({', '.join(actual)})" + f" match the requested {requested_dim}: OK" ) - return 0 def _allowed_file_variables(ds, considered_variable: str) -> set[str]: @@ -1163,8 +1445,11 @@ def _allowed_file_variables(ds, considered_variable: str) -> set[str]: A file carries one variable of the data request, but CF lets that variable bring companions: the bounds of a coordinate (which is how 'time_bounds' reaches every FL file), the container variable a 'grid_mapping' points at, - and any auxiliary coordinates the variable names. Those belong in the - file; anything else is something the data request did not ask for. + the auxiliary coordinates a variable names, the cell measures it is + normalised by, and the ancillary variables that qualify it. Every one of + those is a variable the file has to carry for the requested variable to + mean what it says, so none of them is an extra; anything else is something + the data request did not ask for. """ allowed = {considered_variable} | set(ds.coords) @@ -1180,29 +1465,35 @@ def _allowed_file_variables(ds, considered_variable: str) -> set[str]: if grid_mapping: allowed.add(str(grid_mapping)) allowed.update(str(attributes.get("coordinates", "")).split()) + allowed.update(str(attributes.get("ancillary_variables", "")).split()) + # cell_measures names each variable after the measure it supplies + # ('area: areacello volume: volcello'), so the keywords are dropped. + allowed.update( + token + for token in str(attributes.get("cell_measures", "")).split() + if not token.endswith(":") + ) return allowed def _check_file_variables( - log_file, + reporter: Reporter, ds, file_name: str, considered_variable: str, file_variables, requested_dim: str, report_naming_issues: list, -) -> tuple[int, bool]: +) -> bool: """Check the variables a file contains against the one its name promises. The file name states which variable of the data request a file carries, and every other check reads that variable, so a file that does not contain it - has nothing to check: the second element of the return value is False, and - the caller skips the remaining checks for the file rather than reporting a - clean bill of health on a variable that was never looked at. + has nothing to check: the return value is False, and the caller skips the + remaining checks for the file rather than reporting a clean bill of health + on a variable that was never looked at. """ - errors = 0 - if considered_variable not in file_variables: message = ( f"the file name promises variable '{considered_variable}', but the" @@ -1217,35 +1508,38 @@ def _check_file_variables( f" '{near_misses[0]}' may be a misspelling of" f" '{considered_variable}'." ) - log_file.write(f" - ERROR: {message}\n") + reporter.error(message) report_naming_issues.append( f"Compliance check ignored: in the file {file_name}, {message}" ) - return errors + 1, False + return False - log_file.write( - f" - Variable '{considered_variable}' from the file name is present in" - f" the file: OK\n" + reporter.ok( + f"Variable '{considered_variable}' from the file name is present in" + f" the file: OK" ) allowed = _allowed_file_variables(ds, considered_variable) unexpected = sorted(name for name in file_variables if name not in allowed) for name in unexpected: - log_file.write( - f" - ERROR: unexpected variable '{name}' in the file. A file holds" - f" one variable of the data request -- here '{considered_variable}'" - f" -- along with its coordinates and any bounds or grid-mapping" - f" variables, and nothing else.\n" + # A warning: the requested variable is present and fully checkable, and + # a reader taking it out of the file is unaffected by what sits beside + # it. What the extra says is that the file was probably not written for + # this submission, which is worth a look and is not a fault. + reporter.warning( + f"unexpected variable '{name}' in the file. A file is expected to" + f" hold one variable of the data request -- here" + f" '{considered_variable}' -- along with its coordinates and the" + f" companion variables CF lets it name (bounds, grid mapping, cell" + f" measures, ancillary variables). '{considered_variable}' is" + f" checked as normal." ) - errors += 1 if not unexpected: - log_file.write(" - No unexpected variables in the file: OK\n") + reporter.ok("No unexpected variables in the file: OK") - errors += _check_variable_dimensions( - log_file, ds, considered_variable, requested_dim - ) + _check_variable_dimensions(reporter, ds, considered_variable, requested_dim) - return errors, True + return True # CF requires only that the units attribute be "a string that can be recognized @@ -1340,98 +1634,92 @@ def _units_match(actual: str, expected: str) -> bool: def _check_numerical( - log_file, + reporter: Reporter, ds, ivar: str, ismip_meta: list, var_index: int, region: str, isscalar: bool, -) -> int: - errors = 0 - - log_file.write("NUMERICAL Tests \n") +) -> None: + reporter.write("NUMERICAL Tests \n") var_units = ds[ivar].attrs.get("units") expected_units = ismip_meta[var_index]["units"] if var_units is None: - log_file.write( - f" - ERROR: The variable '{ivar}' has no 'units' attribute. The data" - f" request asks for '{expected_units}'.\n" + reporter.error( + f"The variable '{ivar}' has no 'units' attribute. The data" + f" request asks for '{expected_units}'." ) - errors += 1 elif var_units == expected_units: - log_file.write(" - The unit is correct: " + var_units + "\n") + reporter.ok("The unit is correct: " + var_units) elif _units_match(var_units, expected_units): - log_file.write( - " - The unit is correct: " + reporter.ok( + "The unit is correct: " + var_units + " (equivalent to the requested " + expected_units - + ")\n" + + ")" ) else: - log_file.write( - " - ERROR: The unit of the variable is " + reporter.error( + "The unit of the variable is " + var_units + " and should be " + expected_units - + " \n" + + " " ) - errors += 1 if not isscalar and region not in ("AIS", "GrIS"): - log_file.write( - " - Value range: not checked (the allowed range depends on the" - " region, which the file name does not identify).\n" + reporter.note( + "Value range: not checked (the allowed range depends on the" + " region, which the file name does not identify)." ) elif not isscalar: + # The severity of an out-of-range value is per variable, from the + # range_severity column of the data request; see _range_severity. + report_range = ( + reporter.warning + if ismip_meta[var_index].get("range_severity") == "warning" + else reporter.error + ) if False in ds[ivar].isnull(): if ( ds[ivar].min(skipna=True).item() >= ismip_meta[var_index]["min_value_" + region.lower()] ): - log_file.write(" - The minimum value successfully verified.\n") + reporter.ok("The minimum value successfully verified.") else: - log_file.write( - " - ERROR: The minimum value (" + report_range( + "The minimum value (" + str(ds[ivar].min(skipna=True).values.item(0)) + ") is out of range. Min value accepted: " + str(ismip_meta[var_index]["min_value_" + region.lower()]) - + "\n" ) - errors += 1 if ( ds[ivar].max(skipna=True).item() <= ismip_meta[var_index]["max_value_" + region.lower()] ): - log_file.write(" - The maximum value successfully verified.\n") + reporter.ok("The maximum value successfully verified.") else: - log_file.write( - " - ERROR: The maximum value (" + report_range( + "The maximum value (" + str(ds[ivar].max(skipna=True).values.item(0)) + ") is out of range. Max value accepted: " + str(ismip_meta[var_index]["max_value_" + region.lower()]) - + "\n" ) - errors += 1 else: - log_file.write(" - ERROR: The array only contains missing values.\n") - errors += 1 - - return errors + reporter.error("The array only contains missing values.") def _check_spatial( - log_file, + reporter: Reporter, ds, grid_extent: list, possible_resolution: list, -) -> int: - errors = 0 - - log_file.write("SPATIAL Tests \n") +) -> None: + reporter.write("SPATIAL Tests \n") coords = ds.coords.to_dataset() Xbottomleft = int(min(coords["x"]).values.item()) Ybottomleft = int(min(coords["y"]).values.item()) @@ -1439,54 +1727,49 @@ def _check_spatial( Ytopright = int(max(coords["y"]).values.item()) if Xbottomleft == grid_extent[0] and Ybottomleft == grid_extent[1]: - log_file.write(" - Grid: Lowest left corner is well defined.\n") + reporter.ok("Grid: Lowest left corner is well defined.") else: - log_file.write( - " - ERROR: Lowest left corner of the grid [" + reporter.error( + "Lowest left corner of the grid [" + str(Xbottomleft) + "," + str(Ybottomleft) + "] is not correctly defined. [" + str(grid_extent[0]) + "," + str(grid_extent[1]) - + "] Expected\n" + + "] Expected" ) - errors += 1 if Xtopright == grid_extent[2] and Ytopright == grid_extent[3]: - log_file.write(" - Grid: Upper right corner is well defined.\n") + reporter.ok("Grid: Upper right corner is well defined.") else: - log_file.write( - " - ERROR: Upper right corner of the grid [" + reporter.error( + "Upper right corner of the grid [" + str(Xtopright) + "," + str(Ytopright) + "] is not correctly defined. [" + str(grid_extent[2]) + "," + str(grid_extent[3]) - + "] Expected\n" + + "] Expected" ) - errors += 1 Xresolution = round((coords["x"][1].values - coords["x"][0].values) / 1000, 0) Yresolution = round((coords["y"][1].values - coords["y"][0].values) / 1000, 0) if Xresolution in set(possible_resolution) and Yresolution in set(possible_resolution): - log_file.write( - " - The grid resolution (" + reporter.ok( + "The grid resolution (" + str(int(Xresolution)) - + " km) was successfully verified.\n" + + " km) was successfully verified." ) else: - log_file.write( - " - ERROR: resolution x=" + reporter.error( + "resolution x=" + str(Xresolution) + " km, y=" + str(Yresolution) + " km is not an authorized grid resolution. Allowed: " + str(possible_resolution) - + " km\n" + + " km" ) - errors += 1 - - return errors def _check_time( - log_file, + reporter: Reporter, ds, dim: set, experiments: list, @@ -1494,7 +1777,7 @@ def _check_time( var_type: str = "", requested_dim: str = "", filename_years=None, -) -> int: +) -> None: """Check a file's time axis against the one its experiment calls for. The axis is checked by reconstructing the axis the file should have had and @@ -1503,51 +1786,47 @@ def _check_time( 2015, 2016, 2299, 2300 spans the right years with a 365-day first interval and is 282 time steps short of what was asked for. """ - errors = 0 - - log_file.write("TIME Tests \n") + reporter.write("TIME Tests \n") if not ({"t"}.issubset(dim) or {"time"}.issubset(dim)): if {"x", "y"}.issubset(dim): # Static spatial variable (x,y) — no time axis is expected. - log_file.write(" - Time axis: N/A (static spatial variable)\n") - return errors - log_file.write( - " - ERROR: The time dimension is missing. Time Tests have been ignored.\n" + reporter.note("Time axis: N/A (static spatial variable)") + return + reporter.error( + "The time dimension is missing. Time Tests have been ignored." ) - return errors + 1 + return time_dim = "time" if "time" in ds.dims else "t" unlimited_dims = ds.encoding.get("unlimited_dims", set()) if time_dim in unlimited_dims: - log_file.write(" - Time is a record (unlimited) dimension: OK\n") + reporter.ok("Time is a record (unlimited) dimension: OK") else: - log_file.write( - f" - ERROR: dimension '{time_dim}' is not a record (unlimited) dimension.\n" + reporter.error( + f"dimension '{time_dim}' is not a record (unlimited) dimension." ) - errors += 1 try: ds = xr.decode_cf(ds, decode_times=xr.coders.CFDatetimeCoder(use_cftime=True)) except Exception: - log_file.write( - " - ERROR: The time coordinate could not be decoded. Time checks cannot proceed.\n" + reporter.error( + "The time coordinate could not be decoded. Time checks cannot proceed." ) # we can't proceed because the next steps will crash - return errors + 1 + return if not _strictly_increasing(ds.coords["time"]): - log_file.write( - " - ERROR: the time series is not monotonically increasing. Time segments may have been concatenated in the wrong order.\n" + reporter.error( + "the time series is not monotonically increasing. Time segments may have been concatenated in the wrong order." ) - return errors + 1 + return index_exp = [dic["experiment"] for dic in experiments].index(experiment_name) exp = experiments[index_exp] actual = list(ds["time"].values) for message in _check_filename_year_range(exp, filename_years): - log_file.write(f" - ERROR: {message}\n") - errors += 1 + reporter.error(message) # The nominal years the run as a whole covers. The annual variables carry # one time step for each of them; the snapshot variables carry a few. @@ -1555,33 +1834,29 @@ def _check_time( exp, _axis_start_year(exp, filename_years, actual, var_type) ) if not run_years: - log_file.write( - f" - ERROR: the time axis starts in nominal year" + reporter.error( + f"the time axis starts in nominal year" f" {_timestamp_to_nominal_year(actual[0], var_type)}, after experiment" f" '{experiment_name}' ends in {exp['end_year']}. The expected time" - f" axis cannot be determined.\n" + f" axis cannot be determined." ) - return errors + 1 + return if requested_dim == "x,y,z,t": - return errors + _check_snapshot_time_axis( - log_file, actual, exp, var_type, run_years - ) + _check_snapshot_time_axis(reporter, actual, exp, var_type, run_years) + return messages = _compare_time_axis(actual, run_years, var_type) for message in messages: - log_file.write(f" - ERROR: {message}\n") - errors += len(messages) + reporter.error(message) if not messages: - log_file.write( - f" - Time axis: {len(actual)} annual {var_type} time step(s) covering" + reporter.ok( + f"Time axis: {len(actual)} annual {var_type} time step(s) covering" f" nominal years {run_years[0]}-{run_years[-1]}, as" - f" experiment '{experiment_name}' requires: OK\n" + f" experiment '{experiment_name}' requires: OK" ) - return errors - def _axis_start_year(exp: dict, filename_years, actual: list, var_type: str) -> int: """The nominal year the expected time axis should begin at. @@ -1604,8 +1879,8 @@ def _axis_start_year(exp: dict, filename_years, actual: list, var_type: str) -> def _check_snapshot_time_axis( - log_file, actual: list, exp: dict, var_type: str, run_years: list[int] -) -> int: + reporter: Reporter, actual: list, exp: dict, var_type: str, run_years: list[int] +) -> None: """Check the sparse snapshot axis of an x,y,z,t variable (e.g. litemp). Unlike the annual variables, these carry a handful of snapshots rather than @@ -1617,54 +1892,59 @@ def _check_snapshot_time_axis( single snapshot at an arbitrary year passed. """ required = _required_snapshot_years(exp, run_years) - permitted = required | { - y for y in TOLERATED_SNAPSHOT_YEARS if run_years[0] <= y <= run_years[-1] - } actual_at = _nominal_year_index(actual, var_type) - errors = 0 + errors_before = reporter.total_errors + warnings_before = reporter.total_warnings missing = sorted(required - set(actual_at)) if missing: - log_file.write( - f" - ERROR: required snapshot nominal year(s) missing:" + reporter.error( + f"required snapshot nominal year(s) missing:" f" {_format_year_runs(missing)}. Experiment '{exp['experiment']}'" f" covering {run_years[0]}-{run_years[-1]} requires snapshots at" - f" {_format_year_runs(sorted(required))}.\n" + f" {_format_year_runs(sorted(required))}." ) - errors += 1 - unexpected = sorted(set(actual_at) - permitted) + # A warning, not an error: the data request specifies snapshots as a + # *minimum* set, so over-delivering 3D temperature is not non-compliance. + # This is also how a snapshot at 2000 is now reported -- the README asked + # for one until recently and the data request never did, so a file carrying + # it is named rather than either failed or silently accepted (issue #12). + # The asymmetry with the annual axis, where an extra year is an error, is + # deliberate: that axis is pinned end to end by experiments_ismip7.csv, so + # an extra year there means the file does not match the experiment it names. + unexpected = sorted(set(actual_at) - required) if unexpected: - log_file.write( - f" - ERROR: snapshot nominal year(s) the experiment does not call" + reporter.warning( + f"snapshot nominal year(s) the experiment does not call" f" for: {_format_year_runs(unexpected)}. Required:" - f" {_format_year_runs(sorted(required))}.\n" + f" {_format_year_runs(sorted(required))}." ) - errors += 1 mismatch = _timestamp_mismatch_message( - actual_at, sorted(set(actual_at) & permitted), var_type + actual_at, sorted(set(actual_at) & required), var_type ) if mismatch: - log_file.write(f" - ERROR: {mismatch}\n") - errors += 1 + reporter.error(mismatch) - if errors == 0: - log_file.write( - f" - Snapshot time axis: nominal year(s)" + if ( + reporter.total_errors == errors_before + and reporter.total_warnings == warnings_before + ): + reporter.ok( + f"Snapshot time axis: nominal year(s)" f" {_format_year_runs(sorted(actual_at))} cover everything experiment" - f" '{exp['experiment']}' requires: OK\n" + f" '{exp['experiment']}' requires: OK" ) - log_file.write( - " - Annual cadence checks: N/A (snapshot variable — the time axis holds" - " sparse snapshots by design).\n" + reporter.note( + "Annual cadence checks: N/A (snapshot variable — the time axis holds" + " sparse snapshots by design)." ) - return errors def _check_attributes( - log_file, + reporter: Reporter, ds, ivar: str, ismip_meta: list, @@ -1672,40 +1952,34 @@ def _check_attributes( isscalar: bool, var_type: str, region: str, -) -> int: - errors = 0 - - log_file.write("ATTRIBUTE Tests \n") +) -> None: + reporter.write("ATTRIBUTE Tests \n") # Sub-test 1: global attributes required_global = ["group", "model", "contact_name", "contact_email"] - global_errors = 0 + errors_before = reporter.total_errors for attr in required_global: if attr not in ds.attrs: - log_file.write(f" - ERROR (attributes): global attribute '{attr}' is missing.\n") - global_errors += 1 + reporter.error(f"global attribute '{attr}' is missing.") expected_crs = "epsg:3413" if region == "GrIS" else "epsg:3031" actual_crs = ds.attrs.get("crs") if region not in ("AIS", "GrIS"): - log_file.write( - " - Global attribute 'crs': not checked (the expected value depends" - " on the region, which the file name does not identify).\n" + reporter.note( + "Global attribute 'crs': not checked (the expected value depends" + " on the region, which the file name does not identify)." ) elif actual_crs is None: - log_file.write(" - ERROR (attributes): global attribute 'crs' is missing.\n") - global_errors += 1 + reporter.error("global attribute 'crs' is missing.") elif actual_crs.lower() != expected_crs: - log_file.write( - f" - ERROR (attributes): global attribute 'crs' is '{actual_crs}'," - f" expected '{expected_crs}' (case-insensitive) for region {region}.\n" + reporter.error( + f"global attribute 'crs' is '{actual_crs}'," + f" expected '{expected_crs}' (case-insensitive) for region {region}." ) - global_errors += 1 - if global_errors == 0: - log_file.write(" - Global attributes: OK\n") - errors += global_errors + if reporter.total_errors == errors_before: + reporter.ok("Global attributes: OK") # Sub-test 2: coordinate attributes - coord_errors = 0 + errors_before = reporter.total_errors time_coord = None for name in ("time", "t"): if name in ds.coords: @@ -1713,10 +1987,9 @@ def _check_attributes( break is_static_spatial = time_coord is None and {"x", "y"}.issubset(set(ds.coords)) if time_coord is None and not is_static_spatial: - log_file.write(" - ERROR (attributes): coordinate 'time' not found.\n") - coord_errors += 1 + reporter.error("coordinate 'time' not found.") elif time_coord is None and is_static_spatial: - log_file.write(" - Time coordinate: N/A (static spatial variable)\n") + reporter.note("Time coordinate: N/A (static spatial variable)") else: # xarray decodes 'units' and 'calendar' into .encoding; 'bounds' stays in .attrs time_var = ds[time_coord] @@ -1726,61 +1999,50 @@ def _check_attributes( time_attrs_required.append("bounds") for attr in time_attrs_required: if attr not in combined: - log_file.write( - f" - ERROR (attributes): coordinate '{time_coord}' missing attribute '{attr}'.\n" + reporter.error( + f"coordinate '{time_coord}' missing attribute '{attr}'." ) - coord_errors += 1 if "units" in combined and combined["units"] != "days since 1850-01-01": - log_file.write( - f" - ERROR (attributes): time 'units' is '{combined['units']}', expected 'days since 1850-01-01'.\n" + reporter.error( + f"time 'units' is '{combined['units']}', expected 'days since 1850-01-01'." ) - coord_errors += 1 if "calendar" in combined and combined["calendar"] != "standard": - log_file.write( - f" - ERROR (attributes): time 'calendar' is '{combined['calendar']}', expected 'standard'.\n" + reporter.error( + f"time 'calendar' is '{combined['calendar']}', expected 'standard'." ) - coord_errors += 1 if not isscalar: spatial_coords = ("x", "y", "z") if "z" in ds.coords else ("x", "y") for coord in spatial_coords: if coord in ds.coords: if "units" not in ds[coord].attrs: - log_file.write( - f" - ERROR (attributes): coordinate '{coord}' missing attribute 'units'.\n" + reporter.error( + f"coordinate '{coord}' missing attribute 'units'." ) - coord_errors += 1 else: - log_file.write( - f" - ERROR (attributes): coordinate '{coord}' not found.\n" - ) - coord_errors += 1 - if coord_errors == 0: - log_file.write(" - Coordinate attributes: OK\n") - errors += coord_errors + reporter.error(f"coordinate '{coord}' not found.") + if reporter.total_errors == errors_before: + reporter.ok("Coordinate attributes: OK") # Sub-test 3: variable standard_name - var_errors = 0 + errors_before = reporter.total_errors expected_standard_name = ismip_meta[var_index].get("standard_name") if expected_standard_name is not None and ivar in ds: if "standard_name" not in ds[ivar].attrs: - log_file.write( - f" - ERROR (attributes): variable '{ivar}' missing 'standard_name' attribute.\n" + reporter.error( + f"variable '{ivar}' missing 'standard_name' attribute." ) - var_errors += 1 elif ds[ivar].attrs["standard_name"] != expected_standard_name: - log_file.write( - f" - ERROR (attributes): variable '{ivar}' standard_name" + reporter.error( + f"variable '{ivar}' standard_name" f" '{ds[ivar].attrs['standard_name']}'" - f" does not match expected '{expected_standard_name}'.\n" + f" does not match expected '{expected_standard_name}'." ) - var_errors += 1 - if var_errors == 0: - log_file.write(" - Variable attributes: OK\n") - errors += var_errors + if reporter.total_errors == errors_before: + reporter.ok("Variable attributes: OK") # Sub-test 4: _FillValue must equal the default netCDF4 fill value; # if missing_value is also present it must equal _FillValue. - fill_errors = 0 + errors_before = reporter.total_errors if ivar in ds: dtype = ds[ivar].dtype nc4_dtype_key = dtype.kind + str(dtype.itemsize) @@ -1789,65 +2051,63 @@ def _check_attributes( # xarray moves missing_value from attrs to encoding on read (CF fill-value handling) missing_value = ds[ivar].attrs.get("missing_value") or ds[ivar].encoding.get("missing_value") if fill_value is None: - log_file.write(f" - ERROR (attributes): variable '{ivar}' missing '_FillValue'.\n") - fill_errors += 1 + reporter.error(f"variable '{ivar}' missing '_FillValue'.") elif default_fill is not None and fill_value != default_fill: - log_file.write( - f" - ERROR (attributes): variable '{ivar}' _FillValue {fill_value}" - f" does not match default netCDF4 fill value {default_fill} for dtype {dtype}.\n" + reporter.error( + f"variable '{ivar}' _FillValue {fill_value}" + f" does not match default netCDF4 fill value {default_fill} for dtype {dtype}." ) - fill_errors += 1 if fill_value is not None and missing_value is not None and fill_value != missing_value: - log_file.write( - f" - ERROR (attributes): variable '{ivar}' _FillValue {fill_value}" - f" and missing_value {missing_value} are not equal.\n" + reporter.error( + f"variable '{ivar}' _FillValue {fill_value}" + f" and missing_value {missing_value} are not equal." ) - fill_errors += 1 - if fill_errors == 0: - log_file.write(" - Fill value attributes: OK\n") - errors += fill_errors - - # Sub-test 5: main variable and time must be single-precision float (f4) - dtype_errors = 0 + if reporter.total_errors == errors_before: + reporter.ok("Fill value attributes: OK") + + # Sub-test 5: main variable and time must be single-precision float (f4). + # The two are not the same finding. A float64 data variable is twice the + # size it should be, for the archive and for everyone who has to move it, + # so it is an error. A time axis is one number per record, so storing it + # as float64 cannot meaningfully inflate a file and the size argument does + # not reach it: it is a warning. + errors_before = reporter.total_errors + warnings_before = reporter.total_warnings if ivar in ds and ds[ivar].dtype != np.float32: - log_file.write( - f" - ERROR (attributes): variable '{ivar}' dtype is {ds[ivar].dtype}," - f" expected float32 (f4).\n" + reporter.error( + f"variable '{ivar}' dtype is {ds[ivar].dtype}," + f" expected float32 (f4)." ) - dtype_errors += 1 if time_coord is not None: # xarray decodes CF time to datetime objects in memory; check the on-disk dtype from encoding time_encoded_dtype = ds[time_coord].encoding.get("dtype", ds[time_coord].dtype) if time_encoded_dtype != np.float32: - log_file.write( - f" - ERROR (attributes): coordinate '{time_coord}' on-disk dtype is {time_encoded_dtype}," - f" expected float32 (f4).\n" + reporter.warning( + f"coordinate '{time_coord}' on-disk dtype is {time_encoded_dtype}," + f" expected float32 (f4)." ) - dtype_errors += 1 - if dtype_errors == 0: - log_file.write(" - Dtype attributes: OK\n") - errors += dtype_errors + if ( + reporter.total_errors == errors_before + and reporter.total_warnings == warnings_before + ): + reporter.ok("Dtype attributes: OK") # Sub-test 6: scale_factor and add_offset must not be present - pack_errors = 0 + errors_before = reporter.total_errors if ivar in ds: # xarray moves these to .encoding on decode; check both locations combined = {**ds[ivar].attrs, **ds[ivar].encoding} for forbidden in ("scale_factor", "add_offset"): if forbidden in combined: - log_file.write( - f" - ERROR (attributes): variable '{ivar}' must not have '{forbidden}'.\n" + reporter.error( + f"variable '{ivar}' must not have '{forbidden}'." ) - pack_errors += 1 - if pack_errors == 0: - log_file.write(" - Packing attributes: OK\n") - errors += pack_errors - - return errors + if reporter.total_errors == errors_before: + reporter.ok("Packing attributes: OK") def _run_variable_checks( - log_file, + reporter: Reporter, ds, file_name: str, considered_variable: str, @@ -1858,16 +2118,16 @@ def _run_variable_checks( ismip_meta, experiments, report_naming_issues, -): - var_naming_errors = 0 - var_num_errors = 0 - var_spatial_errors = 0 - var_time_errors = 0 - var_attr_errors = 0 +) -> None: + naming_reporter = reporter.category("naming") + num_reporter = reporter.category("num") + spatial_reporter = reporter.category("spatial") + time_reporter = reporter.category("time") + attr_reporter = reporter.category("attr", qualifier="attributes") - log_file.write(" \n") - log_file.write("Experiment: " + experiment_name + " - File: " + file_name + "\n") - log_file.write(" \n") + reporter.write(" \n") + reporter.write("Experiment: " + experiment_name + " - File: " + file_name + "\n") + reporter.write(" \n") header_ds = ds.to_dict(data=False) dim = set(list(header_ds["coords"].keys())) @@ -1876,18 +2136,18 @@ def _run_variable_checks( isscalar = ismip_meta[index]["dim"] == "t" var_type = ismip_meta[index].get("type", "") - naming = _check_naming(log_file, file_name, region, dim, isscalar, report_naming_issues) - var_naming_errors += naming.errors + naming = _check_naming( + naming_reporter, file_name, region, dim, isscalar, report_naming_issues + ) if not naming.can_continue: - return var_naming_errors, var_num_errors, var_spatial_errors, var_time_errors, var_attr_errors + return - file_var_errors, has_variable = _check_file_variables( - log_file, ds, file_name, considered_variable, file_variables, + has_variable = _check_file_variables( + naming_reporter, ds, file_name, considered_variable, file_variables, ismip_meta[index]["dim"], report_naming_issues, ) - var_naming_errors += file_var_errors if not has_variable: - return var_naming_errors, var_num_errors, var_spatial_errors, var_time_errors, var_attr_errors + return grid_extent = AIS_GRID_EXTENT if region == "AIS" else GrIS_GRID_EXTENT possible_resolution = AIS_POSSIBLE_RESOLUTION if region == "AIS" else GrIS_POSSIBLE_RESOLUTION @@ -1898,52 +2158,71 @@ def _run_variable_checks( # or -- when nothing in it is a requested name -- against none at all. ivar = considered_variable var_index = index - log_file.write("** Tested Variable: " + ivar + "\n") - log_file.write(" \n") + reporter.write("** Tested Variable: " + ivar + "\n") + reporter.write(" \n") - var_num_errors += _check_numerical(log_file, ds, ivar, ismip_meta, var_index, region, isscalar) + _check_numerical(num_reporter, ds, ivar, ismip_meta, var_index, region, isscalar) if not isscalar and region not in ("AIS", "GrIS"): - log_file.write("SPATIAL Tests \n") - log_file.write( - " - Not checked: the expected grid extent and resolutions depend on" - " the region, which the file name does not identify.\n" + spatial_reporter.write("SPATIAL Tests \n") + spatial_reporter.note( + "Not checked: the expected grid extent and resolutions depend on" + " the region, which the file name does not identify." ) elif not isscalar: - var_spatial_errors += _check_spatial(log_file, ds, grid_extent, possible_resolution) + _check_spatial(spatial_reporter, ds, grid_extent, possible_resolution) - var_time_errors += _check_time( - log_file, ds, dim, experiments, experiment_name, var_type, + _check_time( + time_reporter, ds, dim, experiments, experiment_name, var_type, ismip_meta[index]["dim"], naming.filename_years, ) - var_attr_errors += _check_attributes(log_file, ds, ivar, ismip_meta, var_index, isscalar, var_type, region) + _check_attributes( + attr_reporter, ds, ivar, ismip_meta, var_index, isscalar, var_type, region + ) + + +def _warning_phrase(warnings: int) -> str: + """How the console mentions warnings, when there are any to mention. - return var_naming_errors, var_num_errors, var_spatial_errors, var_time_errors, var_attr_errors + Phrased so that it cannot be read as a failure: a run that reports only + warnings has passed. + """ + if warnings == 0: + return "" + return f" ({warnings} warning(s) — see the log)" -def _print_experiment_summary(experiment_name: str, exp_errors: int) -> None: +def _print_experiment_summary( + experiment_name: str, exp_errors: int, exp_warnings: int = 0 +) -> None: print(experiment_name, ": compliance check processed.") if exp_errors > 0: print( - "Found", exp_errors, "errors. Check compliance_checker_log.txt for details." + "Found", + exp_errors, + f"errors{_warning_phrase(exp_warnings)}." + " Check compliance_checker_log.txt for details.", ) else: - print("Successfully verified with no errors") + print("Successfully verified with no errors" + _warning_phrase(exp_warnings)) print() -def _print_total_summary(source_path: str, total_errors: int) -> None: +def _print_total_summary( + source_path: str, total_errors: int, total_warnings: int = 0 +) -> None: print("-------------------------------------------------------------------------") print(source_path, ": compliance check processed.") if total_errors > 0: print( "Found a total of", total_errors, - "errors. Check compliance_checker_log.txt for details.", + f"errors{_warning_phrase(total_warnings)}." + " Check compliance_checker_log.txt for details.", ) else: - print("Successfully verified with no errors") + print("Successfully verified with no errors" + _warning_phrase(total_warnings)) print("-------------------------------------------------------------------------") @@ -1992,43 +2271,54 @@ def _write_log_header( log_file.write(" \n") -def _insert_synthesis( - source_path: str, - exp_counter: int, - file_counter: int, - total_errors: int, - total_file_errors: int, - total_naming_errors: int, - total_num_errors: int, - total_spatial_errors: int, - total_time_errors: int, - total_attr_errors: int, - report_naming_issues, -) -> None: +# The reporting categories, in the order the synthesis block lists them, with +# the label each one carries there. +SYNTHESIS_CATEGORIES = ( + # 'Variable presence' rather than 'Mandatory variables': the category now + # also carries the warning about non-mandatory variables not submitted, + # which the old label contradicted. + ("file", "Variable presence "), + ("naming", "Naming Tests "), + ("num", "Numerical Tests "), + ("spatial", "Spatial Tests "), + ("time", "Time Tests "), + ("attr", "Attribute Tests "), +) + +# The summary key each category's counts live under, by severity. +_SUMMARY_KEYS = { + "file": "total_file_{severity}s", + "naming": "total_naming_{severity}s", + "num": "total_num_{severity}s", + "spatial": "total_spatial_{severity}s", + "time": "total_time_{severity}s", + "attr": "total_attr_{severity}s", +} + + +def _insert_synthesis(source_path: str, summary: dict) -> None: + report_naming_issues = summary["report_naming_issues"] + with open(os.path.join(source_path, "compliance_checker_log.txt"), "r") as f: contents = f.readlines() iline = 11 - contents.insert(iline, str(exp_counter) + " experiments checked.\n") - iline += 1 - contents.insert(iline, str(file_counter) + " files checked.\n") - iline += 2 - contents.insert(iline, str(total_errors) + " error(s) detected.\n") - iline += 1 - contents.insert(iline, " - Mandatory variables: " + str(total_file_errors) + " error(s)\n") - iline += 1 - contents.insert(iline, " - Naming Tests : " + str(total_naming_errors) + " error(s)\n") - iline += 1 - contents.insert(iline, " - Numerical Tests : " + str(total_num_errors) + " error(s)\n") - iline += 1 - contents.insert(iline, " - Spatial Tests : " + str(total_spatial_errors) + " error(s)\n") - iline += 1 - contents.insert(iline, " - Time Tests : " + str(total_time_errors) + " error(s)\n") + contents.insert(iline, str(summary["exp_counter"]) + " experiments checked.\n") iline += 1 - contents.insert(iline, " - Attribute Tests : " + str(total_attr_errors) + " error(s)\n") - iline += 2 - contents.insert(iline, "0 warning(s) detected.\n") + contents.insert(iline, str(summary["file_counter"]) + " files checked.\n") iline += 2 + for severity in ("error", "warning"): + contents.insert( + iline, str(summary[f"total_{severity}s"]) + f" {severity}(s) detected.\n" + ) + iline += 1 + for category, label in SYNTHESIS_CATEGORIES: + count = summary[_SUMMARY_KEYS[category].format(severity=severity)] + contents.insert(iline, f" - {label}: {count} {severity}(s)\n") + iline += 1 + # Step over one of the blank lines the header left behind, so that each + # block is separated from the next. + iline += 1 if report_naming_issues: contents.insert(iline, "Naming tests errors report: \n") iline += 1 diff --git a/isschecker/data/ISMIP7_variable_request.csv b/isschecker/data/ISMIP7_variable_request.csv index 889301e..142347e 100644 --- a/isschecker/data/ISMIP7_variable_request.csv +++ b/isschecker/data/ISMIP7_variable_request.csv @@ -1,43 +1,43 @@ -long_name,Dim,Type,Variable Name,standard_name,units,Mandatory (yes/no),Comment,min_value_ais,max_value_ais,min_value_gris,max_value_gris -Ice thickness,"x,y,t",ST,lithk,land_ice_thickness,m,yes,Ice thickness of the ice sheet,0,5500,0,5000 -Surface elevation,"x,y,t",ST,orog,surface_altitude,m,yes,Surface elevation of the ice sheet,0,5500,0,4500 -Bedrock elevation,"x,y,t",ST,topg,bedrock_altitude,m,yes,The bedrock topography (may change during the projections),-7000,5500,-4000,4000 -Ice base elevation,"x,y,t",ST,base,,m,yes,Should get standard name land_ice_temperature,-4000,4000,-4000,4000 -Geothermal heat flux,"x,y,t",FL,hfgeoubed,upward_geothermal_heat_flux_in_land_ice,W m-2,no,Geothermal Heat flux at the ice interface,0,0.3,0,0.3 -Surface mass balance flux,"x,y,t",FL,acabf,land_ice_surface_specific_mass_balance_flux,kg m-2 s-1,yes,Surface Mass Balance flux,-0.0006,0.001,-0.0006,0.001 -Basal mass balance flux beneath grounded ice,"x,y,t",FL,libmassbfgr,land_ice_basal_specific_mass_balance_flux,kg m-2 s-1,yes,Basal mass balance flux (only beneath grounded ice),-0.0003,0.0001,-0.0003,0.0001 -Basal mass balance flux beneath floating ice,"x,y,t",FL,libmassbffl,land_ice_basal_specific_mass_balance_flux,kg m-2 s-1,yes,Basal mass balance flux (only beneath floating ice),-0.008,0.001,-0.008,0.001 -Ice thickness imbalance,"x,y,t",FL,dlithkdt,tendency_of_land_ice_thickness,m s-1,yes,dHdt,-0.0001,0.0001,-0.0001,0.0001 -Surface velocity in x,"x,y,t",ST,xvelsurf,land_ice_surface_x_velocity,m s-1,no,u-velocity at land ice surface,-0.0004,0.0004,-0.0008,0.0008 -Surface velocity in y,"x,y,t",ST,yvelsurf,land_ice_surface_y_velocity,m s-1,no,v-velocity at land ice surface,-0.0004,0.0004,-0.0008,0.0008 -Surface velocity in z,"x,y,t",ST,zvelsurf,land_ice_surface_upward_velocity,m s-1,no,w-velocity at land ice surface,-4.00E-06,4.00E-06,-8.00E-06,8.00E-06 -Basal velocity in x,"x,y,t",ST,xvelbase,land_ice_basal_x_velocity,m s-1,no,u-velocity at land ice base,-0.0004,0.0004,-0.0008,0.0008 -Basal velocity in y,"x,y,t",ST,yvelbase,land_ice_basal_y_velocity,m s-1,no,v-velocity at land ice base,-0.0004,0.0004,-0.0008,0.0008 -Basal velocity in z,"x,y,t",ST,zvelbase,land_ice_basal_upward_velocity,m s-1,no,w-velocity at land ice base,-5.00E-06,5.00E-06,-8.00E-06,8.00E-06 -Mean velocity in x,"x,y,t",ST,xvelmean,land_ice_vertical_mean_x_velocity,m s-1,yes,Vertical mean land ice velocity ,-0.0008,0.0008,-0.0008,0.0008 -Mean velocity in y,"x,y,t",ST,yvelmean,land_ice_vertical_mean_y_velocity,m s-1,yes,The vertical mean land ice velocity is the average from the bedrock to the surface of the ice,-0.0008,0.0008,-0.0008,0.0008 -Surface temperature,"x,y,t",ST,litemptop,temperature_at_top_of_ice_sheet_model,K,no,Ice temperature at surface,183,290,183,290 -Depth average temperature,"x,y,t",ST,litempavg,,K,no,Should get standard name land_ice_temperature,183,290,183,290 -Basal temperature beneath grounded ice sheet,"x,y,t",ST,litempbotgr,temperature_at_base_of_ice_sheet_model,K,no,Ice temperature at base of grounded ice sheet,183,290,183,290 -Basal temperature beneath floating ice shelf,"x,y,t",ST,litempbotfl,temperature_at_base_of_ice_sheet_model,K,no,Ice temperature at base of floating ice shelf,183,290,183,290 -Basal drag,"x,y,t",ST,strbasemag,land_ice_basal_drag,Pa,yes,Basal drag,0,1.00E+06,0,1.00E+06 -Calving flux,"x,y,t",FL,licalvf,land_ice_specific_mass_flux_due_to_calving,kg m-2 s-1,yes,Ice mass change resulting from iceberg calving. Only for grid cells in contact with ocean,-1.00E+11,0,-1.00E+11,0 -Grounding line flux,"x,y,t",FL,ligroundf,,kg m-2 s-1,yes,Flux of ice mass across the grounding line. Only for grounding line grid cells. Should get standard name: land_ice_specific_grounding_line_flux,-1.00E+09,1.00E+11,-1.00E+09,1.00E+11 -Ice front melt flux,"x,y,t",FL,lifmassbf,,kg m-2 s-1,yes,Ice mass change resulting from ice front melting. Only for grid cells in contact with ocean. Should get standard name: land_ice_specific_mass_flux_due_to_ice_front_melting,-1.00E+11,0,-1.00E+11,0 -Land ice area fraction,"x,y,t",ST,sftgif,land_ice_area_fraction,1,yes,"Fraction of grid cell covered by land ice (ice sheet, ice shelf, ice cap, glacier)",0,1,0,1 -Grounded ice sheet area fraction,"x,y,t",ST,sftgrf,grounded_ice_sheet_area_fraction,1,yes,"Fraction of grid cell covered by grounded ice sheet, where grounded indicates that the quantity correspond to the ice sheet that flows over bedrock",0,1,0,1 -Floating ice sheet area fraction,"x,y,t",ST,sftflf,floating_ice_shelf_area_fraction,1,yes,Fraction of grid cell covered by ice sheet flowing over seawater,0,1,0,1 -Thermal forcing at the ice base under floating ice shelves,"x,y,t",ST,tfbase,,K,no,Thermal forcing interpolated to the ice draft under floating ice shelves (fill value for purely grounded ice or not ice),-1,30,-1,30 -Anomaly in geopotential height from reference geoid,"x,y,t",ST,deltag,geopotential_height_anomaly,m,no,The change in geoid height should be relative to the reference geoid,-4000,4000,-4000,4000 -Reference geoid,"x,y",ST,refgeoid,geoid_height_above_reference_ellipsoid,m,no,This field is calculated with respect to the WGS84 reference ellipsoid,-500,500,-500,500 -Ice temperature,"x,y,z,t",ST,litemp,land_ice_temperature,K,no,"3D temperature snapshots at (1900 if in historical), initial state, 2014, 2100, 2200, and 2300. Output on the model native vertical coordinate. Only for models that have 3D temperature calculation",183,274,183,274 -Total ice mass,t,ST,lim,land_ice_mass,kg,yes,"spatial integration, volume times density",0,1.00E+25,0,1.00E+25 -Mass above floatation,t,ST,limnsw,land_ice_mass_not_displacing_sea_water,kg,yes,"spatial integration, volume times density",0,1.00E+25,0,1.00E+25 -Grounded ice area,t,ST,iareagr,grounded_ice_sheet_area,m^2,yes,spatial integration,0,1.00E+25,0,1.00E+25 -Floating ice area,t,ST,iareafl,floating_ice_shelf_area,m^2,yes,spatial integration,0,1.00E+25,0,1.00E+25 -Total SMB flux,t,FL,tendacabf,tendency_of_land_ice_mass_due_to_surface_mass_balance,kg s-1,yes,spatial integration,0,1.00E+25,0,1.00E+25 -Total BMB flux beneath grounded ice,t,FL,tendlibmassbfgr,tendency_of_land_ice_mass_due_to_basal_mass_balance,kg s-1,yes,spatial integration,0,1.00E+25,0,1.00E+25 -Total BMB flux beneath floating ice,t,FL,tendlibmassbffl,tendency_of_land_ice_mass_due_to_basal_mass_balance,kg s-1,yes,spatial integration (computed beneath floating ice only),0,1.00E+25,0,1.00E+25 -Total calving flux,t,FL,tendlicalvf,tendency_of_land_ice_mass_due_to_calving,kg s-1,yes,spatial integration,0,1.00E+25,0,1.00E+25 -Total ice front melting flux,t,FL,tendlifmassbf,,kg s-1,yes,"spatial integration, should get standard name: tendency_of_land_ice_mass_due_to_ice_front_melting",0,1.00E+25,0,1.00E+25 -Total grounding line flux,t,FL,tendligroundf,,kg s-1,yes,"spatial integration, should get a standard name",0,1.00E+25,0,1.00E+25 \ No newline at end of file +long_name,Dim,Type,Variable Name,standard_name,units,Mandatory (yes/no),Comment,min_value_ais,max_value_ais,min_value_gris,max_value_gris,range_severity +Ice thickness,"x,y,t",ST,lithk,land_ice_thickness,m,yes,Ice thickness of the ice sheet,0,5500,0,5000,error +Surface elevation,"x,y,t",ST,orog,surface_altitude,m,yes,Surface elevation of the ice sheet,0,5500,0,4500,error +Bedrock elevation,"x,y,t",ST,topg,bedrock_altitude,m,yes,The bedrock topography (may change during the projections),-7000,5500,-4000,4000,error +Ice base elevation,"x,y,t",ST,base,,m,yes,Should get standard name land_ice_temperature,-4000,4000,-4000,4000,error +Geothermal heat flux,"x,y,t",FL,hfgeoubed,upward_geothermal_heat_flux_in_land_ice,W m-2,no,Geothermal Heat flux at the ice interface,0,0.3,0,0.3,error +Surface mass balance flux,"x,y,t",FL,acabf,land_ice_surface_specific_mass_balance_flux,kg m-2 s-1,yes,Surface Mass Balance flux,-0.0006,0.001,-0.0006,0.001,error +Basal mass balance flux beneath grounded ice,"x,y,t",FL,libmassbfgr,land_ice_basal_specific_mass_balance_flux,kg m-2 s-1,yes,Basal mass balance flux (only beneath grounded ice),-0.0003,0.0001,-0.0003,0.0001,error +Basal mass balance flux beneath floating ice,"x,y,t",FL,libmassbffl,land_ice_basal_specific_mass_balance_flux,kg m-2 s-1,yes,Basal mass balance flux (only beneath floating ice),-0.008,0.001,-0.008,0.001,error +Ice thickness imbalance,"x,y,t",FL,dlithkdt,tendency_of_land_ice_thickness,m s-1,yes,dHdt,-0.0001,0.0001,-0.0001,0.0001,error +Surface velocity in x,"x,y,t",ST,xvelsurf,land_ice_surface_x_velocity,m s-1,no,u-velocity at land ice surface,-0.0004,0.0004,-0.0008,0.0008,error +Surface velocity in y,"x,y,t",ST,yvelsurf,land_ice_surface_y_velocity,m s-1,no,v-velocity at land ice surface,-0.0004,0.0004,-0.0008,0.0008,error +Surface velocity in z,"x,y,t",ST,zvelsurf,land_ice_surface_upward_velocity,m s-1,no,w-velocity at land ice surface,-4.00E-06,4.00E-06,-8.00E-06,8.00E-06,error +Basal velocity in x,"x,y,t",ST,xvelbase,land_ice_basal_x_velocity,m s-1,no,u-velocity at land ice base,-0.0004,0.0004,-0.0008,0.0008,error +Basal velocity in y,"x,y,t",ST,yvelbase,land_ice_basal_y_velocity,m s-1,no,v-velocity at land ice base,-0.0004,0.0004,-0.0008,0.0008,error +Basal velocity in z,"x,y,t",ST,zvelbase,land_ice_basal_upward_velocity,m s-1,no,w-velocity at land ice base,-5.00E-06,5.00E-06,-8.00E-06,8.00E-06,error +Mean velocity in x,"x,y,t",ST,xvelmean,land_ice_vertical_mean_x_velocity,m s-1,yes,Vertical mean land ice velocity ,-0.0008,0.0008,-0.0008,0.0008,error +Mean velocity in y,"x,y,t",ST,yvelmean,land_ice_vertical_mean_y_velocity,m s-1,yes,The vertical mean land ice velocity is the average from the bedrock to the surface of the ice,-0.0008,0.0008,-0.0008,0.0008,error +Surface temperature,"x,y,t",ST,litemptop,temperature_at_top_of_ice_sheet_model,K,no,Ice temperature at surface,183,290,183,290,error +Depth average temperature,"x,y,t",ST,litempavg,,K,no,Should get standard name land_ice_temperature,183,290,183,290,error +Basal temperature beneath grounded ice sheet,"x,y,t",ST,litempbotgr,temperature_at_base_of_ice_sheet_model,K,no,Ice temperature at base of grounded ice sheet,183,290,183,290,error +Basal temperature beneath floating ice shelf,"x,y,t",ST,litempbotfl,temperature_at_base_of_ice_sheet_model,K,no,Ice temperature at base of floating ice shelf,183,290,183,290,error +Basal drag,"x,y,t",ST,strbasemag,land_ice_basal_drag,Pa,yes,Basal drag,0,1.00E+06,0,1.00E+06,error +Calving flux,"x,y,t",FL,licalvf,land_ice_specific_mass_flux_due_to_calving,kg m-2 s-1,yes,Ice mass change resulting from iceberg calving. Only for grid cells in contact with ocean,-1.00E+11,0,-1.00E+11,0,error +Grounding line flux,"x,y,t",FL,ligroundf,,kg m-2 s-1,yes,Flux of ice mass across the grounding line. Only for grounding line grid cells. Should get standard name: land_ice_specific_grounding_line_flux,-1.00E+09,1.00E+11,-1.00E+09,1.00E+11,error +Ice front melt flux,"x,y,t",FL,lifmassbf,,kg m-2 s-1,yes,Ice mass change resulting from ice front melting. Only for grid cells in contact with ocean. Should get standard name: land_ice_specific_mass_flux_due_to_ice_front_melting,-1.00E+11,0,-1.00E+11,0,error +Land ice area fraction,"x,y,t",ST,sftgif,land_ice_area_fraction,1,yes,"Fraction of grid cell covered by land ice (ice sheet, ice shelf, ice cap, glacier)",0,1,0,1,error +Grounded ice sheet area fraction,"x,y,t",ST,sftgrf,grounded_ice_sheet_area_fraction,1,yes,"Fraction of grid cell covered by grounded ice sheet, where grounded indicates that the quantity correspond to the ice sheet that flows over bedrock",0,1,0,1,error +Floating ice sheet area fraction,"x,y,t",ST,sftflf,floating_ice_shelf_area_fraction,1,yes,Fraction of grid cell covered by ice sheet flowing over seawater,0,1,0,1,error +Thermal forcing at the ice base under floating ice shelves,"x,y,t",ST,tfbase,,K,no,Thermal forcing interpolated to the ice draft under floating ice shelves (fill value for purely grounded ice or not ice),-1,30,-1,30,error +Anomaly in geopotential height from reference geoid,"x,y,t",ST,deltag,geopotential_height_anomaly,m,no,The change in geoid height should be relative to the reference geoid,-4000,4000,-4000,4000,error +Reference geoid,"x,y",ST,refgeoid,geoid_height_above_reference_ellipsoid,m,no,This field is calculated with respect to the WGS84 reference ellipsoid,-500,500,-500,500,error +Ice temperature,"x,y,z,t",ST,litemp,land_ice_temperature,K,no,"3D temperature snapshots at (1900 if in historical), initial state, 2014, 2100, 2200, and 2300. Output on the model native vertical coordinate. Only for models that have 3D temperature calculation",183,274,183,274,error +Total ice mass,t,ST,lim,land_ice_mass,kg,yes,"spatial integration, volume times density",0,1.00E+25,0,1.00E+25,error +Mass above floatation,t,ST,limnsw,land_ice_mass_not_displacing_sea_water,kg,yes,"spatial integration, volume times density",0,1.00E+25,0,1.00E+25,error +Grounded ice area,t,ST,iareagr,grounded_ice_sheet_area,m^2,yes,spatial integration,0,1.00E+25,0,1.00E+25,error +Floating ice area,t,ST,iareafl,floating_ice_shelf_area,m^2,yes,spatial integration,0,1.00E+25,0,1.00E+25,error +Total SMB flux,t,FL,tendacabf,tendency_of_land_ice_mass_due_to_surface_mass_balance,kg s-1,yes,spatial integration,0,1.00E+25,0,1.00E+25,error +Total BMB flux beneath grounded ice,t,FL,tendlibmassbfgr,tendency_of_land_ice_mass_due_to_basal_mass_balance,kg s-1,yes,spatial integration,0,1.00E+25,0,1.00E+25,error +Total BMB flux beneath floating ice,t,FL,tendlibmassbffl,tendency_of_land_ice_mass_due_to_basal_mass_balance,kg s-1,yes,spatial integration (computed beneath floating ice only),0,1.00E+25,0,1.00E+25,error +Total calving flux,t,FL,tendlicalvf,tendency_of_land_ice_mass_due_to_calving,kg s-1,yes,spatial integration,0,1.00E+25,0,1.00E+25,error +Total ice front melting flux,t,FL,tendlifmassbf,,kg s-1,yes,"spatial integration, should get standard name: tendency_of_land_ice_mass_due_to_ice_front_melting",0,1.00E+25,0,1.00E+25,error +Total grounding line flux,t,FL,tendligroundf,,kg s-1,yes,"spatial integration, should get a standard name",0,1.00E+25,0,1.00E+25,error \ No newline at end of file diff --git a/tests/reference/compliance_checker_log.txt b/tests/reference/compliance_checker_log.txt index 9da4c02..9803d10 100644 --- a/tests/reference/compliance_checker_log.txt +++ b/tests/reference/compliance_checker_log.txt @@ -13,14 +13,20 @@ Verified directory: 27 files checked. 0 error(s) detected. - - Mandatory variables: 0 error(s) + - Variable presence : 0 error(s) - Naming Tests : 0 error(s) - Numerical Tests : 0 error(s) - Spatial Tests : 0 error(s) - Time Tests : 0 error(s) - Attribute Tests : 0 error(s) -0 warning(s) detected. +1 warning(s) detected. + - Variable presence : 1 warning(s) + - Naming Tests : 0 warning(s) + - Numerical Tests : 0 warning(s) + - Spatial Tests : 0 warning(s) + - Time Tests : 0 warning(s) + - Attribute Tests : 0 warning(s) ==================================================================================== @@ -34,6 +40,7 @@ Hint: Use Cltr+F to look for specific problems. ********************************************************** Mandatory variables Test: historical : all mandatory variables exist. +WARNING: experiment historical carries no files for the non-mandatory variable(s): ['hfgeoubed', 'xvelsurf', 'yvelsurf', 'zvelsurf', 'xvelbase', 'yvelbase', 'zvelbase', 'litemptop', 'litempavg', 'litempbotgr', 'litempbotfl', 'tfbase', 'deltag', 'refgeoid', 'litemp']. This is expected if your model does not represent them; it is listed only so that a variable lost from a submission does not pass unnoticed. Experiment: historical - File: acabf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc diff --git a/tests/reference/compliance_checker_log_non_mandatory.txt b/tests/reference/compliance_checker_log_non_mandatory.txt new file mode 100644 index 0000000..c3e9bea --- /dev/null +++ b/tests/reference/compliance_checker_log_non_mandatory.txt @@ -0,0 +1,1411 @@ +************************************************************************************ +************* Ice Sheet Model Simulations - Compliance Checker ************* +************************************************************************************ +isschecker version: +verification criteria: ISMIP7_variable_request.csv +date: +source: https://github.com/ismip/ISM_SimulationChecker + +------------------------------------------------------------------------------------ +Verified directory: +------------------------------------------------------------------------------------ +1 experiments checked. +42 files checked. + +0 error(s) detected. + - Variable presence : 0 error(s) + - Naming Tests : 0 error(s) + - Numerical Tests : 0 error(s) + - Spatial Tests : 0 error(s) + - Time Tests : 0 error(s) + - Attribute Tests : 0 error(s) + +0 warning(s) detected. + - Variable presence : 0 warning(s) + - Naming Tests : 0 warning(s) + - Numerical Tests : 0 warning(s) + - Spatial Tests : 0 warning(s) + - Time Tests : 0 warning(s) + - Attribute Tests : 0 warning(s) + + +==================================================================================== +================ DETAILED RESULTS ================ +==================================================================================== +Hint: Use Cltr+F to look for specific problems. + + + ********************************************************** + ** Experiment: historical + ********************************************************** + + Mandatory variables Test: historical : all mandatory variables exist. + +Experiment: historical - File: acabf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'acabf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'acabf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: acabf + +NUMERICAL Tests + - The unit is correct: kg m-2 s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - acabf - File:acabf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: base_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'base' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'base' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: base + +NUMERICAL Tests + - The unit is correct: m + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - base - File:base_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: deltag_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'deltag' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'deltag' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: deltag + +NUMERICAL Tests + - The unit is correct: m + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - deltag - File:deltag_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: dlithkdt_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'dlithkdt' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'dlithkdt' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: dlithkdt + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - dlithkdt - File:dlithkdt_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: hfgeoubed_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'hfgeoubed' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'hfgeoubed' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: hfgeoubed + +NUMERICAL Tests + - The unit is correct: W m-2 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - hfgeoubed - File:hfgeoubed_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: iareafl_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'iareafl' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'iareafl' dimensions (time) match the requested t: OK +** Tested Variable: iareafl + +NUMERICAL Tests + - The unit is correct: m^2 +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - iareafl - File:iareafl_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: iareagr_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'iareagr' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'iareagr' dimensions (time) match the requested t: OK +** Tested Variable: iareagr + +NUMERICAL Tests + - The unit is correct: m^2 +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - iareagr - File:iareagr_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: libmassbffl_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'libmassbffl' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'libmassbffl' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: libmassbffl + +NUMERICAL Tests + - The unit is correct: kg m-2 s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - libmassbffl - File:libmassbffl_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: libmassbfgr_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'libmassbfgr' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'libmassbfgr' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: libmassbfgr + +NUMERICAL Tests + - The unit is correct: kg m-2 s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - libmassbfgr - File:libmassbfgr_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: licalvf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'licalvf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'licalvf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: licalvf + +NUMERICAL Tests + - The unit is correct: kg m-2 s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - licalvf - File:licalvf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: lifmassbf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'lifmassbf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'lifmassbf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: lifmassbf + +NUMERICAL Tests + - The unit is correct: kg m-2 s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - lifmassbf - File:lifmassbf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: ligroundf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'ligroundf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'ligroundf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: ligroundf + +NUMERICAL Tests + - The unit is correct: kg m-2 s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - ligroundf - File:ligroundf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: lim_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'lim' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'lim' dimensions (time) match the requested t: OK +** Tested Variable: lim + +NUMERICAL Tests + - The unit is correct: kg +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - lim - File:lim_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: limnsw_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'limnsw' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'limnsw' dimensions (time) match the requested t: OK +** Tested Variable: limnsw + +NUMERICAL Tests + - The unit is correct: kg +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - limnsw - File:limnsw_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: litemp_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'litemp' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'litemp' dimensions (time, z, y, x) match the requested x,y,z,t: OK +** Tested Variable: litemp + +NUMERICAL Tests + - The unit is correct: K + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Snapshot time axis: nominal year(s) 2013-2014 cover everything experiment 'historical' requires: OK + - Annual cadence checks: N/A (snapshot variable — the time axis holds sparse snapshots by design). +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - litemp - File:litemp_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: litempavg_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'litempavg' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'litempavg' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: litempavg + +NUMERICAL Tests + - The unit is correct: K + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - litempavg - File:litempavg_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: litempbotfl_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'litempbotfl' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'litempbotfl' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: litempbotfl + +NUMERICAL Tests + - The unit is correct: K + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - litempbotfl - File:litempbotfl_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: litempbotgr_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'litempbotgr' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'litempbotgr' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: litempbotgr + +NUMERICAL Tests + - The unit is correct: K + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - litempbotgr - File:litempbotgr_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: litemptop_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'litemptop' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'litemptop' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: litemptop + +NUMERICAL Tests + - The unit is correct: K + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - litemptop - File:litemptop_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: lithk_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'lithk' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'lithk' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: lithk + +NUMERICAL Tests + - The unit is correct: m + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - lithk - File:lithk_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: orog_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'orog' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'orog' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: orog + +NUMERICAL Tests + - The unit is correct: m + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - orog - File:orog_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: refgeoid_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_0000-0000.nc + +NAMING Tests + - Filename year range: N/A (static spatial variable) + - Variable 'refgeoid' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'refgeoid' dimensions (y, x) match the requested x,y: OK +** Tested Variable: refgeoid + +NUMERICAL Tests + - The unit is correct: m + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time axis: N/A (static spatial variable) +ATTRIBUTE Tests + - Global attributes: OK + - Time coordinate: N/A (static spatial variable) + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - refgeoid - File:refgeoid_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_0000-0000.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: sftflf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'sftflf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'sftflf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: sftflf + +NUMERICAL Tests + - The unit is correct: 1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - sftflf - File:sftflf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: sftgif_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'sftgif' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'sftgif' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: sftgif + +NUMERICAL Tests + - The unit is correct: 1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - sftgif - File:sftgif_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: sftgrf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'sftgrf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'sftgrf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: sftgrf + +NUMERICAL Tests + - The unit is correct: 1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - sftgrf - File:sftgrf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: strbasemag_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'strbasemag' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'strbasemag' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: strbasemag + +NUMERICAL Tests + - The unit is correct: Pa + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - strbasemag - File:strbasemag_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: tendacabf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'tendacabf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'tendacabf' dimensions (time) match the requested t: OK +** Tested Variable: tendacabf + +NUMERICAL Tests + - The unit is correct: kg s-1 +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - tendacabf - File:tendacabf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: tendlibmassbffl_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'tendlibmassbffl' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'tendlibmassbffl' dimensions (time) match the requested t: OK +** Tested Variable: tendlibmassbffl + +NUMERICAL Tests + - The unit is correct: kg s-1 +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - tendlibmassbffl - File:tendlibmassbffl_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: tendlibmassbfgr_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'tendlibmassbfgr' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'tendlibmassbfgr' dimensions (time) match the requested t: OK +** Tested Variable: tendlibmassbfgr + +NUMERICAL Tests + - The unit is correct: kg s-1 +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - tendlibmassbfgr - File:tendlibmassbfgr_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: tendlicalvf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'tendlicalvf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'tendlicalvf' dimensions (time) match the requested t: OK +** Tested Variable: tendlicalvf + +NUMERICAL Tests + - The unit is correct: kg s-1 +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - tendlicalvf - File:tendlicalvf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: tendlifmassbf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'tendlifmassbf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'tendlifmassbf' dimensions (time) match the requested t: OK +** Tested Variable: tendlifmassbf + +NUMERICAL Tests + - The unit is correct: kg s-1 +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - tendlifmassbf - File:tendlifmassbf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: tendligroundf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'tendligroundf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'tendligroundf' dimensions (time) match the requested t: OK +** Tested Variable: tendligroundf + +NUMERICAL Tests + - The unit is correct: kg s-1 +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual FL time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - tendligroundf - File:tendligroundf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: tfbase_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'tfbase' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'tfbase' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: tfbase + +NUMERICAL Tests + - The unit is correct: K + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - tfbase - File:tfbase_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: topg_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'topg' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'topg' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: topg + +NUMERICAL Tests + - The unit is correct: m + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - topg - File:topg_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: xvelbase_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'xvelbase' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'xvelbase' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: xvelbase + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - xvelbase - File:xvelbase_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: xvelmean_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'xvelmean' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'xvelmean' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: xvelmean + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - xvelmean - File:xvelmean_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: xvelsurf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'xvelsurf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'xvelsurf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: xvelsurf + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - xvelsurf - File:xvelsurf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: yvelbase_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'yvelbase' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'yvelbase' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: yvelbase + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - yvelbase - File:yvelbase_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: yvelmean_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'yvelmean' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'yvelmean' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: yvelmean + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - yvelmean - File:yvelmean_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: yvelsurf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'yvelsurf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'yvelsurf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: yvelsurf + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - yvelsurf - File:yvelsurf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: zvelbase_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'zvelbase' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'zvelbase' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: zvelbase + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - zvelbase - File:zvelbase_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- + +Experiment: historical - File: zvelsurf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc + +NAMING Tests + - Filename year range 2013-2014 is well formed: OK + - Variable 'zvelsurf' from the file name is present in the file: OK + - No unexpected variables in the file: OK + - Variable 'zvelsurf' dimensions (time, y, x) match the requested x,y,t: OK +** Tested Variable: zvelsurf + +NUMERICAL Tests + - The unit is correct: m s-1 + - The minimum value successfully verified. + - The maximum value successfully verified. +SPATIAL Tests + - Grid: Lowest left corner is well defined. + - Grid: Upper right corner is well defined. + - The grid resolution (16 km) was successfully verified. +TIME Tests + - Time is a record (unlimited) dimension: OK + - Time axis: 2 annual ST time step(s) covering nominal years 2013-2014, as experiment 'historical' requires: OK +ATTRIBUTE Tests + - Global attributes: OK + - Coordinate attributes: OK + - Variable attributes: OK + - Fill value attributes: OK + - Dtype attributes: OK + - Packing attributes: OK + +---------------------------------------------------------- +historical - zvelsurf - File:zvelsurf_GrIS_ISMIP7_SYNTH1_m001_CESM2-WACCM_f001_historical_C001_2013-2014.nc +No errors. Good job ! +No warnings. +---------------------------------------------------------- diff --git a/tests/test_compliance_checker.py b/tests/test_compliance_checker.py index 02819ab..5c2e34c 100644 --- a/tests/test_compliance_checker.py +++ b/tests/test_compliance_checker.py @@ -1,9 +1,11 @@ +import io import shutil from datetime import datetime from pathlib import Path import netCDF4 import pytest +import xarray as xr # The installed package is what is tested: run `pip install --no-deps # --no-build-isolation -e .` (or without -e) before pytest. @@ -246,6 +248,16 @@ def add_variable_to_file(file_path: Path, name: str) -> None: dataset.createVariable(name, "f4", ("time",)) +def add_companion_variable( + file_path: Path, name: str, attribute: str, value: str +) -> None: + """Add a variable and have the requested one name it, the CF way.""" + variable_name = file_path.name.split("_")[0] + with netCDF4.Dataset(file_path, "a") as dataset: + dataset.createVariable(name, "f4", ("time",)) + dataset.variables[variable_name].setncattr(attribute, value) + + def _laid_out_as(values, old_dimensions, new_dimensions): """Re-lay values from `old_dimensions` to `new_dimensions`. @@ -320,6 +332,155 @@ def test_checker_reports_missing_mandatory_variable(case_dir): assert "mandatory variable(s) is (are) missing" in summary["log_text"] +def test_checker_warns_about_non_mandatory_variables_not_submitted(xyt_case_dir): + """A dropped optional file gets a signal, and never gets called an error. + + Nothing reported this before, so a group that meant to submit litemp and + lost it in a script heard nothing at all. + """ + dataset_for_variable(xyt_case_dir, "litemp").unlink() + + summary = run_xyt_checker(xyt_case_dir) + + assert summary["total_errors"] == 0, summary["log_text"] + assert summary["total_file_warnings"] == 1 + assert summary["total_warnings"] == 1 + assert ( + "WARNING: experiment historical carries no files for the non-mandatory" + " variable(s): ['litemp']" in summary["log_text"] + ) + # Wording, not accusation: a model that does not represent a variable is the + # common case, and the line has to read that way. + assert ( + "This is expected if your model does not represent them" + in summary["log_text"] + ) + # And it is not a fault, so it stays out of the list of faults. + synthesis = summary["log_text"].split("DETAILED RESULTS")[0] + assert "Naming tests errors report:" not in synthesis + + +def test_non_mandatory_warning_names_every_variable_on_one_line(xyt_case_dir): + """One line per experiment, not one warning per variable. + + A model with a narrow scope would otherwise look far worse than one that + dropped a single file, which is the opposite of what this is for. + """ + for variable in ("litemp", "refgeoid", "tfbase"): + dataset_for_variable(xyt_case_dir, variable).unlink() + + summary = run_xyt_checker(xyt_case_dir) + + assert summary["total_warnings"] == 1 + assert summary["log_text"].count("carries no files for the non-mandatory") == 1 + for variable in ("litemp", "refgeoid", "tfbase"): + assert f"'{variable}'" in summary["log_text"] + + +def test_non_mandatory_warning_is_scoped_to_the_selected_variable_list(xyt_case_dir): + """A scalars-only run says nothing about x,y,t variables it never looked at.""" + summary = checker.run_checker( + source_path=str(xyt_case_dir), + variable_list="ismip7_scalars", + version="tests", + ) + + assert "carries no files for the non-mandatory" not in summary["log_text"] + + +def test_a_file_with_warnings_and_no_errors_still_passes(xyt_case_dir): + """Warnings never change a verdict, at any level of the report.""" + dataset_for_variable(xyt_case_dir, "litemp").unlink() + + summary = run_xyt_checker(xyt_case_dir) + + assert summary["total_errors"] == 0 + assert summary["total_warnings"] > 0 + assert "No errors. Good job !" in summary["log_text"] + assert " error(s). Please review before sharing." not in summary["log_text"] + + +def write_not_modelled(case_dir: Path, text: str) -> None: + (case_dir / "not_modelled.txt").write_text(text) + + +def test_not_modelled_suppresses_the_not_submitted_warning(xyt_case_dir): + dataset_for_variable(xyt_case_dir, "litemp").unlink() + write_not_modelled( + xyt_case_dir, + "# ISMIP7: variables this model does not represent.\n" + "\n" + "litemp # no 3D temperature in this configuration\n", + ) + + summary = run_xyt_checker(xyt_case_dir) + + assert summary["total_errors"] == 0, summary["log_text"] + assert summary["total_warnings"] == 0 + assert "carries no files for the non-mandatory" not in summary["log_text"] + # What was claimed is on the record, not merely the absence of a warning. + assert ( + "Declared not modelled (not_modelled.txt): ['litemp']" + in summary["log_text"] + ) + + +def test_not_modelled_cannot_opt_out_of_a_mandatory_variable(xyt_case_dir): + """The declaration is about optional variables, and says so twice over. + + The missing mandatory file is still an error, and claiming it as not + modelled is an error of its own -- a silent no-op would look like + protection while protecting nothing. + """ + dataset_for_variable(xyt_case_dir, "lithk").unlink() + write_not_modelled(xyt_case_dir, "lithk\n") + + summary = run_xyt_checker(xyt_case_dir) + + assert summary["total_file_errors"] == 2 + assert "mandatory variable(s) is (are) missing: ['lithk']" in summary["log_text"] + assert ( + "ERROR: not_modelled.txt lists 'lithk', which the data request makes" + " mandatory" in summary["log_text"] + ) + + +def test_not_modelled_faults_a_mandatory_variable_outside_the_selected_list(case_dir): + """A declaration is about the submission, not about one run of the checker.""" + write_not_modelled(case_dir, "lithk\n") + + summary = run_checker(case_dir) + + assert summary["total_file_errors"] == 1 + assert ( + "ERROR: not_modelled.txt lists 'lithk', which the data request makes" + " mandatory" in summary["log_text"] + ) + + +def test_not_modelled_faults_a_name_that_is_not_in_the_data_request(xyt_case_dir): + write_not_modelled(xyt_case_dir, "litempbot\n") + + summary = run_xyt_checker(xyt_case_dir) + + assert summary["total_file_errors"] == 1 + assert ( + "ERROR: not_modelled.txt lists 'litempbot', which is not a variable of" + " the data request" in summary["log_text"] + ) + assert "The closest requested name is 'litempbotgr'" in summary["log_text"] + + +def test_no_not_modelled_file_changes_nothing(xyt_case_dir): + dataset_for_variable(xyt_case_dir, "litemp").unlink() + + summary = run_xyt_checker(xyt_case_dir) + + assert summary["total_errors"] == 0 + assert summary["total_warnings"] == 1 + assert "Declared not modelled" not in summary["log_text"] + + def test_checker_reports_invalid_esm_in_filename(case_dir): rename_file_part( first_dataset(case_dir), @@ -662,17 +823,53 @@ def test_checker_reports_swapped_variable_in_file(case_dir): assert "does not match expected 'land_ice_mass" not in summary["log_text"] -def test_checker_reports_unexpected_variable_in_file(case_dir): +def test_checker_warns_about_unexpected_variable_in_file(case_dir): + """An extra variable is a warning: the requested one is still fully checked. + + Nothing downstream has to work around a companion the data request did not + ask for, so this is a 'look at this', not a 'fix this'. + """ add_variable_to_file(dataset_for_variable(case_dir, "lim"), "mask") summary = run_checker(case_dir) - assert summary["total_naming_errors"] == 1 - assert summary["total_errors"] == 1 - assert "unexpected variable 'mask' in the file" in summary["log_text"] + assert summary["total_errors"] == 0, summary["log_text"] + assert summary["total_naming_warnings"] == 1 + assert summary["total_warnings"] == 1 + assert ( + "WARNING: unexpected variable 'mask' in the file" in summary["log_text"] + ) # An extra variable does not make the file uncheckable, so the requested - # one is still checked. + # one is still checked... assert "** Tested Variable: lim\n" in summary["log_text"] + # ... and the file's verdict is unchanged by the warning. + assert "No errors. Good job !" in summary["log_text"] + assert "1 warning(s). Please review." in summary["log_text"] + + +@pytest.mark.parametrize( + "attribute, value", + [ + ("cell_measures", "area: cellarea"), + ("ancillary_variables", "cellarea"), + ], +) +def test_checker_accepts_cf_companion_variables(case_dir, attribute, value): + """A companion the requested variable names is part of the file, not an extra. + + The allowlist already understood bounds, grid_mapping and coordinates, so a + CF-legal cell_measures or ancillary_variables companion -- carried precisely + so that the requested variable means what it says -- was reported as a + violation for no reason. + """ + add_companion_variable( + dataset_for_variable(case_dir, "lim"), "cellarea", attribute, value + ) + + summary = run_checker(case_dir) + + assert summary["total_errors"] == 0, summary["log_text"] + assert "ERROR: unexpected variable" not in summary["log_text"] def test_generated_xyt_dataset_passes_checker(xyt_case_dir): @@ -706,27 +903,36 @@ def test_checker_reports_litemp_missing_required_snapshot(xyt_case_dir): ) -def test_checker_reports_litemp_snapshot_year_not_requested(xyt_case_dir): +def test_checker_warns_about_litemp_snapshot_year_not_requested(xyt_case_dir): + """An unasked-for snapshot is a warning, not an error. + + The data request specifies the snapshot years as a minimum set, so + over-delivering 3D temperature is not non-compliance. It is still worth + naming: a year nobody asked for is usually a sign of a mistake. + """ set_time_values( dataset_for_variable(xyt_case_dir, "litemp"), state_timestamps([1975, 2014]) ) summary = run_xyt_checker(xyt_case_dir) - assert summary["total_time_errors"] >= 1 + # 1975 replaces the required 2013 snapshot here, so the missing one is + # still an error; what changed is that the extra one no longer is. + assert summary["total_time_warnings"] == 1 assert ( - "snapshot nominal year(s) the experiment does not call for: 1975" + "WARNING: snapshot nominal year(s) the experiment does not call for: 1975" in summary["log_text"] ) -def test_checker_accepts_litemp_snapshot_at_2000(tmp_path): - """2000 is tolerated but not required, pending issue #12. +def test_checker_warns_about_litemp_snapshot_at_2000(tmp_path): + """2000 is not required, and is now said out loud rather than tolerated. The generator no longer writes it and the data request does not ask for it, - but the README did until now, so a group that followed the README must not - be failed for a year that is still in dispute. Nothing else covers the - 'permitted but not required' path once the generator stops emitting it. + but the README did until recently, so a group that followed the README must + not be failed for a year that is still in dispute (issue #12). Silently + accepting it told such a group nothing at all; a warning is the honest + report -- it is here, it was not asked for, it is not held against you. """ root = tmp_path / "gen" generate_test_files.create_netcdf_file( @@ -750,8 +956,12 @@ def test_checker_accepts_litemp_snapshot_at_2000(tmp_path): source_path=str(core_dir), variable_list="ismip7_xyt", version="tests" ) - assert summary["total_time_errors"] == 0, summary["log_text"] - assert "does not call for" not in summary["log_text"] + assert summary["total_errors"] == 0, summary["log_text"] + assert summary["total_time_warnings"] == 1 + assert ( + "WARNING: snapshot nominal year(s) the experiment does not call for: 2000" + in summary["log_text"] + ) def test_checker_reports_missing_variable_dimension(xyt_case_dir): @@ -822,6 +1032,176 @@ def test_checker_ignores_variables_outside_the_selected_list(xyt_case_dir): assert summary["total_naming_errors"] == 0 +def set_variable_dtype(file_path: Path, variable_name: str, datatype: str) -> None: + """Rewrite one variable with a different on-disk dtype. + + netCDF cannot retype a variable in place, so the file is rebuilt; every + other variable, every attribute and the unlimited dimension are copied + across unchanged. + """ + rebuilt = file_path.with_name(file_path.name + ".rebuilt") + with netCDF4.Dataset(file_path) as source, netCDF4.Dataset(rebuilt, "w") as target: + target.setncatts(source.__dict__) + for name, dimension in source.dimensions.items(): + target.createDimension( + name, None if dimension.isunlimited() else len(dimension) + ) + for name, variable in source.variables.items(): + fill_value = ( + variable.getncattr("_FillValue") + if "_FillValue" in variable.ncattrs() + else None + ) + created = target.createVariable( + name, + datatype if name == variable_name else variable.datatype, + variable.dimensions, + fill_value=fill_value, + ) + created.setncatts( + {k: v for k, v in variable.__dict__.items() if k != "_FillValue"} + ) + created[:] = variable[:] + rebuilt.replace(file_path) + + +def test_checker_warns_about_a_time_coordinate_that_is_not_float32(case_dir): + """A float64 time axis is a warning; a float64 data variable is an error. + + The size argument that governs the data variable does not reach the time + axis: it is one number per record, so double precision cannot meaningfully + inflate a file. + """ + set_variable_dtype(dataset_for_variable(case_dir, "lim"), "time", "f8") + + summary = run_checker(case_dir) + + assert summary["total_errors"] == 0, summary["log_text"] + assert summary["total_attr_warnings"] == 1 + assert ( + "WARNING (attributes): coordinate 'time' on-disk dtype is float64" + in summary["log_text"] + ) + + +def test_checker_reports_a_main_variable_that_is_not_float32(case_dir): + """The other half of the split: the data variable keeps erroring.""" + set_variable_dtype(dataset_for_variable(case_dir, "lim"), "lim", "f8") + + summary = run_checker(case_dir) + + assert summary["total_attr_errors"] == 1 + assert summary["total_errors"] == 1 + assert ( + "ERROR (attributes): variable 'lim' dtype is float64" in summary["log_text"] + ) + + +@pytest.mark.parametrize( + "range_severity, errors, warnings", + [ + ("error", 2, 0), + ("warning", 0, 2), + # A blank cell, an unrecognised value and a missing column all have to + # mean what the checker did before the column existed. + (None, 2, 0), + ("nonsense", 2, 0), + ], +) +def test_range_severity_is_honoured(xyt_case_dir, range_severity, errors, warnings): + """Driven by a synthetic criteria row, not by editing the shipped CSV. + + The mechanism and the data are then tested independently: every shipped row + is `error` today (see test_shipped_range_severities_are_all_errors), so + nothing here would be exercised by a run over the real request. + """ + criteria = { + "variable": "lithk", + "dim": "x,y,t", + "units": "m", + "standard_name": "land_ice_thickness", + "type": "ST", + # Bounds no data can satisfy, so both ends fire. + "min_value_gris": 1.0e9, + "max_value_gris": -1.0e9, + } + if range_severity is not None: + criteria["range_severity"] = range_severity + + log = io.StringIO() + reporter = checker.Reporter(log).category("num") + with xr.open_dataset( + dataset_for_variable(xyt_case_dir, "lithk"), decode_times=False + ) as ds: + checker._check_numerical( + reporter, ds, "lithk", [criteria], 0, "GrIS", isscalar=False + ) + + assert reporter.total_errors == errors, log.getvalue() + assert reporter.total_warnings == warnings, log.getvalue() + assert "is out of range" in log.getvalue() + + +def test_shipped_range_severities_are_all_errors(): + """The mechanism ships with no classification changed. + + Which variables have bounds a legitimate model can exceed is case-by-case + work, done as data-only changes after this. Until then every row is `error`, + so behaviour is exactly what it was. + """ + ismip_meta, _, _, _, _ = checker._load_criteria("ismip7") + + assert {entry["range_severity"] for entry in ismip_meta} == {"error"} + + +def run_main(monkeypatch, source_path, variable_list="ismip7_scalars") -> int: + monkeypatch.setattr( + "sys.argv", + [ + "ismip7-compliance-checker", + "--source-path", + str(source_path), + "--variable-list", + variable_list, + ], + ) + return checker.main() + + +def test_exit_status_is_zero_for_a_compliant_submission(monkeypatch, case_dir): + assert run_main(monkeypatch, case_dir) == 0 + + +def test_exit_status_is_non_zero_when_there_are_errors(monkeypatch, case_dir): + first_dataset(case_dir).unlink() + + assert run_main(monkeypatch, case_dir) != 0 + + +def test_exit_status_is_zero_when_there_are_only_warnings(monkeypatch, xyt_case_dir): + """The whole point of a warning is that it does not fail a run.""" + dataset_for_variable(xyt_case_dir, "litemp").unlink() + + summary = run_xyt_checker(xyt_case_dir) + assert summary["total_warnings"] > 0 and summary["total_errors"] == 0 + + assert run_main(monkeypatch, xyt_case_dir, "ismip7_xyt") == 0 + + +def test_exit_status_is_non_zero_for_a_missing_source_directory(monkeypatch, tmp_path): + """Nothing was checked, so the zero error count is not a pass.""" + assert run_main(monkeypatch, tmp_path / "nowhere") != 0 + + +def test_exit_status_is_non_zero_for_a_source_directory_with_no_files( + monkeypatch, tmp_path +): + empty = tmp_path / "empty" + empty.mkdir() + + assert run_main(monkeypatch, empty) != 0 + + def test_checker_reports_missing_contact_email_attribute(case_dir): remove_global_attribute(first_dataset(case_dir), "contact_email") diff --git a/tests/test_golden_log.py b/tests/test_golden_log.py index 031c31d..694b17a 100644 --- a/tests/test_golden_log.py +++ b/tests/test_golden_log.py @@ -9,12 +9,18 @@ It relies on the generator being seeded, so the input data is fixed too. +There are two references, because a submission of mandatory variables alone +does not reach every check. litemp -- the data request's only x,y,z,t variable, +and so the only one whose sparse snapshot axis the time checks look at -- is not +mandatory, and neither is the static x,y refgeoid. The second reference covers +them. + To adopt a change in the log as the new expected output, inspect the diff the -failure prints, then regenerate the reference: +failure prints, then regenerate the references: ISSCHECKER_UPDATE_GOLDEN_LOG=1 pytest tests/test_golden_log.py -and commit the updated reference alongside the change that caused it. +and commit the updated references alongside the change that caused them. """ import difflib @@ -27,7 +33,14 @@ from isschecker import checker from isschecker import generate as generate_test_files -REFERENCE_LOG = Path(__file__).parent / "reference" / "compliance_checker_log.txt" +REFERENCE_DIR = Path(__file__).parent / "reference" + +# The submissions the references cover: what the generator writes, and where the +# expected log for it is stored. +GOLDEN_CASES = { + "mandatory": ("compliance_checker_log.txt", False), + "non_mandatory": ("compliance_checker_log_non_mandatory.txt", True), +} # Fixed so that the log is reproducible. GRID_NAME = "GrIS_16000m" @@ -57,10 +70,12 @@ def _mask_volatile(text: str, source_path: str) -> str: return text.replace(source_path, "") -@pytest.fixture(scope="module") -def generated_log(tmp_path_factory) -> str: - """Return the log of a checker run over a fixed synthetic dataset, masked.""" - output_root = tmp_path_factory.mktemp("golden_log_data") +@pytest.fixture(scope="module", params=sorted(GOLDEN_CASES)) +def golden_case(request, tmp_path_factory) -> tuple[Path, str]: + """The reference log for one submission, and the log this run produced.""" + reference_name, include_non_mandatory = GOLDEN_CASES[request.param] + + output_root = tmp_path_factory.mktemp(f"golden_log_data_{request.param}") created_files = generate_test_files.create_netcdf_file( None, grid_name=GRID_NAME, @@ -69,6 +84,7 @@ def generated_log(tmp_path_factory) -> str: nyears=NYEARS, include_scalars=True, include_xyt=True, + include_non_mandatory=include_non_mandatory, output_root=output_root, seed=SEED, ) @@ -80,21 +96,26 @@ def generated_log(tmp_path_factory) -> str: variable_list=VARIABLE_LIST, version=VERSION, ) - return _mask_volatile(summary["log_text"], str(core_dir)) + return ( + REFERENCE_DIR / reference_name, + _mask_volatile(summary["log_text"], str(core_dir)), + ) + +def test_log_matches_reference(golden_case): + reference_log, generated_log = golden_case -def test_log_matches_reference(generated_log): if os.environ.get("ISSCHECKER_UPDATE_GOLDEN_LOG"): - REFERENCE_LOG.parent.mkdir(parents=True, exist_ok=True) - REFERENCE_LOG.write_text(generated_log) - pytest.skip(f"Wrote a new reference log to {REFERENCE_LOG}") + reference_log.parent.mkdir(parents=True, exist_ok=True) + reference_log.write_text(generated_log) + pytest.skip(f"Wrote a new reference log to {reference_log}") - assert REFERENCE_LOG.exists(), ( - f"No reference log at {REFERENCE_LOG}. Create one with " + assert reference_log.exists(), ( + f"No reference log at {reference_log}. Create one with " "ISSCHECKER_UPDATE_GOLDEN_LOG=1 pytest tests/test_golden_log.py" ) - expected = REFERENCE_LOG.read_text().splitlines(keepends=True) + expected = reference_log.read_text().splitlines(keepends=True) actual = generated_log.splitlines(keepends=True) if actual != expected: diff --git a/tests/test_reporter.py b/tests/test_reporter.py new file mode 100644 index 0000000..02af2da --- /dev/null +++ b/tests/test_reporter.py @@ -0,0 +1,171 @@ +"""Direct tests of the Reporter, which the rest of the suite only exercises. + +Every other test reaches the reporter through a checker run, so what it writes +and what it counts are only ever observed together and only in the combinations +the fixtures happen to produce. These pin down the pieces: the line formats, the +split between severities and categories, and the roll-up from a file's sub-total +to the run's total, which is the property the per-file footer and the synthesis +block both rest on. +""" + +import io + +import pytest + +from isschecker.checker import Reporter + + +@pytest.fixture +def log(): + return io.StringIO() + + +def test_error_and_warning_line_formats(log): + reporter = Reporter(log).category("naming") + + reporter.error("something is wrong.") + reporter.warning("something is unusual.") + reporter.ok("Something checked out: OK") + reporter.note("Something else: N/A") + + assert log.getvalue() == ( + " - ERROR: something is wrong.\n" + " - WARNING: something is unusual.\n" + " - Something checked out: OK\n" + " - Something else: N/A\n" + ) + + +def test_qualifier_labels_both_severities(log): + reporter = Reporter(log).category("attr", qualifier="attributes") + + reporter.error("global attribute 'crs' is missing.") + reporter.warning("coordinate 'time' on-disk dtype is float64.") + + assert log.getvalue() == ( + " - ERROR (attributes): global attribute 'crs' is missing.\n" + " - WARNING (attributes): coordinate 'time' on-disk dtype is float64.\n" + ) + + +def test_bullet_is_configurable_for_experiment_level_findings(log): + """Experiment-level findings are written flush left, not as list items.""" + reporter = Reporter(log).category("file", bullet="") + + reporter.error("In experiment historical, ... is (are) missing: ['lim']") + + assert log.getvalue() == ( + "ERROR: In experiment historical, ... is (are) missing: ['lim']\n" + ) + + +def test_ok_and_note_are_not_counted(log): + reporter = Reporter(log) + naming = reporter.category("naming") + + naming.ok("fine: OK") + naming.note("not checked.") + + assert reporter.total_errors == 0 + assert reporter.total_warnings == 0 + + +def test_warnings_are_counted_apart_from_errors(log): + reporter = Reporter(log) + naming = reporter.category("naming") + + naming.error("wrong.") + naming.warning("unusual.") + naming.warning("also unusual.") + + assert reporter.total_errors == 1 + assert reporter.total_warnings == 2 + assert reporter.error_count("naming") == 1 + assert reporter.warning_count("naming") == 2 + + +def test_counts_are_kept_per_category(log): + reporter = Reporter(log) + + reporter.category("time").error("wrong axis.") + reporter.category("num").error("out of range.") + reporter.category("num").error("also out of range.") + reporter.category("attr").warning("unusual dtype.") + + assert reporter.error_count("time") == 1 + assert reporter.error_count("num") == 2 + assert reporter.error_count("attr") == 0 + assert reporter.warning_count("attr") == 1 + assert reporter.total_errors == 3 + assert reporter.total_warnings == 1 + + +def test_a_count_reports_one_line_as_several_findings(log): + """Four missing mandatory variables are one line and four errors.""" + reporter = Reporter(log) + + reporter.category("file", bullet="").error( + "these mandatory variable(s) is (are) missing: ['a', 'b', 'c', 'd']", + count=4, + ) + + assert log.getvalue().count("\n") == 1 + assert reporter.error_count("file") == 4 + + +def test_child_sub_totals_roll_up_into_the_parent(log): + """What a file's footer counts and what the synthesis counts are one thing.""" + run = Reporter(log) + + first_file = run.child() + first_file.category("time").error("wrong axis.") + first_file.category("attr").warning("unusual dtype.") + + second_file = run.child() + second_file.category("num").error("out of range.") + + assert first_file.total_errors == 1 + assert first_file.total_warnings == 1 + assert second_file.total_errors == 1 + assert second_file.total_warnings == 0 + + assert run.total_errors == 2 + assert run.total_warnings == 1 + assert run.error_count("time") == 1 + assert run.error_count("num") == 1 + assert run.warning_count("attr") == 1 + + +def test_a_child_of_a_child_reaches_the_top(log): + """The run, the experiment and the file are three levels, not two.""" + run = Reporter(log) + experiment = run.child() + file = experiment.child() + + file.category("spatial").error("bad corner.") + + assert file.total_errors == 1 + assert experiment.total_errors == 1 + assert run.total_errors == 1 + + +def test_children_do_not_see_each_others_counts(log): + run = Reporter(log) + first = run.child() + second = run.child() + + first.category("naming").error("wrong name.") + + assert second.total_errors == 0 + + +def test_all_reporters_write_to_the_same_log(log): + run = Reporter(log) + + run.child().category("naming").error("first.") + run.child().category("time").warning("second.") + + assert log.getvalue() == ( + " - ERROR: first.\n" + " - WARNING: second.\n" + )