Repository navigation
Conversation
Tasks that do not log turned spdlog off entirely, so a critical message raised only on such a task, such as a NaN found by state validation on the task that owns the cell, was lost along with the reason for the abort that followed. These tasks now keep critical messages only, on stderr and labeled with the task. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
State validation gave only the number of NaN or out-of-range entries in each field, which left no way to find them in a large mesh without rerunning. Each critical message now also gives the global cell ID, latitude, longitude and layer of the first such entry, and the value when it is out of range. checkOceanState can return the same issues to the caller, which the unit test uses to check the reported location. The first entry is searched for only when a count is nonzero, so a valid state costs no more than before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each task decided on its own whether to abort, so only the tasks that found a NaN called MPI_Abort and no collective work was possible first. validateOceanState now sums the NaN and out-of-range counts over all tasks, in the reduction it already did for negative salinity, so every task makes the same decision. Before aborting, every task writes the stream named by the new option State::Validation::WriteStreamOnAbort (default History), giving a record of the bad state to inspect. The time steppers pass their clock for the time stamp, which is the start of the failing step. The master task now logs the totals over all tasks, both before an abort and when the run continues. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The user's guide gains a section on state validation, its options and what it reports, and says where tasks that do not log write critical messages. The design doc covers the location of the first bad value, the collective abort and the stream written before it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xylar
added
enhancement
Author
TestingCTests pass on Chrysalis with Intel and Open MPI (61/61). I also applied these commits on top of Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When state validation aborts, the run now says what failed and where, and leaves the bad state behind to inspect. Closes #647.
State: Validation: WriteStreamOnAbort(defaultHistory).For reviewers:
AbortOnOOB: false) are printed by each affected rank every step.Model answers should not change.
Checklist
Linting
Building
Testing
chrysalis, intel, openmpi
Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.