Repository navigation
Add the cs_nshm_2022 ingest script - #7
Merged
Merged
Conversation
Adapted from nz_sim_validation_ingest.py for run3's intensity_measures.h5 layout (one netCDF4 group per IM, one dataset per component), trimmed to geom and rotd50, 25 pSA periods and no FAS or GMM records. Magnitudes stay BoldM, the group convention, rather than converting to Mw. The events table takes the NSHM catalogue magnitude from a CSV exported from nshmdb. The build stops on inconsistent input: NaNs, site values that differ between ruptures, distances that differ between realisations, or a magnitude or hypocentre that disagrees with the h5. It writes a .partial file and renames it only after validate(), exact row counts and a 1,000-record spot check against the h5 all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A manifest now lists every realisation with its rel_id, pilot flag and
files, so the script knows neither campaign's layout. An event takes its
geometry from its first main realisation and its domain from the union of
all of them. site_event keeps the main campaign's distances wherever it
has them and the pilot's elsewhere, flagged {"fault_geometry": "pilot"}.
Each hypocentre is placed on its own realisation's geometry, which differs
for the pilot. db_meta now speaks to researchers, not to our run layout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UgVr1htJMAqx8tABJLTnD5
The default build stays lean (geom and rotd50, 25 periods, no FAS), but a full IMDB can now be built from the same script. eas carries FAS only, so its records go in their own batch with no pSA or scalar rows, and rotd records get no FAS row. Every realisation's frequency grid must match the first one's, as must the period grid under --psa-periods all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UgVr1htJMAqx8tABJLTnD5
- Release ibis's in-memory tables after each event. ibis 12 registers every inserted DataFrame with DuckDB and frees them only at exit, so the whole build stayed in memory outside DuckDB's memory_limit. That was ~17 GiB for the lean build, and enough to OOM any full-content build. - Refuse a realisation.json that describes another event, and a manifest whose pilot flags do not follow its campaigns: either would otherwise build a wrong database silently. - Check site_event against the (site, event) pairs that have records, and optionally the flagged count (--expect-flagged). - Find missing files and an empty manifest before building, and name the realisation in every per-realisation error. - Refuse --fas without a FAS component, and store no periods for eas alone. - Correct db_meta: pilot magnitudes are not drawn, pilot-only events are unflagged, the pilot's bottom edges moved up to 2.6 km (not 1.8), and station coordinates need no internal file names. Add a clean source and drop nshm_rupture_name, which only ever held "Rupture <id>". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UgVr1htJMAqx8tABJLTnD5
The pilot's geometry comes from an earlier NSHM database release, so in the 75 events the main campaign also simulated, events.source_wkt is not the geometry the pilot's hypocentre and flagged distances sit on. Every pilot realisation now carries realisations.metadata.fault_geometry_wkt: a MULTIPOLYGON Z with one polygon per plane, (lon, lat, depth km), straight from its realisation.json. It has depth, so distances can be recomputed from it alone. About 2 MB over all 293 pilot realisations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UgVr1htJMAqx8tABJLTnD5
The script is meant to sit beside the other ingests in scripts/ as a guide, so it now reads top down: - The module docstring lists the inputs, what is specific to cs_nshm_2022, and a map of the file's six sections. - Every part that exists only because of cs_nshm_2022 (the two campaigns, the pilot's distances, the manifest, grid-snapped stations, the NSHM CSV, the Slurm job's provenance) is marked "# cs_nshm_2022:", so a grep lists what another dataset can drop. - build_database is split into read_event, build_event_rows, write_event and verify_database, and all the checks sit together in section 5. Behaviour is unchanged. Apart from build_database and check_manifest, which now reuses group_by_event, every definition parses to the same code as before, and the 56 tests pass. Building the same three-realisation manifest with this commit and with d39293c, both with the default content and with every component, all periods and FAS, gives identical tables; only db_meta.created_at differs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fo9vmUtWnieoTVWuEaYVii
The module docstring pointed at the cs_nshm_2022 container, which imdb's contributors do not have. The uv command it now gives passes all 56 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fo9vmUtWnieoTVWuEaYVii
Every function, method and class now has a numpydoc docstring with the sections imdb's numpydoc_validation checks ask for: Parameters, Returns and, where a function raises its own errors, Raises. Dataclass fields are documented in Attributes sections instead of attribute docstrings, and the module docstring's guide sits under Notes, since numpydoc rejects custom section headings. `numpydoc lint` now reports nothing (it reported 77 findings). Only docstrings changed: every definition parses to the same code with docstrings stripped, every comment is unchanged, and the 56 tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fo9vmUtWnieoTVWuEaYVii
Brings in #6, which lets databases of schema version 1 be opened read-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fo9vmUtWnieoTVWuEaYVii
AndrewRidden-Harper
marked this pull request as ready for review
September 23, 2026 23:53
AndrewRidden-Harper
requested review from
claudio525,
joelridden and
lispandfound
September 23, 2026 23:53
lispandfound
approved these changes
Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/cs_nshm_2022_ingest.pybuilds the IMDB for cs_nshm_2022: physics-based simulations of 550 NSHM 2022 crustal ruptures (664 realisations at 104,892 sites), run with the ucgmsim workflow on BSC's MareNostrum 5.scripts/tests/test_cs_nshm_2022_ingest.pytests it on a small synthetic dataset.Reading it as a guide
It is longer than the other ingest scripts: two simulation campaigns become one dataset, and it ran unattended as a Slurm job, so it checks its own output before publishing it. To keep it navigable:
# cs_nshm_2022:, sogrep -n "cs_nshm_2022:" scripts/cs_nshm_2022_ingest.pylists what another dataset can drop.numpydoc lintreports nothing against this repo'snumpydoc_validationchecks.Possibly worth copying: the
Contentoptions (pSA periods, FAS, components),spot_check(re-reads random records from the source files and compares them exactly), and writing to<out>.partialand renaming only once every check passes.For the library, separately
add_*methods registered with DuckDB until the process exits, outside DuckDB'smemory_limit. The script works around it (release_memtables); releasing them inside imdb would fix it for every ingest.fault_geometries,initial_fault_name,source_wktandtrace_wktare repeated in each ingest script that reads realisation.json and could live in imdb.Testing
The tests skip in CI, which installs neither h5py nor source-modelling. To run them:
With main merged in (#6), the ingest tests and the library's own tests all pass (95).
Commit d39293c built the database now on Dropbox,
QuakeCoRE/gmsim_scratch/cs_nshm_2022/imdb/cs_nshm_2022_ims_no_fas_limited_psa_20260923.duckdb(itsdb_meta.ingest_script_commit). The three later commits only reorganise, document and comment the script: rebuilding a three-realisation sample with the reorganised version gives identical tables, and the docstring commit changes no code.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fo9vmUtWnieoTVWuEaYVii