Skip to content

Skip IDS reads when no rule matches name and occurrence - #26

Open
chupakobra6 wants to merge 1 commit into
iterorganization:developfrom
chupakobra6:codex/skip-unmatched-ids
Open

chupakobra6 wants to merge 1 commit into
iterorganization:developfrom
chupakobra6:codex/skip-unmatched-ids

Conversation

@chupakobra6

Copy link
Copy Markdown
Contributor

Related to #10.

find_matching_rules() loads every populated IDS occurrence before checking whether any selected rule can use it. A rule targeting only core_profiles:3 therefore also reads unrelated data, and an unrelated load error can stop that validation.

Select candidates by IDS name and occurrence before calling DBEntry.get(). Keep version filtering after loading: with autoconvert=False, a stored IDS can have a different DD version from the entry's factory. Additional inputs to cross-IDS rules still load on demand, and wildcard rules retain their existing behavior. Document these loading boundaries.

Validation:

  • 18 regression cases cover exact read counts, empty/no-match selections, wildcard and occurrence matching, stored DD versions, cross-IDS dependencies, both load-error policies, and real netCDF validation with coverage on/off. On unchanged develop, 13 of these cases fail; all pass with this change.
  • Full suite: 203 passed, 2 existing skips on macOS/Python 3.12 with IMAS-Python 2.3.0, and on Python 3.10 with minimum IMAS-Python 2.2.0.
  • Integration run of both follow-ups with Speed up assert bookkeeping: cached node paths and lazy stacks #24 (22eb4cd): 287 passed, 3 expected skips on each of Linux arm64/Python 3.10–3.14.
  • Black, Flake8, MyPy, Isort, and Sphinx with warnings treated as errors pass.

The benefit is avoiding unnecessary reads in filtered workloads. Generic wildcard rules can still require every IDS; this does not accelerate reads that are actually needed or claim to resolve the complete performance problem in #10.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant