3395 hi l1c look up geometric factor by gain state via extended cal prod schema by tmplummer · Pull Request #3398 · IMAP-Science-Operations-Center/imap_processing

tmplummer · 2026-08-19T17:06:51Z

Change Summary

Overview

Adds a gain-state-aware geometric factor lookup to Hi L1C processing, replacing the previous placeholder/stubbed value. A pointing's detector high-voltage deltas, computed and recorded at L1B, are matched against an extended cal-prod ancillary schema at L1C to determine the correct geometric factor per (esa_energy_step, calibration_prod).

Closes: #3395

File changes

imap_processing/hi/utils.py — Extends CalibrationProductConfig with a leading gain_config_id index level and forward-filled gain-match columns. Centralizes all gain-state logic as classmethods: compute_gain_match_values() (raw voltages → field-keyed HV
deltas) and match_gain_config_id() (matches a pointing's HV deltas to a gain_config_id row, returning None on NaN input, no match, or ambiguous match).
imap_processing/hi/hi_l1b.py (de_gain_test_filter) — Computes the pointing's HV deltas via CalibrationProductConfig.compute_gain_match_values() and stores them as L1B DE global attributes, named directly by field (e.g. "mcp_delta_v", "tof_v") with
CalibrationProductConfig.GAIN_MATCH_FIELDS as the single source of truth for field names.
imap_processing/hi/hi_l1c.py — Adds add_pset_geometric_factor(), which reads those HV delta attrs back from the L1B DE product, matches them to a gain_config_id, and populates the PSET's geometric_factor variable in place (pset_ds =
add_pset_geometric_factor(pset_ds, l1b_de_ds, config_df)).
imap_processing/hi/hi_l2.py — Selects a single gain_config_id before converting the cal-prod DataFrame to xarray, to account for the new index level.
CDF attrs (imap_hi_variable_attrs.yaml) — hi_pset_geometric_factor gains a calibration_prod dimension (DEPEND_2/LABL_PTR_2); the variable is no longer summed across calibration products.

Testing

230 tests pass across test_hi_l1b.py, test_hi_l1c.py, test_utils.py, and test_hi_l2.py.
New/updated coverage includes: gain_config_id matching edge cases (exact match, no match, ambiguous match, NaN input), compute_gain_match_values() correctness, and confirmation that add_pset_geometric_factor() does not pick up the pset's unrelated spin_angle_bin dimension.
Not yet consumed by L2 map-combination across differing gain states — left as a follow-on, not covered by new tests.

…schema Replace the separate gain-configuration ancillary file and config_id classification (added in IMAP-Science-Operations-Center#3358, retired in IMAP-Science-Operations-Center#3394) with a gain-state- dependent geometric factor lookup folded directly into the existing imap_hi_{sensor}-cal-prod_*.csv file, keyed on measured detector voltage differences (front/back) and the U-Can voltage, per Paul Janzen's feedback (IMAP-Science-Operations-Center#3391). - hi/utils.py: extend CalibrationProductConfig with a new leading gain_config_id index level, GAIN_MATCH_FIELDS, gain_match_columns (forward-filled per gain_config_id group, placed as the final columns of the file), an extended _validate(), and match_gain_config_id(). - Ancillary CSV (test fixture): add gain_config_id and mcp_delta_v/cem_a_delta_v/cem_b_delta_v/tof_v (+ _tol) columns. Deltas are computed as back-voltage minus front-voltage (positive), matching real flight detector voltages. - hi/hi_l1c.py: implement pset_geometric_factor() to match the L1B DE product's gain_match_{field} attributes (from IMAP-Science-Operations-Center#3394) against CalibrationProductConfig.match_gain_config_id(), recording the geometric factor per (esa_energy_step, calibration_prod) pair -- the PSET geometric_factor variable gains a calibration_prod dimension instead of summing across calibration products. Fix pset_counts()'s config_row.Index positional lookup, which shifts because of the new leading index level. - hi/hi_l2.py: select a single gain_config_id before the CalibrationProductConfig DataFrame is converted to xarray and reindexed against the map dataset (also a consequence of the new index level). - imap_hi_variable_attrs.yaml: add DEPEND_2/LABL_PTR_2 (calibration_prod) to hi_pset_geometric_factor. Not yet consumed by L2 map-combination processing across different gain states -- that remains a follow-on. Part of IMAP-Science-Operations-Center#3391. Closes IMAP-Science-Operations-Center#3395. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…nified location

refactor pset_geometric_factor function

…ataset

Copilot

Pull request overview

Adds gain-state-aware geometric-factor lookup across Hi L1B/L1C processing.

Changes:

Extends calibration-product configuration with gain-state matching.
Records detector voltage deltas and populates PSET geometric factors.
Updates L2 compatibility, CDF dimensions, fixtures, and tests.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`imap_processing/hi/utils.py`	Adds gain matching and schema validation.
`imap_processing/hi/hi_l1b.py`	Records pointing HV deltas.
`imap_processing/hi/hi_l1c.py`	Selects geometric factors by gain state.
`imap_processing/hi/hi_l2.py`	Handles the additional configuration index.
`imap_processing/cdf/config/imap_hi_variable_attrs.yaml`	Adds the calibration-product dimension.
`imap_processing/tests/hi/test_utils.py`	Tests configuration parsing and matching.
`imap_processing/tests/hi/test_hi_l1b.py`	Updates gain-filter tests.
`imap_processing/tests/hi/test_hi_l1c.py`	Tests geometric-factor population.
`imap_processing/tests/hi/data/l1/imap_hi_90sensor-cal-prod_20240101_v001.csv`	Extends the ancillary fixture schema.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        "pos_defl": 1500.0,
+        "neg_defl": 1500.0,


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…n cal prod dataframe

leowerneck · 2026-08-20T14:52:15Z

+        geometric_factor_var["geometric_factor"].values[0] = gain_factor_da.transpose(
+            "esa_energy_step", "calibration_prod"
+        ).values
+    return pset_ds.update(geometric_factor_var)


This actually causes the function to always return None in newer versions of xarray, since pset_ds.update() was changed to update in place. Suggested fix:

Suggested change

return pset_ds.update(geometric_factor_var)

pset_ds.update(geometric_factor_var)

return pset_ds

This fixes 5 failing tests (locally, xarray 2025.12.0) related to this PR . To be more precise,

uv run --with 'xarray==2025.12.0' pytest -n auto

results in 5 failures related to the PR, whereas

uv run --with 'xarray==2025.04.0' pytest -n auto

results in no failures related to the PR. The PR-related failures go away with the proposed fix (though I am still catching a couple locally).

tmplummer and others added 4 commits August 18, 2026 15:56

Move definition of gain match formulas and gain match fields to one u…

126784e

…nified location

Use match gain config in L1C

1bff921

refactor pset_geometric_factor function

Unify the way that the gain voltages are computed and stored in the d…

ee58d00

…ataset

tmplummer requested review from tech3371 and a balanced review from Copilot August 19, 2026 17:07

tmplummer self-assigned this Aug 19, 2026

tmplummer added this to IMAP Aug 19, 2026

Copilot started reviewing on behalf of tmplummer August 19, 2026 17:07 View session

tmplummer requested review from jaredclaypoole and leowerneck August 19, 2026 17:08

Copilot AI reviewed Aug 19, 2026

View reviewed changes

tmplummer and others added 3 commits August 19, 2026 11:25

Apply suggestions from code review

ae43cc7

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Address PR feedback regarding bug with multiple gain configurations i…

602baf9

…n cal prod dataframe

Merge remote branch

48a52fc

leowerneck reviewed Aug 20, 2026

View reviewed changes

Navigation Menu

Uh oh!

3395 hi l1c look up geometric factor by gain state via extended cal prod schema - #3398

3395 hi l1c look up geometric factor by gain state via extended cal prod schema#3398
tmplummer wants to merge 7 commits into
IMAP-Science-Operations-Center:devfrom
tmplummer:3395-hi-l1c-look-up-geometric-factor-by-gain-state-via-extended-cal-prod-schema

tmplummer commented Aug 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leowerneck Aug 20, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	return pset_ds.update(geometric_factor_var)
	pset_ds.update(geometric_factor_var)
	return pset_ds

Uh oh!

Conversation

tmplummer commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

Overview

File changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leowerneck Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tmplummer commented Aug 19, 2026 •

edited

Loading

leowerneck Aug 20, 2026 •

edited

Loading