Skip to content

feat: use LUT for AEF dequantization - #10

Open
Geethen wants to merge 1 commit into
jakenotjay:mainfrom
Geethen:feat/lut-dequantize
Open

feat: use LUT for AEF dequantization#10
Geethen wants to merge 1 commit into
jakenotjay:mainfrom
Geethen:feat/lut-dequantize

Conversation

@Geethen

@Geethen Geethen commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Problem

dequantize_aef recomputed the same int8-to-float32 formula for every pixel value. AEF stores only 256 possible raw values, so large materialized or dask-backed windows spend avoidable time repeating (v / 127.5) ** 2 * sign(v) and applying a separate nodata mask.

Fix

This adds a 256-entry lookup table and maps raw values through it with lut[raw + 128]. The -128 -> NaN nodata mapping is folded into the table.

The plain ndarray path uses the LUT directly. The xr.DataArray path uses the same LUT through xr.apply_ufunc(..., dask="parallelized"), so dask-backed arrays stay lazy and apply the lookup per block. Float chunks with existing NaN gap-fill are handled by preserving NaNs and mapping finite raw int8-valued floats through the LUT.

In local checks on a 2048 x 2048 x 64 materialized AEF window, this changed numpy dequantization from about 2702 ms to 1582 ms, while preserving identical output.

Tests

Adds tests/test_utils.py coverage asserting:

  • the LUT path matches the reference formula for all 256 int8 values, for both int8 and int16 inputs
  • dask-backed DataArrays stay lazy, preserve chunks, and compute to the same values as the eager LUT path
  • float chunks with existing NaN gap-fill preserve NaNs while dequantizing finite raw values

Ran locally:

uv run pytest tests/test_utils.py
uv run ruff check aef_loader/utils.py tests/test_utils.py
uv run ty check

@jakenotjay jakenotjay left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change itself lgtm, but I'd like it applied to the lazy path too for dask, at scale this should improve speed.

Comment thread aef_loader/utils.py Outdated
Comment on lines +91 to +94
# Keep the DataArray path elementwise so dask-backed arrays stay lazy.
nodata_mask = data == nodata_value
normalized = data.astype(np.float32) / divisor
dequantized = (normalized**2) * np.sign(data)

@jakenotjay jakenotjay Aug 6, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this change, but for dask we're still applying the old method. The comment is misleading, we should just apply it via apply_ufunc! Worth benchmarking over a large area with dask, because if its a significant speedup, we'll see a performance improvement on dequantising at large scale.

@Geethen
Geethen force-pushed the feat/lut-dequantize branch from 29421cc to 49d5cf2 Compare August 6, 2026 10:43
@Geethen

Geethen commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, agreed. I updated this so the xr.DataArray path also goes through the LUT via xr.apply_ufunc(..., dask="parallelized"), which keeps dask-backed arrays lazy while applying the same lookup per block. I also added a dask-backed DataArray regression that checks chunk preservation and computed values.

Rebased on current main and ran locally:

uv run pytest tests/test_utils.py
uv run ruff check aef_loader/utils.py tests/test_utils.py
uv sync --group benchmark && uv run ty check

@Geethen Geethen changed the title feat: use LUT for numpy AEF dequantization feat: use LUT for AEF dequantization Aug 6, 2026
@Geethen
Geethen force-pushed the feat/lut-dequantize branch from 49d5cf2 to 65d324a Compare August 6, 2026 10:50
@Geethen

Geethen commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

I also ran a live dask-backed source.coop check against public 2018 AEF data. The dequantized dask result stayed lazy before compute and matched an explicit reference-formula calculation from the same raw window exactly (equal_nan=True, max_abs_diff=0.0).

One useful thing this exposed: with multiple tiles, current combine_by_coords(join="outer") can still promote raw int8 data to float because of NaN gap-fill. I hardened the LUT helper for that case too: float chunks preserve existing NaNs and finite raw int8-valued floats are mapped through the LUT. Added a dask regression for that path as well.

Validation after that hardening:

uv run pytest tests/test_utils.py
uv run ruff check aef_loader/utils.py tests/test_utils.py
uv run ty check

@Geethen
Geethen force-pushed the feat/lut-dequantize branch from 65d324a to 2fc8ea9 Compare August 6, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants