Repository navigation
Landsat 8/9 OLI Collection-2 Level-1 preprocessor (M1), with tests - #43
SpectraWorksVision wants to merge 1 commit into
Conversation
f53d740 to
65bde66
Compare
|
Thanks for this PR! A heads-up: I've moved further experimental code out of the v2 development branch, which meant rewriting the history of Could you rebase it onto the current git remote add upstream https://github.com/MarcYin/SIAC.git # if not already set
git fetch upstream feat/refactor
git rebase --onto upstream/feat/refactor ad2cdf2 feature/landsat-c2-l1
git push --force-with-lease origin feature/landsat-c2-l1Sorry for the extra step, and thanks again! Generated by Claude Code |
… synthetic end-to-end tests
Adds the M1 stage for Landsat so an unpacked USGS Collection-2 Level-1
scene can enter the staged pipeline through SIAC.process / process_scene.
- adapters/satellite/landsat.py: LandsatPreprocessor, registered as
"l8", "l9" and "landsat". Platform comes from MTL SPACECRAFT_ID.
- MTL.txt (ODL) and MTL.json parsing; per-band file lookup by MTL
name with a suffix fallback.
- TOA reflectance: (REFLECTANCE_MULT * DN + REFLECTANCE_ADD) divided by
per-pixel cos(SZA) from the C2 SZA angle band; scene-centre
SUN_ELEVATION only when the angle band is absent. DN 0 is fill (NaN).
- Geometry: SZA/SAA/VZA/VAA angle bands (int16, 0.01 deg) block-averaged
to a coarse grid (default 100 px = 3 km), circular mean for azimuths,
footprint from QA_PIXEL so outside-scene zeros never enter a mean.
- Cloud classes: configured provider as for S2, plus
cloud_mask.provider = "product_qa" mapping C2 QA_PIXEL bits
(fill -> missing, cloud/dilated/cirrus -> cloud, shadow -> shadow).
- Rejects Landsat 4-7 and Level-2 products explicitly.
- detect_sensor: tells L8 from L9, accepts MTL.json and an MTL file path.
- SensorName gains l8/l9; CloudMaskProvider gains product_qa.
- Landsat catalog: nominal L1 TOA rescaling 2e-5 / -0.1 (2.75e-5 / -0.2
is the Level-2 surface-reflectance scale).
- landsat.py added to typecheck-scoped.
Tests:
- tests/unit/test_landsat_preprocessor.py: synthetic C2 L1 scene (MTL,
bands, angle bands, QA_PIXEL); rescaling and per-pixel SZA, fill,
geometry block means incl. azimuth wrap, QA class mapping, M1 contract,
L9 resolving to the L8 OLI emulator weights.
- tests/integration/test_e2e_synthetic_landsat.py: catalog OLI config
(B1-B7, 30 m) through M1-M6 for L8 and L9, mirroring
test_e2e_synthetic.py, plus a seam test from a synthetic C2 scene
through LandsatPreprocessor and build_execution_plan. No generic-stage
change was needed.
- test_config: l8/l9 accepted, an unsupported sensor still rejected.
Not yet covered: S2-named defaults (surface_driven_solve_bands, M4 PSF
10 m and B11/B12 reference, RGB quicklook), Landsat search/download,
QA_RADSAT saturation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
70b3570 to
6f4028e
Compare
|
A real-scene run of this branch: Landsat 9 over Railroad Valley on 16 June 2025, compared with same-day RadCalNet over its 1 km site, with the RadCalNet spectra convolved with the OLI-2 response. Differences against RadCalNet, in reflectance:
The L1 TOA already carries the band 1-4 and band 7 offsets, so on this scene they are there before atmospheric correction. One scene, so not an accuracy figure. Case folder with inputs, per-band values and run identity: https://github.com/SpectraWorksOSS/SIAC/tree/validation/cases/validation/cases/landsat9_rvus_2025-06-16 |
Opened as a draft for discussion: Looks like this collided with your own v2 rebuild progress, so it may need reshaping to fit your plans. I see you’ve just started on Landsat on the labels side; if this overlaps with your own reader, take what helps and close the rest.
What it does
It lets an unpacked USGS Collection-2 Level-1 scene enter the staged pipeline through SIAC.process / process_scene. This is the preprocessor half of the Landsat M1 step in docs/EARTHACCESS_PLAN.md; the search/download source is not included.
adapters/satellite/landsat.py: LandsatPreprocessor, registered as l8, l9 and landsat. The platform comes from the MTL SPACECRAFT_ID.
Reads MTL.txt (ODL) and MTL.json, and finds band files by MTL name with a suffix fallback.
TOA reflectance: REFLECTANCE_MULT * DN + REFLECTANCE_ADD, divided by the per-pixel cos(SZA) from the C2 SZA angle band. It falls back to the scene-centre SUN_ELEVATION when the angle band is absent. DN 0 is treated as fill.
Geometry: the SZA/SAA/VZA/VAA angle bands (int16, 0.01°) are block-averaged to a coarse grid (default 3 km), using a circular mean for azimuths, with the footprint from QA_PIXEL so out-of-scene zeros never enter a mean.
Cloud classes come from the configured provider, as for S2, or from the new cloud_mask.provider = "product_qa", which maps C2 QA_PIXEL bits (fill to missing; cloud, dilated cloud and cirrus to cloud; shadow to shadow).
Landsat 4-7 and Level-2 products are rejected explicitly.
detect_sensor: now tells L8 from L9, and accepts MTL.json or an MTL file path.
Config: SensorName gains l8/l9; CloudMaskProvider gains product_qa.
Catalog fix: the OLI default_ref_scale/default_ref_offset were 2.75e-5 / -0.2, which is the Level-2 surface-reflectance scale. They are now the Level-1 TOA values, 2e-5 / -0.1. The preprocessor reads the per-band values from the MTL anyway.
No generic stage needed changing.
Tests
Not yet covered
The defaults that still assume Sentinel-2 bands (surface_driven_solve_bands, the M4 PSF 10 m target and B11/B12 reference, the RGB quicklook), Landsat search and download, and QA_RADSAT saturation masking.