[testable] Full-sensor plate solving + cached optical calibration - #557
Draft
mrosseel wants to merge 414 commits into
Draft
[testable] Full-sensor plate solving + cached optical calibration#557mrosseel wants to merge 414 commits into
mrosseel wants to merge 414 commits into
Conversation
…cceed) Brickbots/PiFinder runs the workflow on pull_request:synchronize from PR brickbots#379 (mrosseel:nixos -> brickbots:main). With ATTIC_TOKEN now also set on brickbots, build-emulated's 'Push to Attic' step succeeds — the last failing step. But stamp-build then tries to checkout the PR head ref (mrosseel:nixos) and 'git push' a pifinder-build.json commit there, which can't work from brickbots' Actions runner (no write access to the fork). The PR run therefore failed at the stamp step even after attic was wired correctly. Gating stamp-build on github.event_name == 'push' keeps the canonical stamp on the mrosseel:nixos push run (where it works) and skips it on brickbots PR runs (which only need to verify the build). Net effect: both repos' CI runs in parallel without stomping — - Both build and push the same closure to cache.pifinder.eu (attic FastCDC-dedups, so the second push is a no-op), - Only mrosseel stamps pifinder-build.json, - build-migration-tarball already gates on github.ref == refs/heads/nixos so it only runs on mrosseel push. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sync the fork with upstream up to 0e0ec03 (39 commits since the last sync at e4b623a): typed positioning model (PointingEstimate / ImuSample / AlignedResult), polar alignment, comet vectorization, single-instance lock, resolution-flexible UI, i18n fonts. Conflict resolution (18 files): - Adopt upstream's typed positioning model across solver, integrator, state, imu_pi, status, main, base, console, object_details, fonts, menu_structure, auto_exposure, camera_interface. - Keep NixOS layers: software.py (store-path upgrade UI), utils.py (build_json, writable comet_file, robust pifinder_dir), sys_utils.py. - plot.py: restore top-level `import pandas` (upstream added module-level uses; the fork had made it lazy) and drop the redundant lazy imports. - solver.py taken verbatim from upstream; the cedar-detect dev-spawn is proposed upstream separately (brickbots#478). - Drop fork-deleted nox/pip tooling (requirements.txt, version.txt). deps: add xlrd to python-packages.nix (pyerfa already present). Verification: ruff clean; 479 unit tests pass. Remaining failures are not from this merge -- 6 test_software + 4 test_t9_search pre-exist on origin/nixos; 5 test_comets fail on skyfield 1.53 (upstream vectorization, has a runtime fallback). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skyfield's propagate() lays a batched Kepler orbit out as (3, #orbits, #times) but sets output_shape = (3,) + t1.shape, so a batched orbit propagated to a single scalar time raises "cannot reshape array of size 3N into shape (3,)" on skyfield >= 1.46. (The fork uses nixpkgs' skyfield 1.53; upstream pins 1.45, which tolerated it.) Give every comet the same target time as an (N, 1) column so output_shape matches the (3, N, 1) result, then squeeze the time axis. Verified against the per-comet path (0 AU difference) and the existing tests/test_comets.py oracle (7/7 pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…che) The upstream merge auto-merged catalogs.py entirely to the fork's side, silently dropping upstream's T9 search (brickbots#464: KEYPAD_DIGIT_TO_CHARS, search_by_t9) and the catalog disk cache; test_t9_search (an upstream test) failed as a result. Take upstream's catalogs.py wholesale, matching the upstream main.py and menu_structure the merge already adopted. Trade- off: drops the fork's priority-fast-path / background-loader in favour of upstream's cache + loader (the agreed "take upstream catalogs" choice). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MIN_NIXOS_VERSION was intentionally bumped 2.5.0 -> 3.0.0 (d705057, prep for 3.0), but test_software still asserted 2.5.x/2.6.x as qualifying. Shift the qualifying mock releases and _meets_min_version cases into the 3.x line so the suite matches the current minimum; below-min (2.4.0) and draft cases are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
main.py created (and server.py switched) the pifinder_logconf.json symlink relative to the cwd, which on NixOS is the read-only /nix/store -> the service crash-looped at startup (OSError: Read-only file system). Follow the same split config.json already uses: the logconf presets stay read-only in the source tree (utils.pifinder_dir/python/logconf_*.json) and the active selection is persisted as a bare filename in the writable data dir (PiFinder_data/log_config), resolved via utils.active_logconf_path(). Storing the name (not a store-path symlink) keeps the choice valid across upgrades (which GC old store paths) and reboots. No NixOS workaround needed; also removes the write-to-source-tree antipattern on Raspberry Pi OS (upstreamable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The upstream merge kept nixos's outer tetra3_dir (python/PiFinder/tetra3)
while taking upstream's solver.py, which built the DB path as
tetra3_dir/data/default_database.npz. But the submodule nests the package
at tetra3/tetra3, so the DB is actually at tetra3/tetra3/data -> the solver
crashed at startup with FileNotFoundError.
Load it by its canonical name, Tetra3("default_database"); tetra3 resolves
the bundled DB from its own package data dir regardless of the inner/outer
layout. Validated on-device (loads from python/tetra3/data).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
release.yml now installs Nix + Attic like build.yml and pushes the release closure to a dedicated, never-GC'd `pifinder-release` cache. Devices and the migration first-boot trust it ahead of the dev `pifinder` cache and cache.nixos.org. Removes the last Cachix usage from active config. Docs (RELEASE.md, ADR 0004, NIXOS_STATUS.md) document the dev-vs-retained-release two-cache split and the per-cache retention caveat. The pifinder-release trusted-public-key is a placeholder until the cache is bootstrapped server-side. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 6 .direnv/ nix-direnv cache files were tracked but regenerate on every direnv reload, so rebases baked divergent copies into the nixos stack. Gitignore + untrack stops that churn going forward. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…erivations Replace the 524-line nixos/pkgs/python-packages.nix (26 manually packaged PyPI deps, each with hand-chased hashes and build patches) with a uv-managed workspace realized into the Nix store via uv2nix. Changes: * Deps declared in python/pyproject.toml, pinned in python/uv.lock (117 pkgs) * nixos/pkgs/uv-python.nix builds the runtime/dev virtualenvs; the 5 native packages (python-libinput, python-prctl, python-pam, dbus-python, pygobject) keep their build patches as uv2nix overrides * flake.nix: add pyproject-nix/uv2nix/pyproject-build-systems inputs, thread via specialArgs, devShell uses the uv2nix devEnv * libcamera Python bindings stay a Nix overlay (not on PyPI) All four nixosConfigurations + the devShell evaluate; the aarch64 build is to be validated by CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cache DeterminateSystems/magic-nix-cache-action now returns HTTP 418 and the GitHub Actions cache rate-limits it (Twirp ResourceExhausted, "rate limit exceeded"), so `nix develop` cannot fetch the dev environment and every lint/test/type-check job fails before ruff/pytest/mypy even run — which all testable PRs inherit. Mirror build.yml/release.yml and substitute from the self-hosted Attic cache cache.pifinder.eu (ADR 0004) instead, falling back to cache.nixos.org when ATTIC_TOKEN is unavailable (e.g. fork PRs) so the job never hard-fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s_utils failures Changes: * networking.nix: pifinder-wifi-fallback service+timer brings up PiFinder-AP when no client connects within 45s (or when AP is forced via the UI), so a device with an unreachable saved network stays reachable and AP survives reboot. * sys_utils.py: persist chosen WiFi mode to PiFinder_data/wifi_mode for the fallback service to restore. * software.py: show a clear 'No internet - check WiFi' screen instead of an empty channel list when GitHub is unreachable. * utils.py: log why get_sys_utils() falls back to the no-op fake (previously swallowed) so a failed NM/dbus/pam import is diagnosable. * remove dead Raspberry Pi OS scripts switch-ap.sh/switch-cli.sh (dhcpcd/hostapd; inert on NixOS). Needs on-device validation of the AP fallback timing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the Pi5 self-hosted runner is slow or unavailable, the fallback now uses GitHub's free hosted arm64 runner (public repos) and builds natively: no QEMU binfmt, no --system aarch64-linux cross flag, and extra-platforms dropped. Timeout cut 360 -> 60 min. Job id kept as build-emulated so downstream needs: references are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t, evdev
These C-extension packages build from sdist in the uv2nix devEnv, but their
overrides did not declare a build backend. When the packages are not cached
(notably the x86_64 lint/test/type-check runners) the sdist build fails and
takes the whole devEnv down with it:
- dbus-python, pygobject: "No module named 'mesonpy'" -> add the meson-python
build system via resolveBuildSystem (alongside the existing pkg-config + C
libs).
- evdev: "No module named 'setuptools'" -> add the setuptools build system
(evdev had no override at all).
Mirrors the existing python-libinput / python-prctl overrides, which already
use resolveBuildSystem { setuptools = []; }.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ders, patches) Surfaced by the first real build. In uv-python.nix: * adafruit-blinka: ignore missing libgpiod.so.2 (vendored non-Pi SoC helpers) * dbus-python/pygobject/pycairo: add ninja for the meson-python build * pygobject: add pycairo for py3cairo.h * evdev: add linuxHeaders and repoint setup.py's header search off /usr/include * rpi-gpio/sh/spidev/pidng: provide the setuptools build backend * python-pam: patch the installed wheel module in postInstall (no src/ at patch time) pyproject.toml: empty [tool.setuptools] package set so the virtual root builds an empty wheel instead of tripping setuptools flat-layout discovery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…arch64) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(a) Configure cache.pifinder.eu/pifinder as a public substituter (public key, no token) in build-emulated and the lint/test/type-check jobs, so fork-PR runs (brickbots) pull from the cache instead of building from source. Gate attic login + push on ATTIC_TOKEN so tokenless runs are pull-only and don't fail at push. (b) Add always() to stamp-build so a skipped build-emulated (when build-native succeeds) no longer skips the stamp — green builds now update pifinder-build.json. Also add nixos/brickbots-attic-setup.md: how to give brickbots a push token. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- delete by NM connection UUID, not by id == ssid. The old match silently no-ops on any entry whose connection id differs from its SSID (e.g. the corrupt "0x20" entry whose SSID held byte-array text), so such networks could never be deleted from the UI. - add: persist via add_connection(save_to_disk=True) first, then make a best-effort activate. Saving no longer depends on being able to activate right now, which failed (and saved nothing) when wlan0 was busy in AP mode or the network was out of range. - get_wifi_networks: re-query NetworkManager live instead of returning a stale cache, so changes made outside the process (AP/CLI switch, repairs) show up without an app restart. - decode SSIDs with errors="replace" so a single undecodable SSID can't crash the whole network list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NetworkManager (like other security-sensitive plugin loaders) refuses to load any plugin file not owned by root. A store with non-root paths — uid 1000 baked into the migration tarball, or imported by single-user nix — makes NM drop its wifi device plugin entirely: wlan0 shows as "unmanaged", WIFI-HW as "missing", and no wifi client connection ever comes up. Add an idempotent boot oneshot (ordered before NetworkManager) that normalises /nix/store and the nix db dir back to root, remounting the store read-write only when a repair is actually needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The uv2nix env builds RPi.GPIO 0.7.1 from PyPI. Its C module init calls get_rpi_info(), which reads the board revision from /proc/device-tree/system/linux,revision or the /proc/cpuinfo "Revision" line. On a NixOS Pi 4 (arm64, mainline device tree) neither is present, so the import raises "This module can only be run on a Raspberry Pi!" — and since adafruit-blinka imports it via `board`, PiFinder crash-loops at startup. Add a postPatch that makes get_rpi_info() fall back to the always-present /proc/device-tree/model string, synthesising a Pi 4 Model B revision when it sees "Raspberry Pi". Detection then succeeds and the app starts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e / read-only) /nix/store is a read-only bind mount of the same device as /. The repair oneshot remounted it with "remount,rw"/"remount,ro", which operates on the shared superblock, so restoring ro flipped / (and /nix/var) read-only too — breaking the next pifinder-upgrade (nix build could not write temproots). Carry "bind" on both remounts so only this mount's per-mount ro flag changes and the rootfs superblock is left alone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wifi_mode() returned a value detected once in __init__. When the radio falls back to AP after startup, NetworkManager shows PiFinder-AP active but the UI still said "Client" (and local_ip() returned the wrong address). Detect the mode live from NetworkManager on each call, mirroring get_wifi_networks(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…owing them The upgrade piped `nix build` through a gawk progress filter that dropped nix's stderr, so a failed upgrade only reported "failed" with no cause. Tee the build output to /run/pifinder/upgrade-nix.log and, on failure, dump its tail to the journal so the actual error is visible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The placeholder "pifinder-release:REPLACE_WITH_PIFINDER_RELEASE_PUBKEY=" isn't valid base64, so nix aborts every operation with "invalid character in Base64 string" — bricking upgrades on the device. Remove it from both services.nix and migration.nix; the device trusts the working `pifinder` cache. The real release key is added only once that cache is provisioned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
python/result and python/result-lib are nix build output symlinks that were committed before .gitignore's result/result-* rule existed, so they stayed tracked and kept getting re-snapshotted into unrelated commits. Untrack them; the existing ignore rule keeps them out from now on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
The NixOS migration UI flow (UIMigrationConfirm/UIMigrationProgress) was removed from ui/software.py in 'refactor(software): delete the NixOS migration UI flow', but a later sqm-fix merge reintroduced the test's import and spec builders for those deleted classes, breaking test collection. Drop the stale references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
test_radec_entry exercised an older dependency-injection API (BlinkingCursor time_provider; CoordinateConverter/CoordinateEntryLogic calc_utils_provider; LayoutConfig no-arg) that radec_entry dropped for module-level calc_utils/time. Patch calc_utils/time at the module level and pass a display stub to LayoutConfig so the suite matches the implementation. 6 stale tests were failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
Surface the ten shielded optical-black rows the IMX290/462 sensor already
transmits and use their trimmed mean as the complete per-frame pedestal for
radiometric SQM. This makes IMX462 SQM zero-touch: no lens cap, dark frames,
or calibration wizard from the user.
Kernel: a second sensor source pad (MEDIA_BUS_FMT_SENSOR_DATA) routes the OB
rows to the receiver as a separate metadata stream without altering the
1920x1080 image matrix.
libcamera: the imx290 cam-helper unpacks the shielded pixels, trims the outer
5% each side and averages the central 90% (preserving sub-ADU resolution on
the 16-bit SensorBlackLevels scale), publishing {l,l,l,l+1}. The +1 marker
lets Python tell a measured value from a static tuning tuple.
Python: camera_pi reads SensorBlackLevels for imx290/imx462 and passes the
native-ADU value as the radiometer sample's optical_black_pedestal. A valid OB
value is the complete pedestal; calibrated then profile pedestals remain the
fallback when OB is missing or unmarked.
Validated on mr2 (IMX462, gain 30) by a same-frame cupboard test: active-green
vs OB dark-accumulation slopes agree to well under 1 ADU/s, so OB is a valid
complete pedestal. Kernel is built via nixos-hardware; pifinder-fast/
pifinder-kernel-cross give a fast x86_64 cross build to seed the binary cache
so CI substitutes the kernel instead of compiling it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
Add the OB pedestal to the SQM architecture note and glossary, and record the decision in ADR-0023. The justification is the same-frame cupboard test on mr2's IMX462 (active-green vs OB slopes agree to <1 ADU/s), scoped to IMX462; IMX296/IMX477 OB remain open. Defines 'optical black' as a glossary term and sets the pedestal precedence (valid OB > calibrated > profile bias). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
python/.venv, mypy/pytest/ruff caches and __pycache__ directories are not needed at runtime and inflated the source output to ~935 MiB, making Attic cache pushes absurdly slow. Removing them drops it to ~19 MiB (cache push ~10-15 s, device upgrade fetch ~30-43 MiB). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
…stal Gate BlackLevelTracker creation behind SQM_BLACK_LEVEL_TRACKER_ENABLED (default False) so the radiometer pedestal is OB when present, else the static profile bias_offset — no tracker in between. Lets us observe the optical-black pedestal's effect in isolation during rollout/validation. Reversible: flip the flag (or revert) to restore the tracker fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
All aperture/annulus/wing radii are defined for the ~1.0-scale Bayer-green photometry images (imx462 green: 490px vs the 512 solve image). The mono imx296 does photometry on its full-res 1088px frame (scale 2.125), where the fixed radii sit inside the star profile: the r=5 aperture holds only 83-94% of a star's flux (measured growth curves, 2026-07-18 sweeps), every local background annulus lands on the PSF itself, and the wing estimator's sky ring eats the wings so it measures f~1 and stays permanently inert. Convert all radii from solve-image pixels to the photometry image's pitch (_scaled_photometry_radii) and give WingEstimator a set_scale() that rescales its patch geometry and clears the rolling window on change. Both the production path (solver.update_sqm) and the calibration wizard's SQM preview use the shared helper. Effect by sensor: - imx462/imx290: radii round to (5, 10, 17) - essentially unchanged. - imx296: radii become (11, 21, 38); on the 2026-07-18 reference sweeps this moves the median error vs a hand-held meter from -0.20 to +0.00 and lets the estimator track real focus drift. - hq: also gets larger radii; its sqm_band_offset (like imx296's) should be refit against reference sweeps after this change. Validated: 121 tests pass across test_sqm, test_solver_sqm, test_sqm_calibration, test_sweep_frame_record, test_solver_shmem (9 new tests cover the scaling).
The sensor's OB clamp pins raw black to a target that moves with sensor state: on the 2026-07-18 imx296 reference sweeps the delivered level was 55.9-56.3 ADU while the device's wizard calibration (bias 58) and the profile constant (60) both over-subtracted. At dark-site signal levels a 2-4 ADU over-subtraction produces a strong SQM-vs-exposure slope (-0.9..-1.5 mag/decade) and kills short exposures outright (background - pedestal <= 0, 'background unresolved'). The BlackLevelTracker already measures the in-session intercept but was suppressed whenever a wizard calibration existed - precisely backwards: a stored bias is one moment's measurement of a moving level. Flip the priority in both SQM paths (radiometric and stellar): a confidently fitted tracked intercept wins for bias; the wizard's dark-current rate remains authoritative and is added on top (the intercept fit cannot separate dark from sky). The tracker's own stderr / deviation-band / lease gates bound how far it can stray, and an accepted fit >2 ADU from the static bias now logs at info. Replayed against the 2026-07-18 imx296 sweeps (radiometric feed, wizard anchor 58): tracker conditions on all four (stderr 0.26-0.32), fits 55.9-56.3, and the recomputed SQM exposure slopes collapse from -0.9..-1.5 to -0.01..+0.52 with the residual offset a tight constant (sd 0.04) for a later band-offset refit. Also: the published black_level_tracked flag now reflects the leased pedestal actually used, not the raw last fit. Validated: 136 tests pass across the SQM suites (4 updated/new).
Rich's 2026-07-18 calibration reports (imx296 + imx462, wizard run interleaved with the reference sweeps) show two capture defects: 1. The first ~3 frames of every stage report actual_exposure_us at the PREVIOUS exposure (req=1us, actual=999999us) yet pass the frame gate, because _capture_and_wait checks the exposure_time field, which echoes the committed setting rather than what the sensor delivered. Stale full-length frames land in the bias/dark stacks. 2. After the exposure change lands, the sensor's optical-black clamp re-settles to a new level over ~3 more frames (imx296: 56 -> 59 ADU). Frames in that window measure neither the old nor the new black level. (On the imx296 the settled clamp is genuinely exposure-dependent: ~56 ADU at 1s vs ~59 ADU at the 29us minimum - a min-exposure bias is only valid for its own regime, which is why the tracked in-session pedestal takes priority in operation.) Together these explain the wizard bias of 58 on a sensor whose 1s clamp level is 56: a median over stale-1s frames at 56, settling frames, and settled-29us frames at 59. Fix: _capture_and_wait gates on actual_exposure_us (exposure_time only as fallback), re-requests a capture when a stale frame arrives, and uses an exposure-proportional tolerance; bias and dark stages discard CLAMP_SETTLE_DISCARD_FRAMES frames after every exposure change (the dark ramp changes exposure per step). Adds ~30-40s to a wizard run. Evidence: dumps/logs/rich-imx296/sqm_calibration_report_imx296_*.json. Validated: 141 tests pass across the SQM suites (5 new).
feat(sqm): per-frame optical-black pedestal for IMX290/IMX462 (zero-touch)
Telemetry menu under Settings > Telemetry: - Record: inline On/Off toggle (no submenu), same pattern as Test Mode. - Sections: one multi-select checklist with inline checkmarks (like the Catalogs filter) for what to record — IMU, SQM, Solves, Targets and Images. Toggling applies live to an in-progress recording. - Max Size: session cap (250 MB / 500 MB / 1 GB / 2 GB / Unlimited). - Load: unchanged. Images ships OFF by default, preserving the old telemetry_images=false: one 512x512 PNG is written per solve, so at ~1 solve/s it costs roughly 300 MB/hour against a few MB/hour for every other section combined. Session size cap. Frames are written by the camera process, so the recorder measures the session directory from its flush loop rather than counting its own writes. On reaching the cap, frame capture is suspended (the one unbounded consumer) while the event log keeps running, so a capped session stays useful instead of going dark. Default 1 GB. Also logs the SQM *ingredients*, not just the end products, so a session stays recomputable if the SQM/airglow maths changes in code later: - radio event gains the raw per-frame inputs — red/blue Bayer backgrounds, optical-black pedestal, photometry image size (green bg and exposure were already there). - the session header snapshots the FULL camera profile constants plus the airglow calibration and camera_type — the constants those ingredients were produced under. The derived published 'sqm' and applied 'floor' are still logged, but explicitly as a what-the-device-showed audit trail, not ground truth. The airglow import is optional: the model is part of the SQM stack only on branches that carry it (deepchart has the radiometer but not airglow), and there a session simply records no airglow constants.
pull_request_target defaults actions/checkout to the repository's default
branch. That branch carries this workflow so the event fires, but not
necessarily .github/scripts/, so update-manifest died with
bash: .github/scripts/publish_manifest.sh: No such file or directory
after a successful 19-minute build -- the image was pushed to Attic but no
device could ever see it, because the manifest was never written.
Check out the PR's base branch instead: it is the branch this workflow file
itself came from, so the scripts are always beside it, and it stays
maintainer-controlled. Checking out the PR head here is not an option -- this
job holds contents:write and the Attic token.
mrosseel
force-pushed
the
test/fullframe-solve
branch
from
July 29, 2026 22:25
cf65033 to
b458b5e
Compare
mrosseel
marked this pull request as draft
July 29, 2026 22:32
…tion
The camera cropped every exposure to a centred square before solving, throwing
away roughly half the sensor area (imx462: 1920x1080 down to 980x980). Cedar
Solve has no trouble with the whole frame, so that discarded area is free
plate-solving sensitivity.
The camera now publishes two frames per exposure: the 512x512 display frame,
unchanged, and a solve frame covering the whole sensor at native scale. Bayer
sensors are binned 2x2 for the solve frame, which removes the RGGB modulation
the old downscale to 512x512 used to smooth away.
Everything downstream of the solver -- SQM photometry, the preview overlay, the
alignment marker -- is written against the display frame, and SQM in particular
is calibrated to its plate scale. So solves are projected back onto the display
frame in one place (project_solution_to_display), keeping those consumers
unchanged. SolveGeometry owns the mapping, composed from the affine transform of
each pipeline stage.
The solver also stopped rediscovering the optics on every frame. It passed a
fixed 12 +/- 4 degree window and never passed a distortion estimate at all. The
first successful solve of a run now captures the measured FOV and lens
distortion; later solves get a +/-0.5 degree window and the measured distortion
as a starting point, which speeds up pattern matching and improves centroid
matching towards the frame corners -- the part of the image the crop used to
discard. A run of failures discards the calibration and reopens the window.
Full frame is on by default and can be turned off with solver_full_frame. It is
disabled automatically when the camera has no sensor profile, or when
camera_rotation is set to something other than a quarter turn, since rotating a
non-square frame by an arbitrary angle would clip the corners.
The 2x2 Bayer bin is gone. It was there because the crop pipeline's downscale
had been smoothing the RGGB checkerboard away implicitly, and losing that was
assumed to bias centroids toward the green sites. Measured against 12 imx462
frames of real sky, the assumption was wrong and the bin was costing exactly
what this change exists to gain:
solve rate median matches median RMSE solves down to
crop (today) 8/12 15.5 15.4" 257 ms
binned 10/12 20.0 40.5" 174 ms
un-binned 11/12 26.0 41.1" 97 ms
Un-binned finds 30% more matches than binned and solves at 2.6x shorter
exposure than the crop -- against 1.5x for the binned version, so binning was
giving away half the sensitivity gain. Angular residuals are statistically
identical (41.1" vs 40.5"), so the centroid-bias worry does not survive
contact with data; the finer sampling is simply what lets marginal stars clear
the detection threshold.
Full frame does cost accuracy against the crop (15.4" -> 41"), from corner
distortion a single-k model cannot fully absorb. That is a real trade, though
41" is negligible against any eyepiece field, and it buys solves on frames the
crop cannot solve at all.
mrosseel
force-pushed
the
test/fullframe-solve
branch
2 times, most recently
from
July 31, 2026 00:14
6c045c4 to
7ae8def
Compare
/api/camera/raw rendered a uint16 sensor frame with Image.fromarray(arr, mode="L"), which reinterprets the 16-bit buffer as 8-bit. The result is interleaved-byte noise that still looks enough like an image to be believed -- I spent a night's captures on it before checking the histogram (median 1, p90 80, 1% at 255) and realising the frames were junk. 16-bit frames now render as mode "I;16" PNGs, preserving every ADU, and a test pins the round trip. Adds /api/camera/rawfull for the whole sensor including the margins the crop discards. Naming follows the exposure sweep's TIFFs: raw is the crop, rawfull is everything. The full frame is ~4 MB, so it is published on demand rather than on every capture -- the endpoint asks, the camera fulfils the request once and clears it. Between them these make the frames the pipeline actually works on retrievable from a running device, which is what debugging a solve failure in the field needs and what nothing currently offers.
Carries the full-frame solver (no Bayer binning -- measured to cost matches without buying accuracy), the session optical calibration, and the raw/rawfull camera API needed to pull frames off a device when a solve fails.
mrosseel
force-pushed
the
test/fullframe-solve
branch
from
July 31, 2026 01:19
7ae8def to
5376c53
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Device-test build only — not for merge. The reviewable change is #554; this branch is that work merged onto the NixOS deployment branch so it can actually be built and installed.
What it changes
Full-sensor solving. The camera cropped every exposure to a centred square before solving — on the imx462, 1920×1080 down to 980×980, discarding roughly half the sensor. It now publishes two frames per exposure: the 512×512 display frame (unchanged — UI, focus, preview, align, SQM) and a full-sensor solve frame at native scale used only by the plate solver. Bayer sensors bin 2×2 for it.
Cached optical calibration. The solver passed a fixed
fov_estimate=12.0, fov_max_error=4.0and never passed a distortion estimate at all. The first successful solve of a run now captures the measured FOV and lens distortion; later solves get a ±0.5° window and that distortion as a starting point.What to check on the device
Matchesat the same exposureFull-frame solving enabled: solve frame 540x960, thenOptical calibration: FOV … distortion …Escape hatch:
solver_full_frame: falsein config reverts to the old crop path without reflashing, so it can be A/B'd in place.Note the horizontal FOV grows only modestly: after the camera's quarter turn the solve frame's horizontal axis is the sensor's short side, which the square crop already spanned. Total area still roughly doubles; the extra sky arrives vertically.
Status
Never run on real camera hardware —
capture_pair, the 2×2 Bayer bin and the geometry mapping all get their first real exercise here. That is the point of this build.ruff, mypy and 1202 unit+smoke tests pass. Benchmarked over 220 archived sweep frames (imx462 + HQ): doubling the solve frame's pixel count costs −0.02 ms in cedar-detect — nothing measurable; total extract+solve stays 0.6–8% of one exposure cycle.
Target device: imx462, which gets the largest area gain and exercises the new Bayer bin path.