Skip to content

fix(runners): one flight that cannot be converted no longer stops the run - #71

Merged
ramondalmau merged 1 commit into
devfrom
fix/isolate-invalid-flights
Oct 20, 2026
Merged

ramondalmau merged 1 commit into
devfrom
fix/isolate-invalid-flights

Conversation

@ramondalmau

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #70 (units audit D2). FleetRunner converted the whole JSON input before any
per-flight error handling, so one flight with an invalid trajectory timestamp (or any
malformed flight object) stopped the run for every flight.

  • neats_json_to_flights gains an opt-in failures list. When given, a flight that cannot
    be converted is left out and recorded as a FlightConversionFailure (input index,
    identifying attrs, error). Without it, behaviour is unchanged (Dataiku recipes and other
    direct callers are not affected).
  • A top level that is an object or a string is a whole-file error: TypeError, not one
    fake failed flight (audit D1 stays open: accepting a bare object is not decided).
  • FleetRunner._load_trajectories and scripts/run_json.py use it: a failed flight becomes
    a Failed at parsing error record and the other flights go on. If every flight fails, the
    run stops before loading the weather and returns only error records.
  • Recorded errors carry no tracebacks, so they do not keep the input in memory.
  • The units audit report records the status (status table, D2 and D1 notes).

Evidence that nothing breaks

  • Default path identical: 148 NEATS JSON files (85,128 flights, 10,995,586 points),
    exact comparison before/after: 0 differences.
  • New path: identical output on every valid file; the one file that crashes today
    (ao_json_sample.json, trajectory stored as a list) now records its 5 flights as
    failures instead of stopping the run.
  • Tests: 51 new/updated unit tests (no data needed), including an all-flights-fail run
    with non-existent weather paths and a check that the input is freed. New end-to-end test:
    5 golden flights + 1 bad flight → the 5 golden results unchanged + 1 error record.
  • Full suite with BADA and weather:
  • Review: 3 independent reviewers + a re-check; all findings fixed or noted below.
  • ruff, ruff format, project mypy clean.

Known, not in this PR

  • Low-altitude results are recognised by the text "low-altitude" in the error message
    (pre-existing); a structured marker would be more robust.
  • Pre-commit mypy hook is broken for everyone (numpy 2.5.4 stubs vs Python 3.11 target);
    commit made with SKIP=mypy, project mypy passes. Separate chore PR to follow.

… run

FleetRunner converted the whole JSON input with neats_json_to_flights
before any per-flight error handling, so one flight with an invalid
trajectory timestamp (or any malformed flight object) raised and stopped
the run for every flight.

neats_json_to_flights gains an opt-in `failures` list. When given, a
flight that cannot be converted is left out and recorded as a
FlightConversionFailure (input index, identifying attrs, error); without
it, behaviour is unchanged. A top level that is an object or a string is
a whole-file error and raises TypeError instead of being recorded flight
by flight. Recorded errors carry no tracebacks, so they do not keep the
input alive.

FleetRunner._load_trajectories and scripts/run_json.py use the list: a
failed flight becomes a "Failed at parsing" error record and the other
flights go on. If every flight fails, the run stops before loading the
weather and returns only error records. The run_json summary now prints
the flight id of nested fleet error records.

The default conversion path is unchanged: on 148 NEATS JSON files
(85,128 flights) its output is identical to before. The units audit
report records the status (D2 follow-up, D1 note).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ramondalmau
ramondalmau merged commit 97a7018 into dev Oct 20, 2026
6 checks passed
@ramondalmau
ramondalmau deleted the fix/isolate-invalid-flights branch October 20, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant