Skip to content

fix(parsing): apply the UTC offset of trajectory timestamps - #70

Merged
ramondalmau merged 1 commit into
devfrom
fix/d2-timestamp-utc
Oct 20, 2026
Merged

ramondalmau merged 1 commit into
devfrom
fix/d2-timestamp-utc

Conversation

@ramondalmau

Copy link
Copy Markdown
Collaborator

Summary

Fixes units audit item D2. neats_json_to_flights dropped the UTC offset of each
trajectory ts without applying it (12:02:00+02:00 became 12:02 UTC instead of 10:02),
and schema-valid RFC 3339 forms (fractional seconds, lowercase z) raised ValueError,
aborting the whole batch.

New helper _ts_to_utc_string in neats_io.py:

  • applies the UTC offset;
  • accepts lowercase t/z and a space separator;
  • truncates fractional seconds (maintainer decision);
  • reads a ts without offset as UTC (maintainer decision);
  • raises ValueError naming the flight and trajectory point for invalid values.

Output contract unchanged: naive UTC string "%Y-%m-%d %H:%M:%S".
The audit report gets a follow-up status table and a status note under D2.

Not in this PR: an invalid ts still stops the whole batch (separate PR).

Evidence that nothing breaks

  • Identical output on all existing data: 148 NEATS JSON files on the NEATS server
    (85,128 flights, 10,995,586 points; every ts is +0000, Z or +00:00) compared
    exactly before/after (values, dtypes, column order, attrs): 0 differences.
  • Full suite with BADA and weather data: 267 passed, 4 skipped, 23 xfailed
    (baseline 231/4/23 + 36 new tests).
  • Adversarial review (3 independent reviewers): differential fuzzing of 309,011
    timestamp forms against the old code and two reference parsers (0 disagreements with
    the references; no UTC input changes); all callers checked incl. Dataiku recipes;
    new tests fail on the old code (19/23). Edge cases they raised are fixed.
  • ruff, ruff format, project mypy clean.

Note on the pre-commit mypy hook

The mirrors-mypy hook fails for every commit, independently of this change: its
isolated env installs numpy 2.5.4, whose stubs use Python 3.12 syntax while mypy targets
3.11. This commit was made with SKIP=mypy; project mypy (uv run mypy src/pyneats/)
passes. A separate PR will fix the hook.

neats_json_to_flights parsed each trajectory `ts` with
strptime("%Y-%m-%dT%H:%M:%S%z") and wrote it back with strftime, which
dropped the UTC offset without applying it: "12:02:00+02:00" became
12:02 UTC instead of 10:02 UTC. Schema-valid RFC 3339 forms such as
fractional seconds or a lowercase "z" raised ValueError and aborted the
whole batch.

The new helper _ts_to_utc_string applies the offset, accepts a lowercase
"t"/"z" and a space separator, truncates fractional seconds, and reads a
ts without offset as UTC. Invalid values raise ValueError naming the
flight and the trajectory point. The output contract is unchanged: a
naive UTC string "%Y-%m-%d %H:%M:%S".

All existing data uses "+0000", "Z" or "+00:00". On 148 NEATS JSON files
(85,128 flights, 10,995,586 points) the output is identical to before.

Fixes units audit item D2; the audit report records the status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ramondalmau
ramondalmau merged commit d9370e9 into dev Oct 20, 2026
6 checks passed
@ramondalmau
ramondalmau deleted the fix/d2-timestamp-utc branch October 20, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant