fix(determinism): byte-identical analysis output for identical runs#101
Open
rahlk wants to merge 1 commit into
Open
fix(determinism): byte-identical analysis output for identical runs#101rahlk wants to merge 1 commit into
rahlk wants to merge 1 commit into
Conversation
) + L2 audit gate (#87) Same flags on the same input now emit byte-identical analysis.json. Four systematic nondeterminism sources closed: - CLI re-execs once with PYTHONHASHSEED=0 (export the var to opt out or pin another seed): PyCG's capped fixpoint iterates hash-ordered sets keyed on module/access-path strings, so an unpinned per-interpreter seed shifted the iteration frontier arbitrarily — observed 1,527 vs 4,712 edges on identical input. Guarded so in-process invocations (Typer CliRunner in the test suite) are never exec'd; Ray workers inherit the pinned seed via an explicit ray.init runtime env. - The PyCG mini-project root is content-derived (sha1 of project + shard files) instead of a random mkdtemp — PyCG state keys on absolute module paths, so a fresh random suffix changed the analysis input every run. An exclusive sidecar flock serializes concurrent analyses of the same shard (a test suite and a manual run on one project would otherwise rmtree each other's tree mid-analysis); distinct projects hash to distinct roots. - Entry points sorted (was filesystem order); emitted call_graph canonically ordered by (src, dst) so identical edge sets always serialize identically. - Jedi inference candidates are tie-broken deterministically (sorted on (full_name, name), not set order) — a union-typed receiver previously resolved to a different member per run; NoneType results are dropped from return-type candidates (Jedi flaps between yielding {NoneType} and {} for the None arm of an Optional, and a real type in the union is the informative pick anyway). Verified: repeated -a 2 runs on the requests fixture are identical in every edge and all but one field; the residue is a single call site whose compiled introspection (an os.environ-derived dict) depends on jedi's helper subprocess surviving the run — environmental and upstream, documented on issue #99. Regression gate: test_l2_runs_are_byte_identical. Also lands the #87 acceptance gate (test_l2_audit_gate_callee_name_equality): >=95% of resolved non-constructor callsites must bind to the invoked attribute name, and no callsite may fall back to a class id when the class declares the exact method. The underlying anchoring fix shipped in 1.0.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #99 (see final status comment there) and completes the #87 acceptance gate.
Four systematic nondeterminism sources closed — hash-seed-driven PyCG fixpoint variance (CLI self-pins
PYTHONHASHSEED=0via a guarded one-time re-exec; Ray workers get it through the runtime env), random PyCG mini-project paths (now content-derived, with a sidecar flock so concurrent analyses of one project serialize instead of deleting each other's tree), filesystem-ordered entry points and unordered edge emission (both canonically sorted), and Jedi union-pick instability (deterministic tie-break + NoneType filtered from return-type candidates).Verified with repeated
-a 2runs on the requests fixture: identical in every edge and all but one field; the residue is one call site whose compiled introspection depends on jedi's helper subprocess surviving the run — environmental/upstream, documented on #99. New gates in the suite:test_l2_runs_are_byte_identical,test_l2_audit_gate_callee_name_equality(#87). Full suite: 177 passed / 5 skipped.