fix: make the random-choose bucket usable, and seed every draw with torch - #60
Open
Hendrik-code wants to merge 1 commit into
Open
fix: make the random-choose bucket usable, and seed every draw with torch#60Hendrik-code wants to merge 1 commit into
Hendrik-code wants to merge 1 commit into
Conversation
…orch * RandomChooseXTransformsGPU wrote into the caller's batch: `out = input`, then `out[i:i+1] = xi` per element. It clones now, as every sibling in gpu/spatial.py already does. * The bucket could not run any transform with a kornia `_param_generator`. It calls apply_transform directly, which skips the forward_parameters step that fills `params`, so RandomLowResTransformGPU and friends raised "params must contain 'scale'" from inside a bucket. It now samples those parameters itself for children that have a generator, and leaves the contrast transforms -- which sample inside apply_transform -- on the empty-params path they expect. * RandomLowResTransformGPU read flags["data_keys"] unguarded. Only MaskSequentialOpsCustom injects that key, so every other caller got a KeyError: calling the transform standalone, or from inside the bucket, which passes the transform's own flags. It defaults to IMAGE, which is what those callers mean. * Blur sigmas and RandConv kernel sizes were drawn with Python's `random`. torch.manual_seed does not reach it, so a "seeded" training run was not reproducible, and under DDP each rank has its own `random` state and picked a different sigma for the same batch. The suite hid this because unit_tests/helpers.py::seed_everything seeds torch, numpy *and* random -- training does not call that. smauglab/transforms/rng.py is where those draws live now, built from the _next_shared_seed / _shared_rand pair that was already sitting in gpu/fromSeg.py, written for exactly this and never called once. That dead copy is deleted rather than left as a third way to draw a random number. unit_tests/test_bucket_and_rng.py: five checks fail against the previous implementation. The RNG one had to be verified separately, by restoring the random.choice call site on its own -- the other four make the module unimportable if reverted together. Models trained before this change saw the old behaviour and will not reproduce against it. No config key, parameter or default changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #59 — last of the four correctness PRs. Deduplication starts in the next one.
The bucket wrote into the caller's batch
RandomChooseXTransformsGPU.apply_transformdidout = input, then assigned back throughout[i:i+1] = xiper element. It clones now, as every sibling ingpu/spatial.pyalready does.The bucket could not run half the transforms it accepts
It calls
apply_transformdirectly, which skips theforward_parametersstep that fillsparams. Any child with a kornia_param_generator— the spatial transforms — raised:It now samples those parameters itself for children that have a generator, and leaves the contrast transforms — which sample inside
apply_transform— on the empty-params path they expect.RandomLowResTransformGPUraisedKeyErroroutside the mask pathIt read
flags["data_keys"]unguarded. OnlyMaskSequentialOpsCustominjects that key, so every other caller failed: calling the transform standalone, or from inside the bucket, which passes the transform's ownflags. It defaults toIMAGE, which is what those callers mean. TheMASKbranch still selects nearest-neighbour resampling, and there's a test pinning that.torch.manual_seeddid not reach every drawBlur sigmas and RandConv kernel sizes were drawn with Python's
random:torch.manual_seeddoes not seed Python'srandom, so a "seeded" training run was not reproducible.randomstate, so ranks picked different sigmas for the same batch.The suite hid this:
unit_tests/helpers.py::seed_everythingseeds torch, numpy andrandom. Training does not call it.New
smauglab/transforms/rng.pyis where those draws live now. It is built from the_next_shared_seed/_shared_randpair that was already sitting ingpu/fromSeg.py, written for exactly this and never called once — that dead copy is deleted here rather than left as a third way to draw a random number.Testing
unit_tests/test_bucket_and_rng.py, 12 tests. Five fail against the parent commit.The RNG one had to be verified on its own, by restoring just the
random.choicecall site — reverting all four source files together makes the module unimportable. The seeding tests deliberately calltorch.manual_seedonly, neverseed_everything, since seedingrandomis what masked the bug.test_it_still_returns_something_transformedguards against the clone turning the bucket into a no-op.Full suite: 69 passed, 168 subtests.
ruffandmypyclean.Compatibility
Models trained before this change saw the old behaviour and will not reproduce against it. No config key, parameter or default changed.