Skip to content

Fix world-to-camera transforms in flow keyframing - #106

Open
zjn20030811 wants to merge 1 commit into
Robbyant:mainfrom
zjn20030811:fix/flow-keyframe-pose-transform
Open

Fix world-to-camera transforms in flow keyframing#106
zjn20030811 wants to merge 1 commit into
Robbyant:mainfrom
zjn20030811:fix/flow-keyframe-pose-transform

Conversation

@zjn20030811

Copy link
Copy Markdown

Summary

  • honor the world-to-camera extrinsic contract in both windowed streaming implementations
  • unproject current-camera points with E_cur^-1 and reproject them with E_kf
  • add an analytic projection regression with non-square geometry and anisotropic intrinsics

Root cause

pose_encoding_to_extri_intri returns OpenCV world-to-camera extrinsics, but the flow-keyframe code treated the current extrinsic as camera-to-world and inverted the keyframe extrinsic. It therefore evaluated

x_kf = E_kf^-1 E_cur x_cur

instead of the required relative transform

x_kf = E_kf E_cur^-1 x_cur

This affects both streaming and windowed keyframe selection paths in gct_stream_window.py and gct_stream_window_v2.py.

The regression uses a 24x40 image, fx != fy, different current/keyframe intrinsics, rotation, translation, and varying depth. The direct homogeneous-projection reference is 4.492928 px; both fixed production implementations return 4.492928 px, while the previous inverse convention returns 6.628602 px. This crosses the 5-pixel flow threshold and changes the keyframe decision.

Validation

  • python -m pytest -q — 2 passed
  • python -m compileall -q lingbot_map tests
  • Black check for the new regression module
  • Ruff fatal checks for the changed Python files
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant