Skip to content

Fix MATLAB MEX error handling for Octave - #204

Merged
ProfFan merged 1 commit into
masterfrom
codex/matlab-octave-error-path
Aug 23, 2026
Merged

Fix MATLAB MEX error handling for Octave#204
ProfFan merged 1 commit into
masterfrom
codex/matlab-octave-error-path

Conversation

@ProfFan

@ProfFan ProfFan commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • include mex.h with normal C++ linkage so Octave headers can declare templates
  • restore the original std::cout stream buffer before mexErrMsgTxt exits the generated gateway
  • update MATLAB wrapper snapshots and add focused regression coverage

Why

Octave mex.h includes C++ template declarations, which cannot appear inside an outer extern "C" block. Separately, generated gateways redirected std::cout to a stack-local mstream but restored it only after the exception handler. Because mexErrMsgTxt does not return, the error path left std::cout pointing at destroyed storage and Octave could segfault while displaying the exception.

This was reproduced while handling a failed GTSAM CustomFactor callback. With the stream restoration in place, the callback failure is reported as a normal interpreter error instead of SIGSEGV.

Testing

  • uv run pytest tests/test_matlab_wrapper.py -q (17 passed)
  • uv run pytest tests -q (128 passed)
  • cmake .
  • rebuilt the GTSAM Octave MEX and verified the failing callback exits with an error rather than signal 11

@ProfFan
ProfFan merged commit 81f48b1 into master Aug 23, 2026
8 checks passed
@ProfFan
ProfFan deleted the codex/matlab-octave-error-path branch August 23, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants