Critical: _detect_regime() calls load_state_dict() on full checkpoint dict — always throws

[bradsmithmba]

Summary

_detect_regime() passes the raw output of torch.load() directly to model.load_state_dict(). Every checkpoint format the trainer produces wraps the state dict inside a larger dict. This mismatch raises a RuntimeError on every ML inference attempt. The broad except Exception clause catches it silently and falls back to the heuristic, making the failure invisible.

Root cause

_detect_regime() (added in PR #42):

model.load_state_dict(torch.load(checkpoint, map_location='cpu', weights_only=True))

RegimeTrainer.save_checkpoint() saves:

checkpoint_data = {
    'epoch': epoch,
    'model_state_dict': self.model.state_dict(),   # <-- state dict is nested here
    'optimizer_state_dict': ...,
    'scheduler_state_dict': ...,
    ...
}
torch.save(checkpoint_data, checkpoint_path)

export_model_for_inference() saves:

inference_state = {
    'state_dict': model.state_dict(),   # <-- different key name
    'model_class': ...,
    ...
}

model.load_state_dict(full_dict) receives a dict whose keys are 'epoch', 'model_state_dict', etc. — not layer parameter names. PyTorch raises RuntimeError: unexpected key(s) in state_dict. The except Exception block swallows it and logs a warning that is easy to miss.

Impact

_detect_regime() always degrades to heuristic when a checkpoint exists, for a different reason than issue #43. Even after fixing the glob patterns, this bug prevents the model from loading. Both bugs must be fixed together for the ML path to activate.

Fix

Unpack the correct key based on the checkpoint format:

data = torch.load(checkpoint, map_location='cpu', weights_only=True)
# Trainer checkpoint format
if 'model_state_dict' in data:
    state_dict = data['model_state_dict']
# Inference export format
elif 'state_dict' in data:
    state_dict = data['state_dict']
else:
    state_dict = data  # raw state dict fallback
model.load_state_dict(state_dict)

Related

Companion to issue #43 (_find_checkpoint() glob patterns never match).