chore: add more baselines (ModernBERT, CodeSage) and CodeNet fixes - #58
Conversation
Register codesage/codesage-small (130M params) for code classification and clone detection tasks. CodeSage uses a custom HuggingFace architecture requiring trust_remote_code, and its StarCoder-based tokenizer needs cls/sep/pad token fallbacks. Clone detection uses mean pooling (same as ModernBERT).
Conv1D was moved from transformers.modeling_utils to transformers.pytorch_utils. CodeSage's remote model code still imports from the old location, causing an ImportError. Add a monkey-patch in all downstream run.py files to restore the symbol before model loading.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5f42687c12
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| export CUDA_VISIBLE_DEVICES="$CUDA_DEVICE" | ||
|
|
||
| cd "$ROOT_DIR/downstream/Clone-detection-BigCloneBench" | ||
| ./run.sh answerdotai/ModernBERT-base "$ROOT_DIR/results/modernbert/Clone-detection-BigCloneBench" |
There was a problem hiding this comment.
This launcher only passes <model_path> <output_dir>, but downstream/Clone-detection-BigCloneBench/run.sh hard-codes --model_type=roberta and --tokenizer_name=roberta-base when invoking code/run.py. When users run this *_modernbert.sh script, they are not actually evaluating a ModernBERT setup (and may get bad/incompatible weight loading), which can invalidate baseline results for this task.
Useful? React with 👍 / 👎.
| export CUDA_VISIBLE_DEVICES="$CUDA_DEVICE" | ||
|
|
||
| cd "$ROOT_DIR/downstream/Defect-detection" | ||
| ./run.sh answerdotai/ModernBERT-base "$ROOT_DIR/results/modernbert/Defect-detection" |
There was a problem hiding this comment.
defect_modernbert.sh calls downstream/Defect-detection/run.sh with only two arguments, but that wrapper hard-codes --model_type=roberta and --tokenizer_name=roberta-base. As a result, this new script cannot run a true ModernBERT baseline and will either run with mismatched settings or fail during model load, breaking experiment validity for this launcher.
Useful? React with 👍 / 👎.
| export CUDA_VISIBLE_DEVICES="$CUDA_DEVICE" | ||
|
|
||
| cd "$ROOT_DIR/downstream/Code-translation" | ||
| ./run.sh answerdotai/ModernBERT-base "$ROOT_DIR/results/modernbert/Code-translation" |
There was a problem hiding this comment.
This script routes to downstream/Code-translation/run.sh, which is hard-coded to RoBERTa for both training and inference (--model_type roberta, --tokenizer_name roberta-base, and inference --model_name_or_path roberta-base). Running translation_modernbert.sh therefore does not evaluate ModernBERT and can produce misleading or failed runs for the advertised baseline.
Useful? React with 👍 / 👎.
1 participant
Summary
ouput_file_pathtypo in CodeNet dataset scriptsTest plan