[csedu-2026] Extending an automatic question generation pipeline with LLM-based free-response tasks: An analysis of performance metrics using student data#13
Conversation
There was a problem hiding this comment.
Pull request overview
Adds the csedu-2026 dataset package to the repository, including documentation and a reproducibility notebook for the paper “Extending an automatic question generation pipeline with LLM-based free-response tasks: An analysis of performance metrics using student data”.
Changes:
- Registers the new csedu-2026 dataset in the root README index.
- Adds a dataset-specific README describing files, fields, and reproducibility notes.
- Adds the analysis notebook and dataset artifacts (Parquet files via Git LFS).
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| README.md | Adds the csedu-2026 entry to the dataset/paper index. |
| csedu-2026/README.md | Documents the dataset scope, schema, and reproducibility notes for the paper. |
| csedu-2026/LLM-Based Free-Response Tasks Analysis.ipynb | Provides the replication analysis notebook used to compute reported metrics. |
| csedu-2026/exam_writing_sessions.parquet | Adds exam-writing session dataset (Git LFS pointer). |
| csedu-2026/glossary_comparison_sessions.parquet | Adds glossary-comparison session dataset (Git LFS pointer). |
| csedu-2026/fitb_sessions.parquet | Adds FITB session dataset (Git LFS pointer). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| " non_correct = fb[ fb.first_attempt != '+' ]\n", | ||
| " no_reveal = non_correct[ ~non_correct.pattern.str.contains( 'r', case=False ) ]\n", | ||
| " rows.append( {\n", | ||
| " 'Course': course,\n", | ||
| " 'FITB (all)': round( non_correct.pattern.str.contains( r'\\+' ).mean() * 100, 1 ),\n", | ||
| " 'FITB (reveal-less)': round( no_reveal.pattern.str.contains( r'\\+' ).sum() / len( non_correct ) * 100, 1 ),\n", |
There was a problem hiding this comment.
Thanks for flagging this — but this is intentional, not a bug. FITB (reveal-less) is deliberately computed over the same denominator as FITB (all) (len(non_correct), i.e. all incorrect-first-attempt sessions), not restricted to the reveal-less subset. The point of the metric is to answer "of all sessions that started with a wrong answer, what fraction eventually succeeded without using reveal" — kept on the same base as the "all" persistence rate so the two are directly comparable, and comparable to glossary comparison's persistence rate too, which uses the same denominator convention.
The current code reproduces the paper's published reveal-less persistence rates exactly (20.0% for CJ, 13.9% for COM). Dividing by len(no_reveal) instead — the suggested change — gives 97.6% and 61.5%, which don't match the paper and would represent a different (narrower) metric than what's reported. Leaving as-is.
No description provided.