diff --git a/README.md b/README.md index fcc7fac..19beaac 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ Directory | Paper [l@s-2025](https://github.com/vitalsource/data/tree/main/l@s-2025) | [Refining sentence selection for automatic cloze question generation with large language models](https://doi.org/10.1145/3698205.3733926) [aied-2025-evallac](https://github.com/vitalsource/data/tree/main/aied-2025-evallac) | [Open-ended questions need personalized feedback: Analyzing LLM-enabled features with student data](https://drive.google.com/file/d/15HCyN1uU6AtIT8aVpMJCS5ZQa75buA_g/view) [aied-2025-itextbooks](https://github.com/vitalsource/data/tree/main/aied-2025-itextbooks) | [Improving textbook accessibility through AI simplification: Readability improvements and meaning preservation](https://ceur-ws.org/Vol-4010/itb25_s2p2.pdf) +[csedu-2026](https://github.com/vitalsource/data/tree/main/csedu-2026) | [Extending an automatic question generation pipeline with LLM-based free-response tasks: An analysis of performance metrics using student data](https://doi.org/10.5220/0014655400004021) ![NEW](https://img.shields.io/badge/status-new-brightgreen) [aied-2026-itextbooks](https://github.com/vitalsource/data/tree/main/aied-2026-itextbooks) | [LLM feedback isn’t automatically better: Static scaffolds outperform dynamic feedback in textbook-embedded practice](https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf) ![NEW](https://img.shields.io/badge/status-new-brightgreen) Unless otherwise noted, our datasets are available under the diff --git a/csedu-2026/LLM-Based Free-Response Tasks Analysis.ipynb b/csedu-2026/LLM-Based Free-Response Tasks Analysis.ipynb new file mode 100644 index 0000000..db24f92 --- /dev/null +++ b/csedu-2026/LLM-Based Free-Response Tasks Analysis.ipynb @@ -0,0 +1,1418 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "title", + "metadata": {}, + "source": [ + "# LLM-Based Free-Response Tasks Analysis\n", + "\n", + "This notebook contains Python code for reproducing the results in our paper:\n", + "\n", + "Van Campenhout, R., Dittel, J. S., Jerome, B., & Johnson, B. G. (2026). Extending an automatic question generation pipeline with LLM-based free-response tasks: An analysis of performance metrics using student data. In *Proceedings of the 18th International Conference on Computer Supported Education* (Vol. 1, pp. 26–38). SCITEPRESS. https://doi.org/10.5220/0014655400004021\n", + "\n", + "Results are presented in the order they appear in the paper, organized by section. For each result, an excerpt from the paper is given, followed by code to compute the result from the dataset provided. Example:\n", + "\n", + ">This resulted in a data set of 12,485 LLM-enabled question sessions (6,847 exam question and 5,638 glossary comparison)...\n", + "\n", + "```python\n", + "sessions = pd.concat( [ exam_writing_sessions, glossary_comparison_sessions ] )\n", + "print( f'{len( sessions )} sessions' )\n", + "```\n", + "\n", + "Please refer to the paper for additional context.\n", + "\n", + "---\n", + "\n", + "In `glossary_comparison_sessions` and `fitb_sessions`, student answer attempts are classified using shorthand symbols. These symbols (`+`, `-`, `x`) are not used in the paper but correspond to the categories defined in the following table:\n", + "\n", + "| Category | Symbol | Description |\n", + "| ----------- | ------ | ------------------------------------------------------------------------------------------------------- |\n", + "| Correct | `+` | The response accurately addressed the key distinction between terms. |\n", + "| Incorrect | `-` | The response did not sufficiently answer the question, despite appearing to be a genuine effort. |\n", + "| Non-Genuine | `x` | The response did not constitute a legitimate attempt (e.g., random characters, \"idk\", irrelevant text). |\n", + "\n", + "`exam_writing_sessions` does not use these symbols — its `pattern` field records one `e` per answer attempt." + ] + }, + { + "cell_type": "markdown", + "id": "imports-header", + "metadata": {}, + "source": [ + "## Read dataset" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "imports", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:46.496022Z", + "iopub.status.busy": "2026-07-24T02:23:46.495666Z", + "iopub.status.idle": "2026-07-24T02:23:47.303209Z", + "shell.execute_reply": "2026-07-24T02:23:47.302468Z" + } + }, + "outputs": [], + "source": [ + "import pandas as pd" + ] + }, + { + "cell_type": "markdown", + "id": "read-ecm-header", + "metadata": {}, + "source": [ + "LLM-enabled question sessions." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "read-data", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.304915Z", + "iopub.status.busy": "2026-07-24T02:23:47.304729Z", + "iopub.status.idle": "2026-07-24T02:23:47.366817Z", + "shell.execute_reply": "2026-07-24T02:23:47.365960Z" + } + }, + "outputs": [], + "source": [ + "exam_writing_sessions = pd.read_parquet( 'exam_writing_sessions.parquet' )\n", + "glossary_comparison_sessions = pd.read_parquet( 'glossary_comparison_sessions.parquet' )" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "read-fitb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.368342Z", + "iopub.status.busy": "2026-07-24T02:23:47.368215Z", + "iopub.status.idle": "2026-07-24T02:23:47.395156Z", + "shell.execute_reply": "2026-07-24T02:23:47.394343Z" + } + }, + "outputs": [], + "source": [ + "fitb_sessions = pd.read_parquet( 'fitb_sessions.parquet' )" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "head-exam", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.396837Z", + "iopub.status.busy": "2026-07-24T02:23:47.396688Z", + "iopub.status.idle": "2026-07-24T02:23:47.409163Z", + "shell.execute_reply": "2026-07-24T02:23:47.408458Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
question_idstudent_idcoursetextbook_idquestionpatternanswerfeedbacksecond_answersecond_feedback
00499745c5b03aec717885c833110d74965ce50efcb05c2...2KQPBPGQKYRFKDFKYHA5COM9781544349848Write an exam question for the section \"Relati...eWhat are the 3 main parts of relational listen...Your question does capture some important aspe...NoneNone
10499745c5b03aec717885c833110d74965ce50efcb05c2...2MQSTPV5RBCZNUTA4TTNCOM9781544349848Write an exam question for the section \"Relati...ehow is relational listening different across t...Your question, \"How is relational listening di...NoneNone
20499745c5b03aec717885c833110d74965ce50efcb05c2...2T6YX2UQS77MJ8VGNB7NCOM9781544349848Write an exam question for the section \"Relati...eWhat is an example of relational listening?Your question, \"What is an example of relation...NoneNone
30499745c5b03aec717885c833110d74965ce50efcb05c2...2TWHRZM72RTYKHNRYVZFCOM9781544349848Write an exam question for the section \"Relati...eDiscuss the concept of relational listening an...Your question does a good job of capturing the...NoneNone
40499745c5b03aec717885c833110d74965ce50efcb05c2...3AKC5SVKH3V33KUC6N3BCOM9781544349848Write an exam question for the section \"Relati...eexplain how relational listening shows the con...Your question, \"Explain how relational listeni...NoneNone
\n", + "
" + ], + "text/plain": [ + " question_id student_id \\\n", + "0 0499745c5b03aec717885c833110d74965ce50efcb05c2... 2KQPBPGQKYRFKDFKYHA5 \n", + "1 0499745c5b03aec717885c833110d74965ce50efcb05c2... 2MQSTPV5RBCZNUTA4TTN \n", + "2 0499745c5b03aec717885c833110d74965ce50efcb05c2... 2T6YX2UQS77MJ8VGNB7N \n", + "3 0499745c5b03aec717885c833110d74965ce50efcb05c2... 2TWHRZM72RTYKHNRYVZF \n", + "4 0499745c5b03aec717885c833110d74965ce50efcb05c2... 3AKC5SVKH3V33KUC6N3B \n", + "\n", + " course textbook_id question \\\n", + "0 COM 9781544349848 Write an exam question for the section \"Relati... \n", + "1 COM 9781544349848 Write an exam question for the section \"Relati... \n", + "2 COM 9781544349848 Write an exam question for the section \"Relati... \n", + "3 COM 9781544349848 Write an exam question for the section \"Relati... \n", + "4 COM 9781544349848 Write an exam question for the section \"Relati... \n", + "\n", + " pattern answer \\\n", + "0 e What are the 3 main parts of relational listen... \n", + "1 e how is relational listening different across t... \n", + "2 e What is an example of relational listening? \n", + "3 e Discuss the concept of relational listening an... \n", + "4 e explain how relational listening shows the con... \n", + "\n", + " feedback second_answer \\\n", + "0 Your question does capture some important aspe... None \n", + "1 Your question, \"How is relational listening di... None \n", + "2 Your question, \"What is an example of relation... None \n", + "3 Your question does a good job of capturing the... None \n", + "4 Your question, \"Explain how relational listeni... None \n", + "\n", + " second_feedback \n", + "0 None \n", + "1 None \n", + "2 None \n", + "3 None \n", + "4 None " + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "exam_writing_sessions.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "head-glossary", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.411150Z", + "iopub.status.busy": "2026-07-24T02:23:47.410992Z", + "iopub.status.idle": "2026-07-24T02:23:47.419582Z", + "shell.execute_reply": "2026-07-24T02:23:47.418860Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
question_idstudent_idcoursetextbook_idquestionpatternfirst_attemptsecond_attemptanswerfeedbacksecond_answersecond_feedback
0041da273050f1811a3146414b755a91921f8c2f286315a...2KQPBPGQKYRFKDFKYHA5COM9781544349848Explain the difference between the term social...++NoneSocial roles are roles functioning to encourag...Your explanation accurately captures the essen...NoneNone
1041da273050f1811a3146414b755a91921f8c2f286315a...2MQSTPV5RBCZNUTA4TTNCOM9781544349848Explain the difference between the term social...--Nonesocial roles are informal and task roles are f...Your explanation touches on an aspect of the r...NoneNone
2041da273050f1811a3146414b755a91921f8c2f286315a...2T6YX2UQS77MJ8VGNB7NCOM9781544349848Explain the difference between the term social...++NoneSocial roles are roles functioning to encurage...Your explanation accurately captures the essen...NoneNone
3041da273050f1811a3146414b755a91921f8c2f286315a...2TWHRZM72RTYKHNRYVZFCOM9781544349848Explain the difference between the term social...++NoneSocial roles aim to maintain group cohesion an...Your explanation is accurate and well-articula...NoneNone
4041da273050f1811a3146414b755a91921f8c2f286315a...3AKC5SVKH3V33KUC6N3BCOM9781544349848Explain the difference between the term social...++NoneSocial roles refer to behaviors and responsibi...Your explanation is accurate and well-articula...NoneNone
\n", + "
" + ], + "text/plain": [ + " question_id student_id \\\n", + "0 041da273050f1811a3146414b755a91921f8c2f286315a... 2KQPBPGQKYRFKDFKYHA5 \n", + "1 041da273050f1811a3146414b755a91921f8c2f286315a... 2MQSTPV5RBCZNUTA4TTN \n", + "2 041da273050f1811a3146414b755a91921f8c2f286315a... 2T6YX2UQS77MJ8VGNB7N \n", + "3 041da273050f1811a3146414b755a91921f8c2f286315a... 2TWHRZM72RTYKHNRYVZF \n", + "4 041da273050f1811a3146414b755a91921f8c2f286315a... 3AKC5SVKH3V33KUC6N3B \n", + "\n", + " course textbook_id question \\\n", + "0 COM 9781544349848 Explain the difference between the term social... \n", + "1 COM 9781544349848 Explain the difference between the term social... \n", + "2 COM 9781544349848 Explain the difference between the term social... \n", + "3 COM 9781544349848 Explain the difference between the term social... \n", + "4 COM 9781544349848 Explain the difference between the term social... \n", + "\n", + " pattern first_attempt second_attempt \\\n", + "0 + + None \n", + "1 - - None \n", + "2 + + None \n", + "3 + + None \n", + "4 + + None \n", + "\n", + " answer \\\n", + "0 Social roles are roles functioning to encourag... \n", + "1 social roles are informal and task roles are f... \n", + "2 Social roles are roles functioning to encurage... \n", + "3 Social roles aim to maintain group cohesion an... \n", + "4 Social roles refer to behaviors and responsibi... \n", + "\n", + " feedback second_answer \\\n", + "0 Your explanation accurately captures the essen... None \n", + "1 Your explanation touches on an aspect of the r... None \n", + "2 Your explanation accurately captures the essen... None \n", + "3 Your explanation is accurate and well-articula... None \n", + "4 Your explanation is accurate and well-articula... None \n", + "\n", + " second_feedback \n", + "0 None \n", + "1 None \n", + "2 None \n", + "3 None \n", + "4 None " + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "glossary_comparison_sessions.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "head-fitb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.421066Z", + "iopub.status.busy": "2026-07-24T02:23:47.420923Z", + "iopub.status.idle": "2026-07-24T02:23:47.427040Z", + "shell.execute_reply": "2026-07-24T02:23:47.426369Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
question_idstudent_idcoursetextbook_idpatternfirst_attemptsecond_attempt
000309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea...3AKC5SVKH3V33KUC6N3BCJ9781071845226xra+x+
100309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea...84AT2R3TAVDEJGPNHGM6CJ9781071845226++None
200309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea...8U2X4APT3CKCUXC4EBTXCJ9781071845226++None
300309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea...8U8N2HN4FBR6ED2V6PMRCJ9781071845226-r+-+
400309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea...A8QHW6ZTFB6VBJZRVDSHCJ9781071845226a++None
\n", + "
" + ], + "text/plain": [ + " question_id student_id \\\n", + "0 00309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea... 3AKC5SVKH3V33KUC6N3B \n", + "1 00309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea... 84AT2R3TAVDEJGPNHGM6 \n", + "2 00309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea... 8U2X4APT3CKCUXC4EBTX \n", + "3 00309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea... 8U8N2HN4FBR6ED2V6PMR \n", + "4 00309d69ce8b082b4630e979a4b7fccc33b03cf53c39ea... A8QHW6ZTFB6VBJZRVDSH \n", + "\n", + " course textbook_id pattern first_attempt second_attempt \n", + "0 CJ 9781071845226 xra+ x + \n", + "1 CJ 9781071845226 + + None \n", + "2 CJ 9781071845226 + + None \n", + "3 CJ 9781071845226 -r+ - + \n", + "4 CJ 9781071845226 a+ + None " + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "fitb_sessions.head()" + ] + }, + { + "cell_type": "markdown", + "id": "methods-header", + "metadata": {}, + "source": [ + "## 2. Methods" + ] + }, + { + "cell_type": "markdown", + "id": "data-collection-header", + "metadata": {}, + "source": [ + "### 2.4. Question Usage Data" + ] + }, + { + "cell_type": "markdown", + "id": "dataset-quote", + "metadata": {}, + "source": [ + ">This resulted in a data set of 12,485 LLM-enabled question sessions (6,847 exam question and 5,638 glossary comparison), encompassing 13,889 interaction events, 101 questions, 246 students, and two different textbooks (Duck & McMahan, 2021; Mallicoat, 2023)." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "dataset-fingerprint", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.428630Z", + "iopub.status.busy": "2026-07-24T02:23:47.428456Z", + "iopub.status.idle": "2026-07-24T02:23:47.440258Z", + "shell.execute_reply": "2026-07-24T02:23:47.439669Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "12485 LLM-enabled question sessions\n", + "6847 exam question sessions\n", + "5638 glossary comparison sessions\n", + "13889 interaction events\n", + "101 questions\n", + "246 students\n", + "2 textbooks\n" + ] + } + ], + "source": [ + "sessions = pd.concat( [ exam_writing_sessions, glossary_comparison_sessions ] )\n", + "print( f'{len( sessions )} LLM-enabled question sessions' )\n", + "print( f'{len( exam_writing_sessions )} exam question sessions' )\n", + "print( f'{len( glossary_comparison_sessions )} glossary comparison sessions' )\n", + "print( f'{sessions.pattern.apply( len ).sum()} interaction events' )\n", + "print( f'{sessions.question_id.nunique()} questions' )\n", + "print( f'{sessions.student_id.nunique()} students' )\n", + "print( f'{sessions.textbook_id.nunique()} textbooks' )" + ] + }, + { + "cell_type": "markdown", + "id": "fitb-quote", + "metadata": {}, + "source": [ + ">For comparative purposes, data from the standard FITB questions (which constitute the majority of standard items) were retrieved for the same courses. The resulting FITB data included 52,995 student-question sessions." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "fitb-count", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.441589Z", + "iopub.status.busy": "2026-07-24T02:23:47.441464Z", + "iopub.status.idle": "2026-07-24T02:23:47.443875Z", + "shell.execute_reply": "2026-07-24T02:23:47.443363Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "52995 FITB sessions\n" + ] + } + ], + "source": [ + "print( f'{len( fitb_sessions )} FITB sessions' )" + ] + }, + { + "cell_type": "markdown", + "id": "per-course-note", + "metadata": {}, + "source": [ + "Per-course session counts. The two courses are CJ 4060 (Women, Gender, and Crime; 47 students) and COMST 1010 (Communication in Everyday Life; 205 students). Per-course analyses filter to sessions in the matching textbook." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "per-course", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.445213Z", + "iopub.status.busy": "2026-07-24T02:23:47.445073Z", + "iopub.status.idle": "2026-07-24T02:23:47.489823Z", + "shell.execute_reply": "2026-07-24T02:23:47.489241Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CJ: 47 students / 959 ECM sessions / 7533 FITB sessions\n", + "COM: 200 students / 11526 ECM sessions / 45462 FITB sessions\n" + ] + } + ], + "source": [ + "CJ_TEXTBOOK = '9781071845226'\n", + "COM_TEXTBOOK = '9781544349848'\n", + "\n", + "for course, textbook in [ ( 'CJ', CJ_TEXTBOOK ), ( 'COM', COM_TEXTBOOK ) ]:\n", + " ew = exam_writing_sessions[ ( exam_writing_sessions.course == course ) & ( exam_writing_sessions.textbook_id == textbook ) ]\n", + " gc = glossary_comparison_sessions[ ( glossary_comparison_sessions.course == course ) & ( glossary_comparison_sessions.textbook_id == textbook ) ]\n", + " fb = fitb_sessions[ ( fitb_sessions.course == course ) & ( fitb_sessions.textbook_id == textbook ) ]\n", + " ecm = pd.concat( [ ew, gc ] )\n", + " print( f'{course}: {ecm.student_id.nunique()} students / {len( ecm )} ECM sessions / {len( fb )} FITB sessions' )" + ] + }, + { + "cell_type": "markdown", + "id": "results-header", + "metadata": {}, + "source": [ + "## 3. Results and Discussion" + ] + }, + { + "cell_type": "markdown", + "id": "engagement-header", + "metadata": {}, + "source": [ + "### 3.1. Engagement" + ] + }, + { + "cell_type": "markdown", + "id": "table1-quote", + "metadata": {}, + "source": [ + "**Table 1**\n", + "\n", + "Mean number of students answering by question type.\n", + "\n", + "| | CJ (N = 47) | COM (N = 205) |\n", + "| ------------------- | ----------: | ------------: |\n", + "| Exam Question | 22.9 | 171.2 |\n", + "| Glossary Comparison | 32.4 | 173.2 |\n", + "| Standard FITB | 37.1 | 174.9 |" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "table1", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.491319Z", + "iopub.status.busy": "2026-07-24T02:23:47.491200Z", + "iopub.status.idle": "2026-07-24T02:23:47.566916Z", + "shell.execute_reply": "2026-07-24T02:23:47.566344Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Exam QuestionGlossary Comp.Standard FITB
Course
CJ22.932.437.1
COM171.2173.2174.9
\n", + "
" + ], + "text/plain": [ + " Exam Question Glossary Comp. Standard FITB\n", + "Course \n", + "CJ 22.9 32.4 37.1\n", + "COM 171.2 173.2 174.9" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rows = []\n", + "for course, textbook in [ ( 'CJ', CJ_TEXTBOOK ), ( 'COM', COM_TEXTBOOK ) ]:\n", + " ew = exam_writing_sessions[ ( exam_writing_sessions.course == course ) & ( exam_writing_sessions.textbook_id == textbook ) ]\n", + " gc = glossary_comparison_sessions[ ( glossary_comparison_sessions.course == course ) & ( glossary_comparison_sessions.textbook_id == textbook ) ]\n", + " fb = fitb_sessions[ ( fitb_sessions.course == course ) & ( fitb_sessions.textbook_id == textbook ) ]\n", + " rows.append( {\n", + " 'Course': course,\n", + " 'Exam Question': ew.groupby( 'question_id' ).student_id.nunique().mean().round( 1 ),\n", + " 'Glossary Comp.': gc.groupby( 'question_id' ).student_id.nunique().mean().round( 1 ),\n", + " 'Standard FITB': fb.groupby( 'question_id' ).student_id.nunique().mean().round( 1 ),\n", + " } )\n", + "pd.DataFrame( rows ).set_index( 'Course' )" + ] + }, + { + "cell_type": "markdown", + "id": "difficulty-header", + "metadata": {}, + "source": [ + "### 3.2. Difficulty" + ] + }, + { + "cell_type": "markdown", + "id": "table2-quote", + "metadata": {}, + "source": [ + ">As shown in Table 2, the glossary comparison questions exhibit a lower mean first-attempt accuracy than the standard FITB items in both CJ and COM. For CJ, glossary comparisons average a 69.8% accuracy rate, compared to 83.7% for standard FITB. In COM, the gap narrows somewhat (63.5% vs. 72.1%).\n", + "\n", + "**Table 2**\n", + "\n", + "Mean difficulty index by question type.\n", + "\n", + "| | CJ (N = 47) | COM (N = 205) |\n", + "| ------------------- | ----------: | ------------: |\n", + "| Glossary Comparison | 69.8 | 63.5 |\n", + "| Standard FITB | 83.7 | 72.1 |" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "table2", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.568237Z", + "iopub.status.busy": "2026-07-24T02:23:47.568096Z", + "iopub.status.idle": "2026-07-24T02:23:47.592496Z", + "shell.execute_reply": "2026-07-24T02:23:47.591834Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Glossary Comp.Standard FITB
Course
CJ69.583.7
COM62.872.1
\n", + "
" + ], + "text/plain": [ + " Glossary Comp. Standard FITB\n", + "Course \n", + "CJ 69.5 83.7\n", + "COM 62.8 72.1" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rows = []\n", + "for course, textbook in [ ( 'CJ', CJ_TEXTBOOK ), ( 'COM', COM_TEXTBOOK ) ]:\n", + " gc = glossary_comparison_sessions[ ( glossary_comparison_sessions.course == course ) & ( glossary_comparison_sessions.textbook_id == textbook ) ]\n", + " fb = fitb_sessions[ ( fitb_sessions.course == course ) & ( fitb_sessions.textbook_id == textbook ) ]\n", + " rows.append( {\n", + " 'Course': course,\n", + " 'Glossary Comp.': ( ( gc.first_attempt == '+' ).mean() * 100 ).round( 1 ),\n", + " 'Standard FITB': ( ( fb.first_attempt == '+' ).mean() * 100 ).round( 1 ),\n", + " } )\n", + "pd.DataFrame( rows ).set_index( 'Course' )" + ] + }, + { + "cell_type": "markdown", + "id": "persistence-header", + "metadata": {}, + "source": [ + "### 3.3. Persistence" + ] + }, + { + "cell_type": "markdown", + "id": "table3-quote", + "metadata": {}, + "source": [ + ">As shown in Table 3, persistence rates, defined as the percentage of students who eventually succeed after an incorrect first attempt, differ sharply between glossary comparison and FITB. This was not anticipated. In CJ, 14.1% of students persisted to a correct response for glossary comparisons, compared to 98.5% for FITB.\n", + "\n", + "**Table 3**\n", + "\n", + "Mean persistence by question type.\n", + "\n", + "| | CJ (N = 47) | COM (N = 205) |\n", + "| ------------------- | ----------: | ------------: |\n", + "| Glossary Comparison | 14.1 | 20.3 |\n", + "| Standard FITB | 98.5 | 88.6 |" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "table3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.593904Z", + "iopub.status.busy": "2026-07-24T02:23:47.593784Z", + "iopub.status.idle": "2026-07-24T02:23:47.621771Z", + "shell.execute_reply": "2026-07-24T02:23:47.621239Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Glossary Comp.Standard FITB
Course
CJ14.498.5
COM20.688.6
\n", + "
" + ], + "text/plain": [ + " Glossary Comp. Standard FITB\n", + "Course \n", + "CJ 14.4 98.5\n", + "COM 20.6 88.6" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rows = []\n", + "for course, textbook in [ ( 'CJ', CJ_TEXTBOOK ), ( 'COM', COM_TEXTBOOK ) ]:\n", + " gc = glossary_comparison_sessions[ ( glossary_comparison_sessions.course == course ) & ( glossary_comparison_sessions.textbook_id == textbook ) ]\n", + " fb = fitb_sessions[ ( fitb_sessions.course == course ) & ( fitb_sessions.textbook_id == textbook ) ]\n", + " # Among sessions with an incorrect first attempt, rate with eventual correct\n", + " gc_persist = gc[ gc.first_attempt != '+' ].pattern.str.contains( r'\\+' ).mean() * 100\n", + " fb_persist = fb[ fb.first_attempt != '+' ].pattern.str.contains( r'\\+' ).mean() * 100\n", + " rows.append( {\n", + " 'Course': course,\n", + " 'Glossary Comp.': round( gc_persist, 1 ),\n", + " 'Standard FITB': round( fb_persist, 1 ),\n", + " } )\n", + "pd.DataFrame( rows ).set_index( 'Course' )" + ] + }, + { + "cell_type": "markdown", + "id": "persistence-reveal-quote", + "metadata": {}, + "source": [ + ">A straightforward follow-up analysis suggested by the above is to separate persistence for \"reveal\" and \"non-reveal\" FITB sessions. Some students do attempt to solve FITB items without clicking the reveal button, even if it is available. For these courses, the \"reveal-less\" FITB persistence rate is substantially lower, 20.0% for CJ and 13.9% for COM, which is much closer to the glossary comparison rate and actually lower than it for COM.\n", + "\n", + "In the FITB session `pattern` field, `r` indicates a reveal-answer action." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "table3-reveal", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.623388Z", + "iopub.status.busy": "2026-07-24T02:23:47.623276Z", + "iopub.status.idle": "2026-07-24T02:23:47.651123Z", + "shell.execute_reply": "2026-07-24T02:23:47.650478Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
FITB (all)FITB (reveal-less)
Course
CJ98.520.0
COM88.613.9
\n", + "
" + ], + "text/plain": [ + " FITB (all) FITB (reveal-less)\n", + "Course \n", + "CJ 98.5 20.0\n", + "COM 88.6 13.9" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rows = []\n", + "for course, textbook in [ ( 'CJ', CJ_TEXTBOOK ), ( 'COM', COM_TEXTBOOK ) ]:\n", + " fb = fitb_sessions[ ( fitb_sessions.course == course ) & ( fitb_sessions.textbook_id == textbook ) ]\n", + " non_correct = fb[ fb.first_attempt != '+' ]\n", + " no_reveal = non_correct[ ~non_correct.pattern.str.contains( 'r', case=False ) ]\n", + " rows.append( {\n", + " 'Course': course,\n", + " 'FITB (all)': round( non_correct.pattern.str.contains( r'\\+' ).mean() * 100, 1 ),\n", + " 'FITB (reveal-less)': round( no_reveal.pattern.str.contains( r'\\+' ).sum() / len( non_correct ) * 100, 1 ),\n", + " } )\n", + "pd.DataFrame( rows ).set_index( 'Course' )" + ] + }, + { + "cell_type": "markdown", + "id": "8e817a02", + "metadata": {}, + "source": [ + ">Although relatively few students revise their incorrect answers for glossary comparisons, those who do can demonstrate more thoughtful engagement (though some may simply copy the feedback verbatim). An example from CJ illustrates how a student fixed an initial error through feedback." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "d0172d5e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.652542Z", + "iopub.status.busy": "2026-07-24T02:23:47.652419Z", + "iopub.status.idle": "2026-07-24T02:23:47.655563Z", + "shell.execute_reply": "2026-07-24T02:23:47.654984Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Prompt: Explain the difference between the term Fear of victimization and Secondary victimization:\n", + "\n", + "Attempt 1: Secondary victimization is when sexual assault victims have bad experiences with assistance, thus causing them more trauma. Fear of victimization is when someone is afraid to report something. \n", + "\n", + "Feedback: Your explanation needs some clarification. Secondary victimization refers to additional trauma experienced by victims, often due to negative responses from others or the system. Fear of victimization involves a heightened fear of crime, often influenced by media portrayals, and is not specifically about reporting. It's important to distinguish between the emotional impact of crime and the fear of potential crime.\n", + "\n", + "Attempt 2: Secondary victimization is when a victim has additional trauma, often due to negative responses from others or the system. Fear of victimization involves a heightened fear of crime. \n", + "\n", + "Feedback: Your explanation captures the essence of both terms well. Secondary victimization involves additional trauma from external responses. Fear of victimization relates to the heightened fear of crime, often influenced by media portrayals. Your distinction between the two is accurate.\n" + ] + } + ], + "source": [ + "idx = 1854\n", + "row = glossary_comparison_sessions.loc[ idx ]\n", + "print( f'Prompt: {row.question}' )\n", + "print()\n", + "print( f'Attempt 1: {row.answer}' )\n", + "print()\n", + "print( f'Feedback: {row.feedback}' )\n", + "print()\n", + "print( f'Attempt 2: {row.second_answer}' )\n", + "print()\n", + "print( f'Feedback: {row.second_feedback}' )" + ] + }, + { + "cell_type": "markdown", + "id": "nongenuine-header", + "metadata": {}, + "source": [ + "### 3.4. Non-Genuine Answers" + ] + }, + { + "cell_type": "markdown", + "id": "table4-quote", + "metadata": {}, + "source": [ + ">Table 4 presents the rate of non-genuine answer submissions: responses that appear to have no meaningful engagement with the question prompt. In CJ, glossary comparisons exhibit 8.0% non-genuine attempts, versus 2.8% for standard FITB. In COM, the gap is 14.5% vs. 8.5%.\n", + "\n", + "**Table 4**\n", + "\n", + "Percentage of non-genuine answers by question type.\n", + "\n", + "| | CJ (N = 47) | COM (N = 205) |\n", + "| ------------------- | ----------: | ------------: |\n", + "| Glossary Comparison | 8.0 | 14.5 |\n", + "| Standard FITB | 2.8 | 8.5 |" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "table4", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.656785Z", + "iopub.status.busy": "2026-07-24T02:23:47.656676Z", + "iopub.status.idle": "2026-07-24T02:23:47.679938Z", + "shell.execute_reply": "2026-07-24T02:23:47.679229Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Glossary Comp. %Standard FITB %
Course
CJ8.32.8
COM15.28.3
\n", + "
" + ], + "text/plain": [ + " Glossary Comp. % Standard FITB %\n", + "Course \n", + "CJ 8.3 2.8\n", + "COM 15.2 8.3" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rows = []\n", + "for course, textbook in [ ( 'CJ', CJ_TEXTBOOK ), ( 'COM', COM_TEXTBOOK ) ]:\n", + " gc = glossary_comparison_sessions[ ( glossary_comparison_sessions.course == course ) & ( glossary_comparison_sessions.textbook_id == textbook ) ]\n", + " fb = fitb_sessions[ ( fitb_sessions.course == course ) & ( fitb_sessions.textbook_id == textbook ) ]\n", + " rows.append( {\n", + " 'Course': course,\n", + " 'Glossary Comp. %': round( ( gc.first_attempt == 'x' ).mean() * 100, 1 ),\n", + " 'Standard FITB %': round( ( fb.first_attempt == 'x' ).mean() * 100, 1 ),\n", + " } )\n", + "pd.DataFrame( rows ).set_index( 'Course' )" + ] + }, + { + "cell_type": "markdown", + "id": "nongenuine-second-quote", + "metadata": {}, + "source": [ + ">As a follow-up analysis, it was checked whether more non-substantive answers occur on the first attempt than on subsequent attempts by the same student. If most non-genuine attempts happen immediately, it may indicate students are simply bypassing the task; if they occur later, students could be experiencing confusion or frustration. For both courses, the percentage of non-genuine answers dropped noticeably on the second attempt: 4.9% for glossary comparison and 0.6% for FITB in CJ, and 5.3% and 0.8%, respectively in COM." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "nongenuine-second", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-24T02:23:47.681314Z", + "iopub.status.busy": "2026-07-24T02:23:47.681205Z", + "iopub.status.idle": "2026-07-24T02:23:47.705278Z", + "shell.execute_reply": "2026-07-24T02:23:47.704662Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Glossary 2nd attempt %FITB 2nd attempt %
Course
CJ4.90.6
COM5.90.8
\n", + "
" + ], + "text/plain": [ + " Glossary 2nd attempt % FITB 2nd attempt %\n", + "Course \n", + "CJ 4.9 0.6\n", + "COM 5.9 0.8" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rows = []\n", + "for course, textbook in [ ( 'CJ', CJ_TEXTBOOK ), ( 'COM', COM_TEXTBOOK ) ]:\n", + " gc = glossary_comparison_sessions[ ( glossary_comparison_sessions.course == course ) & ( glossary_comparison_sessions.textbook_id == textbook ) ]\n", + " fb = fitb_sessions[ ( fitb_sessions.course == course ) & ( fitb_sessions.textbook_id == textbook ) ]\n", + " # Second-attempt non-genuine: among sessions with a second attempt, rate of 'x'\n", + " gc2 = gc[ gc.second_attempt.notna() ]\n", + " fb2 = fb[ fb.second_attempt.notna() ]\n", + " rows.append( {\n", + " 'Course': course,\n", + " 'Glossary 2nd attempt %': round( ( gc2.second_attempt == 'x' ).mean() * 100, 1 ),\n", + " 'FITB 2nd attempt %': round( ( fb2.second_attempt == 'x' ).mean() * 100, 1 ),\n", + " } )\n", + "pd.DataFrame( rows ).set_index( 'Course' )" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/csedu-2026/README.md b/csedu-2026/README.md new file mode 100644 index 0000000..2f3bcd4 --- /dev/null +++ b/csedu-2026/README.md @@ -0,0 +1,195 @@ +# LLM-Based Free-Response Tasks Dataset + +This directory contains the dataset and analysis code for our paper: + +Van Campenhout, R., Dittel, J. S., Jerome, B., & Johnson, B. G. (2026). +Extending an automatic question generation pipeline with LLM-based +free-response tasks: An analysis of performance metrics using student +data. In *Proceedings of the 18th International Conference on Computer +Supported Education* (Vol. 1, pp. 26–38). SCITEPRESS. +https://doi.org/10.5220/0014655400004021 + +## Description + +Automatically generated questions have made it possible to scale +formative practice across digital textbooks, offering students frequent +opportunities to reflect and self-assess as they read. Recent advances +in large language models (LLMs) open the door to expanding these +benefits by enabling more cognitively demanding question types, especially +when paired with personalized feedback. + +In fall 2024, two new LLM-enabled question types were introduced into +the existing automatic question generation system used in the VitalSource +Bookshelf platform as part of a pilot study at a major public university. +The first type, exam question writing, asks students to compose their own +exam questions for a section of the textbook they have just read, +receiving immediate LLM-generated feedback on the quality of their +question. The second type, glossary comparison, asks students to explain +the difference between two closely related terms drawn from the textbook +glossary, again receiving personalized LLM feedback on their answer. +Conventional fill-in-the-blank (FITB) questions served as a benchmark +for comparing engagement, difficulty, persistence, and non-genuine answer +rates. + +The questions appear in a panel next to the textbook content as students +read, with immediate feedback on each attempt. Figure 1 shows an example +FITB question (left) alongside a glossary comparison (right). + +

+A FITB formative practice question in a communications textbook (left) and a glossary comparison (right). +
Figure 1. A FITB formative practice question in a communications textbook (left) and a glossary comparison (right). +

+ +The pilot covered three university courses at Iowa State University +during the fall 2024 semester (August 15–December 26, 2024): CJ 4060 +(Women, Gender, and Crime; 47 students; textbook: Mallicoat, 2023), +COMST 1010 (Communication in Everyday Life; 205 students; textbook: +Duck & McMahan, 2021), and HDFS 2700 (Family Communication and +Relationships; 64 students). This dataset includes only CJ and COMST 1010, since they are the two +courses with both new question types. HDFS 2700's textbook lacked a +glossary, so glossary comparison questions could not be generated for +it; it is therefore excluded from this dataset entirely, so that the +same set of courses is used for every question type compared here. + +The dataset focuses on the two LLM-enabled question types and FITB as a +baseline. It includes session-level summaries for each student-question +pair, along with the text of student answers and LLM-generated feedback +for the open-ended question types. + +The primary research questions for this study were: + +1. Do LLM-enabled question types achieve similar engagement rates to + standard FITB questions? +2. How do glossary comparison questions compare to FITB in difficulty + and persistence? +3. What is the prevalence of non-genuine answers across question types? + +Descriptive statistical analysis was employed to explore these +questions, providing insights into student learning behaviors in the +context of LLM-enabled open-ended practice. + +Further methodological details and analysis results can be found in the +paper above. + +## Data Files and Analysis Code + +The provided files are: + +| File | Description | +| ---------------------------------------------- | -------------------------------------------------------------- | +| `exam_writing_sessions.parquet` | Student-question sessions for exam question writing | +| `glossary_comparison_sessions.parquet` | Student-question sessions for glossary comparison questions | +| `fitb_sessions.parquet` | Student-question sessions for FITB questions | +| `LLM-Based Free-Response Tasks Analysis.ipynb` | Jupyter notebook for replication of data analysis in the paper | + +In `glossary_comparison_sessions.parquet` and `fitb_sessions.parquet`, +student answer attempts are classified using shorthand symbols. These +symbols (`+`, `-`, `x`) are not used in the paper but correspond to the +categories defined in the following table: + +| Category | Symbol | Description | +| ----------- | ------ | ------------------------------------------------------------------------------------------------------- | +| Correct | `+` | The response accurately addressed the key distinction between terms. | +| Incorrect | `-` | The response did not sufficiently answer the question, despite appearing to be a genuine effort. | +| Non-Genuine | `x` | The response did not constitute a legitimate attempt (e.g., random characters, "idk", irrelevant text). | + +A student rating may also appear in any session's `pattern` field, appended +wherever the student rated the question: `F` for a thumbs up (:+1:) or +`f` for a thumbs down (:-1:). + +The session fields for exam question writing are: + +| Field | Type | Definition | +| ----------------- | ----------- | ---------------------------------------------------------------------------------------- | +| `question_id` | string | Unique identifier for question | +| `student_id` | string | Anonymized student identifier | +| `course` | categorical | Course identifier (`CJ` or `COM`) | +| `textbook_id` | string | Textbook unique identifier | +| `question` | string | Text of question prompt | +| `pattern` | string | Concatenation of symbols recording the student's actions on the question | +| `answer` | string | Text of student's first answer attempt | +| `feedback` | string | LLM-generated feedback on student's first answer attempt | +| `second_answer` | string | Text of student's second answer attempt (missing if no second attempt) | +| `second_feedback` | string | LLM-generated feedback on student's second answer attempt (missing if no second attempt) | + +Unlike glossary comparison, exam question writing has no definable +correctness criterion: students are composing their own novel exam +question rather than answering one with a single correct response, so +there is no target answer an LLM judge can score against. Exam question +writing attempts are therefore not classified `+`/`-`/`x`. Instead, the +exam `pattern` field uses the following symbols: + +| Action | Symbol | Description | +| ------- | ------ | -------------------------------------------------------------- | +| Attempt | `e` | The student submitted an exam question writing answer attempt. | + +The session fields for glossary comparison are: + +| Field | Type | Definition | +| ----------------- | ----------- | ---------------------------------------------------------------------------------------- | +| `question_id` | string | Unique identifier for question | +| `student_id` | string | Anonymized student identifier | +| `course` | categorical | Course identifier (`CJ` or `COM`) | +| `textbook_id` | string | Textbook unique identifier | +| `question` | string | Text of question prompt | +| `pattern` | string | Concatenation of symbols recording the student's actions on the question | +| `first_attempt` | categorical | Category of first answer attempt (`+`, `-`, `x`) | +| `second_attempt` | categorical | Category of second answer attempt (missing if no second attempt) | +| `answer` | string | Text of student's first answer attempt | +| `feedback` | string | LLM-generated feedback on student's first answer attempt | +| `second_answer` | string | Text of student's second answer attempt (missing if no second attempt) | +| `second_feedback` | string | LLM-generated feedback on student's second answer attempt (missing if no second attempt) | + +In addition to the symbols above, the FITB `pattern` field encodes the +following symbols: + +| Action | Symbol | Description | +| ----------------- | ------ | ------------------------------------------------------------------------------- | +| Reveal | `r` | The student revealed the correct answer instead of submitting an attempt. | +| Answer suggestion | `a` | A spelling-correction answer suggestion was offered for the student's response. | + +Unlike the exam and glossary files, `fitb_sessions.parquet` does not +include a `question` prompt-text column. The session fields for FITB are: + +| Field | Type | Definition | +| ---------------- | ----------- | ------------------------------------------------------------------------ | +| `question_id` | string | Unique identifier for question | +| `student_id` | string | Anonymized student identifier | +| `course` | categorical | Course identifier (`CJ` or `COM`) | +| `textbook_id` | string | Textbook unique identifier | +| `pattern` | string | Concatenation of symbols recording the student's actions on the question | +| `first_attempt` | categorical | Category of first answer attempt (`+`, `-`, `x`) | +| `second_attempt` | categorical | Category of second answer attempt (missing if no second attempt) | + +## Reproducibility Note + +Session counts, question/student/event counts, and Table 1 reproduce +exactly, since they do not depend on any correctness or genuineness +scoring. FITB questions are additionally scored automatically, so FITB-derived +values in Tables 2–4 also reproduce exactly, with one exception noted +below. + +Glossary comparison questions have a definable correctness criterion (a +specific distinction between two glossary terms), so their attempts are +additionally scored (`+`, `-`, `x`) as a research-only, post-hoc measure +using an LLM judge (GPT-4o mini). This scoring is not perfectly +deterministic across runs, so the glossary comparison values computed +here for Tables 2–4 may differ from the published results by less than +one percentage point; this reflects scoring variance on the free-response +glossary items and does not affect the underlying session data. + +Separately, the FITB non-genuine answer classifier logic was refined +after the paper's analysis. As a result, COM's FITB non-genuine rate in +Table 4 differs from the published value by less than one percentage +point; CJ is unaffected. All other FITB-derived values reproduce exactly. + +## Acknowledgments + +We thank SAGE Publications, Inc. for granting permission to include +student use of the open-ended questions in their textbooks in this +dataset. + +## Contact Us + +If you have questions, please feel free to email +[benny.johnson@vitalsource.com](mailto:benny.johnson@vitalsource.com). diff --git a/csedu-2026/exam_writing_sessions.parquet b/csedu-2026/exam_writing_sessions.parquet new file mode 100644 index 0000000..64164c3 --- /dev/null +++ b/csedu-2026/exam_writing_sessions.parquet @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:789f0987318f64b98848a7dccc52a389ccf7e335f9af43b73748f0b99c71e248 +size 1562775 diff --git a/csedu-2026/fitb_sessions.parquet b/csedu-2026/fitb_sessions.parquet new file mode 100644 index 0000000..9e9122d --- /dev/null +++ b/csedu-2026/fitb_sessions.parquet @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2c7bed95314ea9ff2000bf66d36687e1184ea30967e810a2d47b19db57d6fa93 +size 99549 diff --git a/csedu-2026/glossary_comparison_sessions.parquet b/csedu-2026/glossary_comparison_sessions.parquet new file mode 100644 index 0000000..e578f51 --- /dev/null +++ b/csedu-2026/glossary_comparison_sessions.parquet @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cdad25a704d35a5400259c68cdd7456d3f1f882f7810d20114d9ac681a4e714f +size 896300 diff --git a/csedu-2026/questions_screenshot.png b/csedu-2026/questions_screenshot.png new file mode 100644 index 0000000..36b8845 Binary files /dev/null and b/csedu-2026/questions_screenshot.png differ