AICG
A dual-branch multi-task framework for automated scoring of short-answer mathematics questions, combining cognitive diagnostic supervision with generative AI detection to estimate skill mastery and learning gains alongside scoring.
Timeline (last 90 days)
The AICG (AI-aware Cognitive-supervised Grading) framework adopts a dual-branch multi-task architecture that integrates cognitive supervision and AI detection into the grading process.
The first branch of AICG is based on RoBERTa and integrates regression, ordinal regression, and classification heads simultaneously for scoring individual answers.
The second branch of AICG uses an expert-designed Q-matrix to associate answers with the skills being assessed, and jointly predicts skill mastery and learning gains derived from the G-DINA model.
AICG introduces a Gen-AI detector that estimates an auxiliary probability through supervised contrastive pretraining and linear classification, and converts it into a reliability weight.
On a dataset covering 19 semesters and 49,402 teacher-graded responses, AICG reduces mean absolute error (MAE) by 50.3% relative to RoBERTa regression and improves quadratic weighted Kappa (QWK) from 0.798 to 0.927.
Ablation experiments show that Q-matrix-guided cognitive supervision is the main source of AICG's performance improvement.
AI-aware weighting further reduces overall MAE by 4.2% on top of the unweighted cognitive model, and consistently improves MAE and QWK in the most recent four post-Gen-AI semesters.
All Facts (7)
The AICG (AI-aware Cognitive-supervised Grading) framework adopts a dual-branch multi-task architecture that integrates cognitive supervision and AI detection into the grading process.
50%UnverifiedThe first branch of AICG is based on RoBERTa and integrates regression, ordinal regression, and classification heads simultaneously for scoring individual answers.
50%UnverifiedThe second branch of AICG uses an expert-designed Q-matrix to associate answers with the skills being assessed, and jointly predicts skill mastery and learning gains derived from the G-DINA model.
50%UnverifiedAICG introduces a Gen-AI detector that estimates an auxiliary probability through supervised contrastive pretraining and linear classification, and converts it into a reliability weight.
50%UnverifiedOn a dataset covering 19 semesters and 49,402 teacher-graded responses, AICG reduces mean absolute error (MAE) by 50.3% relative to RoBERTa regression and improves quadratic weighted Kappa (QWK) from 0.798 to 0.927.
50%UnverifiedAblation experiments show that Q-matrix-guided cognitive supervision is the main source of AICG's performance improvement.
50%UnverifiedAI-aware weighting further reduces overall MAE by 4.2% on top of the unweighted cognitive model, and consistently improves MAE and QWK in the most recent four post-Gen-AI semesters.
50%