AICG Framework: A New Paradigm for AI-Aware Cognitive Grading of Short-Answer Math Questions

Introduction: The Dilemma of Automated Grading for Short-Answer Math Questions

Transformer-based language models have made significant progress in automated short-answer grading (ASAG), but existing methods still struggle to accurately understand, reason about, and evaluate complex math answers involving symbols and formulas. More critically, these score-only methods cannot distinguish students with identical scores but vastly different cognitive states, and they also overlook the upward scoring bias potentially introduced by generative AI usage. To address these limitations, researchers propose the AICG (AI-aware Cognitive-supervised Grading) framework, which integrates cognitive supervision and AI detection into the grading pipeline through a dual-branch multi-task architecture, opening a new path for automated grading of short-answer math questions.

Dual-Branch Architecture: From Score Labels to Cognitive Supervision

The core innovation of AICG lies in extending grading from single score prediction to multi-task learning. The first branch is based on RoBERTa and integrates regression, ordinal regression, and classification heads for scoring individual answers. The second branch uses an expert-designed Q-matrix to associate answers with the skills being assessed, and jointly predicts skill mastery and learning gains derived from the G-DINA model. The two branches share an encoder, allowing skill-level objectives to provide cognitive supervision beyond score labels for answer grading. This design enables the model not only to know "what score to assign" but also to understand "what the student has mastered."

Q-Matrix-Guided Cognitive Supervision

Ablation experiments show that Q-matrix-guided cognitive supervision is the primary source of performance improvement. By mapping answers to specific skills, the model can capture differences in cognitive states underlying different responses, thereby alleviating the generalization dilemma caused by reliance on score labels alone. On a dataset covering 19 semesters and 49,402 teacher-graded responses, AICG reduced the mean absolute error (MAE) by 50.3% relative to RoBERTa regression, and increased the quadratic weighted kappa (QWK) from 0.798 to 0.927. This result highlights the value of cognitive supervision in complex math grading.

The G-DINA model (Generalized Deterministic Input, Noisy "And" gate) is a core class of models in cognitive diagnostic assessment (CDA), used to infer students' mastery states of fine-grained skills from their response patterns. It treats each item as a joint function of several skills and characterizes response noise through "guess" and "slip" parameters, thereby estimating skill mastery probabilities on binary or graded responses. Unlike item response theory, which outputs only total scores, G-DINA can provide skill-level diagnostic profiles—this is precisely the statistical foundation for the second branch of AICG to obtain cognitive supervision signals.

The Q-matrix is a knowledge mapping table connecting items and skills, with rows corresponding to items and columns to skills, where matrix elements indicate whether an item assesses a certain skill. It is a prerequisite for the identifiability of cognitive diagnostic models and is typically manually designed by subject experts. The quality of the Q-matrix directly determines the reasonableness of skill mastery estimation; therefore, in AICG, the Q-matrix serves both as the semantic bridge for cognitive supervision and as one source of model interpretability.

Ordinal regression is a modeling method for ordered categorical labels, with common implementations including cumulative link models and deep variants such as CORAL and CORNLoss. Scores on short-answer math questions are inherently ordinal (e.g., 0 < 1 < 2); treating them as ordinary classification would lose distance information between categories, while treating them as regression would ignore the discrete ordinal structure of scores. Ordinal regression learns a bias for each threshold, leveraging ordinal information while maintaining order constraints, and thus often fits the distribution of grading labels better than pure regression or pure classification in this task.

AI-Aware Weighting: Suppressing Scoring Bias Introduced by Generative AI

The proliferation of generative AI has brought new challenges to learning assessment: students may use AI to generate answers, leading to upward scoring bias. AICG introduces a Gen-AI detector that estimates auxiliary probabilities through supervised contrastive pre-training and linear classification, and converts them into reliability weights. These weights attenuate unreliable cognitive evidence during the learning assessment process, thereby reducing the interference of AI assistance with grading accuracy. Experiments show that AI-aware weighting further reduces overall MAE by 4.2% on top of the unweighted cognitive model, and consistently improves MAE and QWK across the four most recent post-Gen-AI semesters. This mechanism provides a practical tool for learning assessment in the AI era.

Supervised contrastive pre-training is a representation learning method that pulls embeddings of samples from the same class closer and pushes those from different classes farther apart, thereby learning a more discriminative and intra-class compact feature space. Applied to the Gen-AI detector, it enables the detector to obtain a more stable decision boundary between AI-generated and human-written text with limited labeled data. A linear classification head then estimates auxiliary probabilities; this combination of "pre-trained representation + lightweight classifier" is typically more robust than end-to-end fine-tuning in small-sample, distribution-shift scenarios, and also facilitates converting detector outputs into interpretable reliability weights.

Experimental Validation and Insights

The study conducted extensive experiments on 49,402 teacher-graded responses covering 19 semesters. Beyond overall performance gains, ablation experiments confirmed the dominant role of Q-matrix-guided cognitive supervision, while AI-aware weighting serves as a supplement that further enhances model robustness in AI-pervasive environments. Interestingly, AICG's stable performance across the four most recent post-Gen-AI semesters indicates that its design can adapt to assessment scenarios after the widespread adoption of generative AI.

Implications for Educational Assessment

The significance of the AICG framework lies not only in improved grading accuracy but also in its integration of cognitive diagnosis and AI detection into a unified framework, providing a more comprehensive perspective for automated grading. For educational technology developers and researchers, this work suggests that future grading systems need to simultaneously consider answer quality, student cognitive states, and the potential impact of AI assistance. Although AICG currently focuses on short-answer math questions, its dual-branch multi-task and AI-aware weighting ideas are expected to transfer to other subject areas requiring complex reasoning.

Quadratic weighted kappa (QWK) is a commonly used metric for assessing grading agreement. It weights the confusion matrix by the squared distance between rating levels, so that errors of "off by one point" and "off by three points" receive different degrees of penalty, and it corrects for chance agreement. QWK typically ranges from 0 to 1, with values closer to 1 indicating higher ordinal agreement with human grading, making it particularly suitable for evaluating ordinal ratings such as short-answer questions. In contrast, MAE measures the absolute magnitude of score deviation; combining the two reflects both error magnitude and ordinal agreement.

Conclusion

Through cognitive supervision and AI-aware weighting, AICG breaks through the limitation of traditional ASAG relying solely on score labels, achieving significant improvements in MAE and QWK for short-answer math grading. As generative AI permeates educational scenarios, this framework—which balances grading accuracy, cognitive diagnosis, and AI bias suppression—provides an important reference for building smarter and more reliable learning assessment systems.