Improving AI Text Classification: A Cascaded Approach for Educational Grading

Related Publication: Improving AI Text Classification: A Cascaded Approach

Full Paper

This blog post describes research presented at the 3rd Workshop on Engineering Interactive Systems Embedding AI Technologies in Trier, Germany (June 2025).

What We Did

This work focused on improving the reliability of AI-assisted grading for open-ended student answers. While Large Language Models (LLMs) are flexible and easy to repurpose, they still struggle with consistent and trustworthy classification in high-stakes educational tasks such as grading. In particular, they tend to produce fluent output while still making classification mistakes, and they often avoid assigning low grades such as “incorrect.”

The core goal was to engineer a grading pipeline that is both more reliable and more usable in practice. We wanted a system that could:

• classify short answers at the rubric level more accurately,
• improve detection of incorrect answers,
• preserve the flexibility of LLMs for feedback generation,
• and keep teachers in control through a transparent review interface.

The specific grading setup used three labels: Incorrect, Partially correct, and Correct. Rather than asking a single model to both decide the grade and generate feedback, we separated these tasks. The classification step is handled first by dedicated transformer models, and only then is the result passed to an LLM-based feedback layer. This makes the feedback generation more grounded, because the LLM no longer has to infer the grade by itself.

A major motivation behind this work was that existing approaches were not sufficient for dependable educational use. A generic LLM baseline (Gemma 3 27B) achieved only 58% accuracy on the evaluation set and showed a clear bias toward marking answers as correct, identifying only 21% of incorrect answers. That kind of failure mode is especially problematic in grading, where missing incorrect answers undermines both fairness and usefulness.

How We Did It

We designed an end-to-end approach with two main parts: a transformer cascade for rubric-level classification and a Mixture-of-Agents (MoA) LLM system for generating feedback.

1. Transformer Cascade for Classification

Instead of training a single model to directly predict all three grading labels at once, we implemented a cascaded architecture using fine-tuned BERT-base-uncased models.

The cascade works in two stages:

• The first transformer performs a binary classification: Correct vs. Not correct
• If an answer is classified as Not correct, it is passed to a second transformer
• The second transformer then distinguishes between Incorrect and Partially correct

This decomposition turns one harder three-class problem into two simpler decisions. That matters because the grading labels are imbalanced and adjacent categories share overlapping features. In the single-model setup, the classifier often confused Correct and Partially correct and was hesitant to label answers as Incorrect. By isolating clearly correct answers first, the cascade can focus the second stage on a more balanced subset, making it easier to distinguish the weaker responses.

For input, we supplied both:

• the student answer
• and a reference answer

Although we aimed to keep required metadata minimal, adding the reference answer produced a substantial performance boost. That led to an important practical conclusion: the extra effort of creating reference answers is justified if it significantly improves grading accuracy.

2. Mixture-of-Agents LLM System

After classification, the assigned label and the original student input are combined into a prompt for a Mixture-of-Agents feedback system.

The purpose of this second layer is not to decide the grade, but to generate clearer and more appropriate feedback based on the already-determined classification. This separation improves alignment between the grade and the explanation. Instead of relying on an LLM to both judge and justify at once, the MoA setup uses the classification result as structured guidance for the feedback stage.

To make this usable for teachers, we paired the feedback mechanism with a traffic-light interface:

• Green for fully correct
• Yellow for partially correct
• Red for incorrect

In the prototype, this idea is applied to programming assignments line by line. Each code line is highlighted with one of the traffic-light colors, and the system generates comments that include reinforcement for correct parts and constructive suggestions for lines that need improvement. The teacher can then accept, revise, or override that feedback.

Evaluation

We evaluated the approach on the ASAG (Automatic Short Answer Grading) dataset, using 130 student responses labeled into the three rubric categories. We compared three systems:

• a generic LLM baseline (Gemma 3 27B),
• a single transformer trained for direct three-class classification,
• and the proposed cascade model.

The LLM baseline was included as a realistic comparison point because it represents the kind of flexible model many people might want to use directly for grading. The single transformer served as the strongest non-cascaded baseline using the same encoder family. Performance was evaluated with standard classification metrics: precision, recall, F1-score, and overall accuracy.

Results

The cascaded approach produced a clear improvement over both baselines.

Compared to the single transformer, the cascade:

• increased recall for Incorrect answers from 0.26 to 0.58
• increased precision for Correct answers from 0.70 to 0.84
• improved overall accuracy from 0.65 to 0.82

This means the cascade more than doubled the system’s ability to correctly identify incorrect answers, while also becoming more precise when marking answers as fully correct. That is exactly the kind of tradeoff improvement that matters in grading: stronger detection of weak answers without becoming overly harsh on strong ones.

The full results for the cascade were:

• Incorrect: Precision 0.85, Recall 0.58, F1 0.69
• Partially Correct: Precision 0.77, Recall 0.77, F1 0.77
• Correct: Precision 0.84, Recall 0.91, F1 0.87
• Overall accuracy: 0.82

By contrast:

• the LLM baseline achieved only 0.58 accuracy and showed a strong bias toward predicting Correct,
• the single transformer improved on that with 0.65 accuracy, but still struggled with confusion between adjacent labels.

The confusion matrices in the paper illustrate this clearly: the LLM rarely labeled answers as incorrect, the single transformer still made many off-diagonal errors between Correct and Partially correct, and the cascade produced a much cleaner diagonal with fewer misclassifications.

Prototype Implementation

Beyond the classification experiment, we also developed a prototype for semi-automatic grading.

The prototype was designed to fit into existing teaching workflows with minimal disruption:

• it visually marks answer quality using a traffic-light metaphor,
• it generates first-draft feedback for the teaching staff,
• and it keeps a human-in-the-loop so staff can review and revise both the grading and the comments before finalizing them.

For programming tasks, the interface analyzes code line by line. This gives teachers a more granular overview than grading an entire submission as a single block. The color-coding helps direct attention to the most important problem areas, while the auto-generated comments reduce the amount of feedback that must be written from scratch. The result is a workflow that aims to save time without hiding the reasoning process from the teacher.

Conclusion

This work shows that better AI grading is not just about using a larger model. Careful system design matters. By splitting rubric prediction into a cascaded transformer architecture and separating that from an LLM-based feedback layer, we were able to build a pipeline that is more accurate, more transparent, and better suited to educational use than a direct one-model approach.

Three lessons stood out most clearly:

1. Cascaded architectures improve reliability Breaking the grading task into simpler decisions reduced confusion between adjacent classes and significantly improved detection of incorrect answers.

2. Transparency matters for adoption The traffic-light interface and editable feedback make the system easier for teachers to understand and trust.

3. AI works best as support, not replacement Keeping the teacher in control ensures that the system augments professional judgment instead of obscuring it.

A key limitation is that the current validation is still preliminary: it was performed on a relatively small dataset from a single course context. Future work should test whether the same benefits hold for longer answers and other modalities such as code, mathematics, or other domain-specific responses.

Future Directions:

• Extending the cascade to longer and more complex responses
• Evaluating the approach across different subjects and answer modalities
• Studying how teachers use and adapt the feedback in real grading workflows
• Exploring richer semi-automatic interfaces that further reduce workload while preserving oversight

This blog post is based on research presented at the 3rd Workshop on Engineering Interactive Systems Embedding AI Technologies, 2025. For complete technical details, see the full paper.