This blog post describes research presented at HHAI 2026. Co-authored with Yarne Dirkx, Davy Vanacken, and Gustavo Rovelo Ruiz.

What We Did

Educators today are largely stuck choosing between two extremes for GenAI: ban it, and risk losing real pedagogical opportunities, or allow it, and risk students leaning on it so much that foundational skills like writing, critical thinking, and problem-solving never fully develop. Surveys of top university guidelines show that the vast majority already push instructors toward course-specific GenAI rules, but most of these rules still address only surface-level concerns like plagiarism, and apply the same policy to every student in a course regardless of where they are in their learning journey.

That last part is the gap we focus on. Decades of research on scaffolding tell us that effective support should adapt as a student progresses, staying within their Zone of Proximal Development (ZPD): challenging enough to be useful, but not so far beyond their current ability that it becomes just doing the work for them. A one-size-fits-all GenAI policy can’t do that. A student who has already mastered a skill and a student who has never attempted it are held to the exact same boundary, even though research shows the same tool can help one and hurt the other.

So instead of asking “Can this student use GenAI?”, a binary question that ignores where the student actually is, we ask: what interactions with GenAI serve this student’s growth right now? This paper proposes a reconceptualization: adaptive boundaries, GenAI constraints that co-evolve with a student’s demonstrated competency, rather than static rules set once at the course level.

Our Framework: Adaptive Boundaries as Pedagogical Scaffolds

A Three-Party Negotiation

We frame adaptive boundaries as a human-AI collaborative negotiation between three parties:

1. Students, who signal competency through their learning behaviors — for example, the depth of questions they ask a GenAI system, patterns in how they complete tasks, or their ability to critically evaluate GenAI output.
2. Instructors, who define learning goals and pedagogical milestones — for example, a writing instructor might require students to demonstrate proper argumentation before GenAI-assisted writing is unlocked, while a programming instructor might allow code generation immediately but require students to explain and modify whatever it produces.
3. The GenAI system itself, which mediates between the two by adapting which of its own features are actually available to a given student at a given time.

Instructors set the milestones; the GenAI system then acts as an adaptive layer that scaffolds its own available features based on each student’s detected competency. Critically, this adaptation is bidirectional — boundaries loosen as competency grows, but they can also tighten again during assessments, or if a student’s interaction patterns start to suggest over-reliance rather than growth.

Three Developmental Stages

Drawing on established models of skill acquisition (the Dreyfus model) and mastery learning (Bloom), we describe three stages where GenAI plays a fundamentally different role:

• Novice stage — most restrictive. Early, unrestricted GenAI access risks undermining exactly the foundational skills a student is trying to build, and the same tool can help one student while hurting another, so the system first needs to build a model of the student before personalizing anything. At this stage, GenAI should expose its reasoning, require metacognitive reflection before answering, and only give direct solutions after a genuine attempt has been made.
• Intermediate stage — more flexible, but monitored. Boundaries loosen enough to target specific competency gaps, but the main risk shifts to over-reliance: students offloading cognitive work onto GenAI instead of building their own skills. The system should watch for signs like passive help-seeking without articulating the underlying knowledge gap, or uncritical acceptance of AI output, and temporarily tighten access if those patterns show up.
• Advanced stage — largely open. GenAI functions closer to a professional tool or collaborator. Even here, boundaries tighten again during assessments where students must demonstrate independent competency, and — importantly — boundaries should include aspirational scaffolds that push slightly beyond what’s already been demonstrated, so a competency-gated system doesn’t accidentally cap a student’s growth at what they’ve already shown.

Together, these stages turn what looks like an access-control policy into an actual pedagogical instrument that grows with the student.

Why This Is Feasible Now

We didn’t want this to be purely speculative, so a large part of the paper argues that the pieces needed to build adaptive boundaries already exist — just not combined yet.

On the pedagogical side, the ZPD and scaffolding theory already establish why support should shrink as competency grows; mastery learning already establishes competency-based (rather than time-based) progression; and Intelligent Tutoring Systems already prove that fine-grained, competency-based adaptation works at scale. Adaptive boundaries extend that logic in two ways: instead of adapting content inside a closed, instructor-authored system, they adapt a student’s access to an external tool used across tasks and courses, and instead of treating the tutoring system as the adaptive instrument, they treat governance itself as the thing that adapts.

On the technical side, learning analytics research has already shown that interaction data — clickstreams, process mining, language analysis of student-GenAI conversations, and classifiers trained on interaction patterns — can reliably surface competency signals. Meanwhile, mainstream GenAI platforms (ChatGPT, Claude, Ollama) already support the kind of personalized, configurable behavior adaptive boundaries need through system prompts, meaning a single GenAI system could plausibly implement all three stages via prompting rather than requiring bespoke infrastructure per stage.

So the missing piece isn’t a new capability — it’s the conceptual integration of governance and configuration with developmental scaffolding: boundaries that are student-specific, evidence-driven, and designed to evolve as competency changes.

Future Research Directions

We lay out four research questions that we think need to be answered before adaptive boundaries can move from concept to reality:

• RQ1 — Negotiation: How should boundaries be negotiated among students, teachers, and GenAI systems, balancing teacher authority, student agency, and equity vs. personalization?
• RQ2 — Detection: What interaction patterns reliably indicate a student is actually ready for a boundary shift, without oversimplifying (e.g., mistaking a short-term performance spike for real understanding)?
• RQ3 — Transparency: How can the logic behind a boundary change be made understandable to students, teachers, and administrators alike, without creating cognitive overload or over-explaining?
• RQ4 — Evaluation: How do we tell whether adaptive boundaries actually work — not just immediate task performance, but durable skill retention, appropriate reliance on GenAI, and student agency — evaluated against both no-GenAI and static-boundary baselines, not just each other?

These questions also surface real risks worth naming up front: competency signals may not work equally well across different student backgrounds, “readiness” is hard to standardize across disciplines, and a boundary that scaffolds one student might quietly cap another.

Conclusion

The core argument of this paper is that GenAI governance in education doesn’t have to be a binary policy decision applied the same way to everyone. The pedagogical theory (ZPD, scaffolding, mastery learning) and the technical capability (learning analytics, configurable GenAI) to do better already exist — what’s missing is putting them together into boundaries that are student-specific, evidence-driven, and built to evolve.

If we extend this work further, the most important next steps are the four research questions themselves: running participatory sessions to understand how negotiation should actually work, testing which interaction signals genuinely predict readiness, prototyping transparency mechanisms that scale from simple to in-depth explanations, and designing the mixed-methods, multi-baseline studies needed to evaluate whether any of this actually improves learning outcomes.

Ultimately, we want GenAI in education to stop being a fixed policy line and start being what scaffolding always was supposed to be: something that knows when to help, when to challenge, and when to step back.

This blog post is based on research presented at HHAI 2026. For complete details, see the full paper.

This blog post describes research presented at the EICS 2025 International Workshops and Doctoral Consortium.

What We Did

This work focused on a practical problem: many people want to use machine learning for their own domain tasks, but most AutoML tools are still not truly designed for beginners. Existing systems often automate model search and optimization, yet they leave major gaps in usability, understanding, and end-to-end workflow support. As a result, novices may be able to click through a pipeline, but still not understand what the system is doing, when to trust it, or how to use the trained model safely afterward.

The goal of this research was to design and evaluate a more trustworthy form of automation for novice users. Instead of focusing on peak algorithmic performance, we focused on building an AutoML workflow that helps users successfully create a working model, understand the main steps, and maintain a sense of control throughout the process.

More specifically, the project had three core contributions:

1. An abstract AutoML pipeline for novices We proposed an end-to-end pipeline covering the full process from data intake to inference, rather than only automating isolated parts of model training.

2. A prototype implementation called NovaClass To examine that pipeline in practice, we built a novice-oriented system for Transformer-based text classification that makes advanced model training more accessible through a guided interface.

3. A user study and resulting design principles We evaluated the prototype with 24 participants and used the results to derive four design principles for future AutoML tools aimed at novices: first-model success, explanations, context-aware abstractions, and predictability through safeguards.

This problem matters because domain experts increasingly want to apply modern AI to their own work—whether that means grading open-ended answers, analyzing text, or building task-specific classifiers—but they are often blocked by programming requirements, framework complexity, and fragile configuration steps. If AutoML tools are going to make AI genuinely more accessible, they need to support not just automation, but also trust, usability, and learning.

How We Did It

1. Designing an Abstract Pipeline

We first designed an abstract end-to-end AutoML pipeline specifically aimed at novice users. The paper frames this pipeline as six connected stages:

1. Data Intake and Upfront Safeguards The system begins by ingesting data and validating it early. This includes identifying issues like missing values, checking labels, and catching common problems before training starts. The idea is to reduce avoidable failures and make the system more robust from the start.

2. Key Model Parameter Configuration Rather than exposing every technical setting, the system only shows the parameters that matter most for novice decision-making—such as selecting which columns should be used as input and which column should be predicted. Other settings can be handled automatically using safe defaults.

3. One-Click Training Once the essential choices are made, the rest of the pipeline is automated. This includes preprocessing, train/validation/test splitting, label conversion, and other steps that usually require technical knowledge.

4. Simplified Results After training, the system presents performance in a simplified way so users can understand outcomes without needing deep ML expertise. The emphasis is on interpretability and clarity rather than only raw metrics.

5. Save Model and Metadata In addition to saving the trained model, the system stores metadata about the training configuration. This creates an inference “contract” that helps the system remember how the model expects data later.

6. Auto-Configured Inference Finally, the saved metadata is used to automatically configure the prediction interface so that users can apply the model consistently without manually rebuilding the setup.

This pipeline was intentionally designed around reliability and novice comprehension, not maximum benchmark performance. The core idea is that a good beginner tool should make it easy to reach a safe and usable baseline first.

2. Building a Prototype: NovaClass

To put the abstract pipeline into practice, we implemented NovaClass, a prototype system for novice-friendly automation of Transformer-based text classification. We chose text classification because it is widely useful—for example in grading, spam detection, and sentiment or emotion analysis—while still being complex enough that current Transformer workflows are difficult for non-experts to configure manually.

NovaClass includes several key features:

• Dataset inspection at upload time Users upload a CSV file, and the system highlights column types, missing rows, label balance, and basic statistics. Users can also inspect class distributions and preview the first rows of the dataset. This helps users understand their data before training begins.

• A simplified configuration interface The interface only exposes decisions novices are expected to understand, especially which columns contain the input text and which column contains the target labels. On page 6 of the paper, the interface is shown with configuration options on the left and an integrated contextual assistant on the right, designed to explain choices in plain language.

• Safe defaults and automated training The system runs a reproducible pipeline with sensible defaults, aiming to help users produce a working classifier on their first attempt. Metadata is saved automatically to reduce mismatches later during inference.

• One-toggle cascade classification NovaClass also includes a simplified way to use a more advanced strategy: cascade classification. Instead of training a single multi-class model, the system can automatically decompose the task into a hierarchy of binary classifiers. On page 7, the paper shows how this is presented in the interface as a one-toggle option, making a more advanced technique accessible without additional configuration burden.

• Metadata-driven inference The inference interface uses saved metadata such as label order, encoders, and strategy so that users can apply the model without manually reconstructing the training setup. The system also shows both the predicted label confidence and full class-probability distribution.

• A conversational assistant NovaClass includes a context-aware assistant that explains metrics like accuracy, recall, and F1-score, and suggests next steps based on what the user is currently doing. The prototype uses IBM Granite 3.3 8B because it followed instructions well in testing, has a large context window, and is small enough to run locally for privacy-sensitive use cases.

3. Conducting a User Study

We evaluated NovaClass through a 24-participant study designed to assess usability, trust, and understanding. The study included users with different levels of prior ML experience and compared novices against participants who had previously trained models.

Participants completed three tasks of increasing complexity:

1. Task 1: Binary text classification Participants configured and trained a classifier for fake news detection. This tested whether the guided configuration helped them choose the right input and label columns.

2. Task 2: Cascade classification Participants trained a cascaded classifier for e-commerce description categorization. This tested whether the interface could make a more advanced classification strategy approachable.

3. Task 3: Diagnosis task Participants analyzed results to identify a class imbalance issue in a dataset. This tested whether the available tools helped users interpret model performance and diagnose problems.

To measure outcomes, we collected:

• demographic and background information,
• PAILQ-6 scores for perceived AI literacy,
• a questionnaire on trust and understandability based on prior AutoML trust research,
• and the User Experience Questionnaire (UEQ).

Results

The study showed that the system worked well as a usable end-to-end workflow, but it also revealed an important gap between task completion and genuine understanding.

Success Metrics

The strongest result was that all 24 participants successfully trained a working binary classifier and a working cascaded classifier. That means the core workflow succeeded in making advanced Transformer-based classification accessible enough for everyone in the study to complete the main tasks.

More detailed task outcomes were:

• Task 1: All participants trained a functioning fake news classifier. Also, 21 of 24 participants (87.5%) reported at least some confidence in training the binary model afterward.
• Task 2: All participants correctly trained a cascade model, and 22 of 24 participants (91.7%) correctly identified the weakest-performing stage in the cascade.
• Task 3: 17 of 24 participants (70.8%) correctly identified label imbalance as the main issue in the dataset.

These results suggest that the combination of guided configuration, automation, and metadata-driven inference successfully reduced common setup errors that would normally block beginners.

User Experience

Participants rated the system positively across all six UEQ dimensions. The highest scores were:

• Efficiency: 1.917
• Attractiveness: 1.778
• Perspicuity: 1.635

The remaining dimensions were also positive:

• Dependability: 1.375
• Stimulation: 1.458
• Novelty: 1.052

The figure on page 10 shows these ratings against the UEQ benchmark, where the strongest results are in efficiency, attractiveness, and perspicuity, with more moderate but still positive scores for dependability and stimulation.

So from a pure UX perspective, NovaClass was received as effective, appealing, and relatively easy to understand.

Trust and Understanding

The more nuanced finding was that experienced users reported significantly higher trust and understanding than novices. This was not just a minor trend; the difference between the groups was statistically significant for the overall trust/understandability score. The median average rating was 4.05 for experienced users versus 3.64 for novices.

Significant differences also appeared on individual questions, including:

• understanding the tool,
• understanding the overall process,
• understanding the data,
• and understanding model evaluation metrics.

The biggest gap appeared in understanding evaluation metrics: experienced users had a median rating of 5, while novices had a median of 2.5. This is especially important because metrics are central to deciding whether a model is actually good enough to use.

Deployment Confidence

When asked whether they would deploy models trained with NovaClass, 17 of 24 participants (70.8%) said yes. Most justified this with statements about “high accuracy” or “high F1-score.” Meanwhile, the participants who said no often cited transparency concerns, such as not knowing what the model based its decisions on or not understanding how the data had been processed.

This result shows a key tension:

• some users may be too willing to trust strong headline metrics,
• while others hesitate because they lack insight into the model’s reasoning and the system’s hidden steps.

In other words, automation can help users finish tasks, but without better explanations, it does not automatically produce appropriate reliance.

Role of the Conversational Assistant

The conversational assistant played a major role across all tasks. Participants frequently used it:

• to select inputs in Task 1,
• to identify the weak stage in Task 2,
• and to diagnose imbalance in Task 3.

In some cases, it was used as often as or more often than static tools like the classification report or confusion matrix. This suggests that context-aware, plain-language guidance is especially valuable for novice users trying to make sense of ML workflows. At the same time, the paper notes that this also raises the cost of occasional hallucinations or imprecise explanations.

Four Design Principles

Based on the study and relevant theory, we derived four design principles for future AutoML systems targeting novices.

P1: Support First-Model Success The system should make it easy for users to produce a working baseline on their first attempt. Early success raises self-efficacy, builds momentum, and reduces the chance that beginners quit before they see value. In practice, this means safe defaults, aggressive upfront validation, automatic preprocessing, one-click training, and advanced functionality that can still be enabled with minimal friction.

P2: Provide Explanations to Build Mental Models and Appropriate Reliance Metrics alone are not enough. Users need simplified, interpretable explanations of what a score means, how to read a visualization, and what limitations remain. The goal is not just to inform, but to help users avoid both over-trust and under-trust. The study’s deployment results showed exactly why this matters.

P3: Use Abstractions and Context-Aware Assistance to Keep Users in Their Zone of Proximal Development Interfaces should hide unnecessary complexity, but still provide targeted support that helps users perform just beyond what they could do alone. This includes adaptive interfaces, stage-aware assistance, and advanced features that stay easy to access. NovaClass’s assistant is one example of this principle in practice.

P4: Ensure Predictability and Safeguards to Strengthen Perceived Control Trust also depends on consistency and control. Systems should be predictable, validate data and settings before training, generate inference interfaces directly from saved metadata, and prevent train-inference mismatches. These safeguards make the workflow feel safer and reduce invisible failure points.

Conclusion

This work shows that making AutoML accessible to novices is not just a technical automation problem. It is equally a human-computer interaction problem. If a tool only automates the hard parts without helping users understand what is happening, then it may reduce effort, but it will not necessarily build trust, correct mental models, or appropriate confidence.

NovaClass demonstrated that a carefully designed workflow can make advanced tasks like Transformer fine-tuning and cascade classification usable even for beginners: every participant completed the main training tasks, and overall UX ratings were positive. But the study also made clear that successful task completion does not automatically mean deep understanding. Experienced users still understood the workflow, the data, and the metrics better than novices.

The main lesson, then, is that good AutoML design for novices should balance:

• automation so people can get started,
• explanations so they understand what happened,
• assistance so they can keep progressing,
• and safeguards so they remain in control.

If we extend this work further, the most important next steps would be:

• building expertise-adaptive interfaces that reveal more detail as users grow,
• improving explainability so metrics and model behavior are easier to interpret,
• and studying how these systems affect learning and confidence over time through longitudinal evaluations.

Ultimately, the project argues for a more human-centered vision of AutoML: not tools that simply automate model building, but tools that help people become more capable, more informed, and more appropriately confident while using AI.

This blog post is based on research presented at EICS 2025. For complete technical details and evaluation results, see the full paper on arXiv.

This blog post describes research presented at the 31st Recent Advances in Retailing & Consumer Science (RARCS) Conference. Co-authored with Lieve Douce, Stephanie Van De Sanden, Kim Willems, Davy Vanacken, and Gustavo Rovelo Ruiz.

What We Did

This research examined whether Virtual Reality (VR) and haptic feedback can help consumers make better-informed decisions when evaluating products online, especially for products whose qualities are difficult to judge before purchase.

The core problem is that many products are experience products: consumers need to interact with them to properly assess them. In the pre-purchase stage, this is often difficult or impossible. That is especially true when decision-making happens online, where sensory information is limited. In our work, we focused on windows as a concrete example. When building or renovating a house, consumers may struggle to anticipate:

• how smoothly and ergonomically a sliding window opens, and
• whether investing in more insulating glazing will improve indoor comfort enough to justify the extra cost.

The research question was whether combining richer visual information through VR with tactile input through haptic feedback could reduce this uncertainty and improve evaluation, confidence, and satisfaction.

This matters for e-commerce and digital product experiences because online interfaces are often visually rich but sensorily poor. From an HCI perspective, the study explores how multisensory interfaces can support better user judgment by helping people form more vivid and realistic mental representations of products before they buy.

How We Did It

We conducted two experimental lab studies on consumer decision-making about windows. The studies were grounded in mental imagery theory, which suggests that richer sensory input helps consumers form more vivid mental images of a product, making it easier to understand how it would function in real life.

Research Design

Both experiments used a between-subjects 2×2 design:

1. Visual Information

   • Video
   • VR

2. Haptic Information

   • No haptic feedback
   • Simulated haptic feedback

This created four conditions in each experiment, allowing us to compare the separate and combined effects of immersive visualization and touch-based feedback.

The two use cases were:

1. Sliding window system

   • Participants evaluated different window sliding mechanisms.
   • The haptic feedback was delivered through a mechanical force-feedback instrument built to simulate the amount of force needed to open a sliding window.

2. Window glazing insulation

   • Participants evaluated different glazing options, specifically double vs. triple glazing.
   • The haptic feedback was delivered through an off-the-shelf haptic glove, which simulated differences in insulation value.

Study Protocol

Participants were placed in a scenario where they had to evaluate two alternatives:

• either two types of window sliding systems, or
• two types of window glazing.

The total sample consisted of 159 consumers in Belgium:

• Average age: 39.77 years
• SD: 12.34
• Female participants: 100 out of 159 (62.9%)

After interacting with the assigned condition, participants completed measures related to both the medium and the product.

We collected data on:

• mental imagery
• ease of evaluation
• perceived informativeness
• multiple customer experience dimensions
• processing fluency
• satisfaction
• preference

In addition, eye-tracking data were logged during the experiments.

By comparing video versus VR and no haptics versus simulated haptics, we could isolate the contribution of each modality and their combined effect on consumer decision-making.

Technical Implementation

The technical setup included:

• Visual platform: a comparison between standard video-based presentation and immersive VR

• Haptic devices:

  • a custom mechanical force-feedback device for the sliding-window scenario
  • an off-the-shelf haptic glove for the glazing-insulation scenario
• Virtual environment: product evaluation scenarios designed to help participants compare realistic window alternatives
• Data collection: self-report survey measures combined with eye-tracking logs

Rather than simulating a generic virtual store, the system was tailored to realistic home-building and renovation decisions, where consumers must judge functional product characteristics that are hard to assess from standard online content alone.

Results

The results showed clear positive effects of enhanced visual and haptic input on consumer decision-making.

Main Findings

1. Mental Imagery and Understanding

   • VR and haptic feedback improved participants’ ability to form vivid mental images of the product.
   • Richer sensory input made it easier for participants to imagine how the product would work in real life.

2. Ease of Evaluation and Informativeness

   • Participants found the products easier to evaluate when more sensory information was available.
   • VR and haptic feedback increased the perceived informativeness of the experience.

3. Customer Experience and Processing Fluency

   • Enhanced visual and tactile input improved overall customer experience.
   • Participants processed the information more fluently, suggesting that multisensory interaction made the decision task feel more natural and intuitive.

4. Satisfaction

   • Consumers reported greater satisfaction with their decision-making experience when VR and haptic feedback were included.

5. Higher-Investment Choices

   • For more expensive or higher-stakes options—such as triple-glazed windows or advanced sliding systems—haptic feedback played a particularly important role.
   • In these cases, touch-based simulation was more influential in persuading consumers to invest in options that improve comfort, such as smoother sliding mechanisms or better indoor climate performance.

Implications for E-Commerce

The findings suggest that VR and haptic feedback can:

• provide richer product information before purchase
• improve consumers’ ability to evaluate experience products online
• reduce uncertainty in higher-stakes purchase decisions
• increase confidence and satisfaction in digital buying journeys

These technologies appear especially valuable for products where functionality, comfort, or physical interaction matter more than appearance alone. In such cases, adding haptic feedback may be more impactful than visual enhancement alone.

For e-commerce, this points to a future where multisensory interfaces are not just novel experiences, but practical tools for helping consumers make better decisions.

Conclusion

This research demonstrates that combining VR and haptic feedback can meaningfully improve online consumer decision-making by giving consumers richer sensory information before purchase.

Key Takeaways:

1. VR and haptic feedback improve mental imagery, ease of evaluation, perceived informativeness, processing fluency, customer experience, and satisfaction.
2. Haptic feedback is especially important for higher-investment or comfort-related product decisions.
3. Multisensory interaction can help bridge the gap between online and offline product evaluation.

From an HCI perspective, the work reinforces the importance of designing systems that go beyond visual presentation alone. When people can better simulate interaction with a product, they can make more informed, confident, and satisfying choices.

Future Directions:

• Investigating long-term adoption of VR shopping
• Exploring more sophisticated haptic feedback (temperature, texture variation)
• Studying different product categories and consumer segments
• Examining the role of social presence in VR shopping environments
• Cost-effectiveness analysis for e-commerce businesses

This blog post is based on research presented at the 31st RARCS Conference, 2025. For details, see the conference abstract.

This blog post describes research presented at the 3rd Workshop on Engineering Interactive Systems Embedding AI Technologies in Trier, Germany (June 2025).

What We Did

This work focused on improving the reliability of AI-assisted grading for open-ended student answers. While Large Language Models (LLMs) are flexible and easy to repurpose, they still struggle with consistent and trustworthy classification in high-stakes educational tasks such as grading. In particular, they tend to produce fluent output while still making classification mistakes, and they often avoid assigning low grades such as “incorrect.”

The core goal was to engineer a grading pipeline that is both more reliable and more usable in practice. We wanted a system that could:

• classify short answers at the rubric level more accurately,
• improve detection of incorrect answers,
• preserve the flexibility of LLMs for feedback generation,
• and keep teachers in control through a transparent review interface.

The specific grading setup used three labels: Incorrect, Partially correct, and Correct. Rather than asking a single model to both decide the grade and generate feedback, we separated these tasks. The classification step is handled first by dedicated transformer models, and only then is the result passed to an LLM-based feedback layer. This makes the feedback generation more grounded, because the LLM no longer has to infer the grade by itself.

A major motivation behind this work was that existing approaches were not sufficient for dependable educational use. A generic LLM baseline (Gemma 3 27B) achieved only 58% accuracy on the evaluation set and showed a clear bias toward marking answers as correct, identifying only 21% of incorrect answers. That kind of failure mode is especially problematic in grading, where missing incorrect answers undermines both fairness and usefulness.

How We Did It

We designed an end-to-end approach with two main parts: a transformer cascade for rubric-level classification and a Mixture-of-Agents (MoA) LLM system for generating feedback.

1. Transformer Cascade for Classification

Instead of training a single model to directly predict all three grading labels at once, we implemented a cascaded architecture using fine-tuned BERT-base-uncased models.

The cascade works in two stages:

• The first transformer performs a binary classification: Correct vs. Not correct
• If an answer is classified as Not correct, it is passed to a second transformer
• The second transformer then distinguishes between Incorrect and Partially correct

This decomposition turns one harder three-class problem into two simpler decisions. That matters because the grading labels are imbalanced and adjacent categories share overlapping features. In the single-model setup, the classifier often confused Correct and Partially correct and was hesitant to label answers as Incorrect. By isolating clearly correct answers first, the cascade can focus the second stage on a more balanced subset, making it easier to distinguish the weaker responses.

For input, we supplied both:

• the student answer
• and a reference answer

Although we aimed to keep required metadata minimal, adding the reference answer produced a substantial performance boost. That led to an important practical conclusion: the extra effort of creating reference answers is justified if it significantly improves grading accuracy.

2. Mixture-of-Agents LLM System

After classification, the assigned label and the original student input are combined into a prompt for a Mixture-of-Agents feedback system.

The purpose of this second layer is not to decide the grade, but to generate clearer and more appropriate feedback based on the already-determined classification. This separation improves alignment between the grade and the explanation. Instead of relying on an LLM to both judge and justify at once, the MoA setup uses the classification result as structured guidance for the feedback stage.

To make this usable for teachers, we paired the feedback mechanism with a traffic-light interface:

• Green for fully correct
• Yellow for partially correct
• Red for incorrect

In the prototype, this idea is applied to programming assignments line by line. Each code line is highlighted with one of the traffic-light colors, and the system generates comments that include reinforcement for correct parts and constructive suggestions for lines that need improvement. The teacher can then accept, revise, or override that feedback.

Evaluation

We evaluated the approach on the ASAG (Automatic Short Answer Grading) dataset, using 130 student responses labeled into the three rubric categories. We compared three systems:

• a generic LLM baseline (Gemma 3 27B),
• a single transformer trained for direct three-class classification,
• and the proposed cascade model.

The LLM baseline was included as a realistic comparison point because it represents the kind of flexible model many people might want to use directly for grading. The single transformer served as the strongest non-cascaded baseline using the same encoder family. Performance was evaluated with standard classification metrics: precision, recall, F1-score, and overall accuracy.

Results

The cascaded approach produced a clear improvement over both baselines.

Compared to the single transformer, the cascade:

• increased recall for Incorrect answers from 0.26 to 0.58
• increased precision for Correct answers from 0.70 to 0.84
• improved overall accuracy from 0.65 to 0.82

This means the cascade more than doubled the system’s ability to correctly identify incorrect answers, while also becoming more precise when marking answers as fully correct. That is exactly the kind of tradeoff improvement that matters in grading: stronger detection of weak answers without becoming overly harsh on strong ones.

The full results for the cascade were:

• Incorrect: Precision 0.85, Recall 0.58, F1 0.69
• Partially Correct: Precision 0.77, Recall 0.77, F1 0.77
• Correct: Precision 0.84, Recall 0.91, F1 0.87
• Overall accuracy: 0.82

By contrast:

• the LLM baseline achieved only 0.58 accuracy and showed a strong bias toward predicting Correct,
• the single transformer improved on that with 0.65 accuracy, but still struggled with confusion between adjacent labels.

The confusion matrices in the paper illustrate this clearly: the LLM rarely labeled answers as incorrect, the single transformer still made many off-diagonal errors between Correct and Partially correct, and the cascade produced a much cleaner diagonal with fewer misclassifications.

Prototype Implementation

Beyond the classification experiment, we also developed a prototype for semi-automatic grading.

The prototype was designed to fit into existing teaching workflows with minimal disruption:

• it visually marks answer quality using a traffic-light metaphor,
• it generates first-draft feedback for the teaching staff,
• and it keeps a human-in-the-loop so staff can review and revise both the grading and the comments before finalizing them.

For programming tasks, the interface analyzes code line by line. This gives teachers a more granular overview than grading an entire submission as a single block. The color-coding helps direct attention to the most important problem areas, while the auto-generated comments reduce the amount of feedback that must be written from scratch. The result is a workflow that aims to save time without hiding the reasoning process from the teacher.

Conclusion

This work shows that better AI grading is not just about using a larger model. Careful system design matters. By splitting rubric prediction into a cascaded transformer architecture and separating that from an LLM-based feedback layer, we were able to build a pipeline that is more accurate, more transparent, and better suited to educational use than a direct one-model approach.

Three lessons stood out most clearly:

1. Cascaded architectures improve reliability Breaking the grading task into simpler decisions reduced confusion between adjacent classes and significantly improved detection of incorrect answers.

2. Transparency matters for adoption The traffic-light interface and editable feedback make the system easier for teachers to understand and trust.

3. AI works best as support, not replacement Keeping the teacher in control ensures that the system augments professional judgment instead of obscuring it.

A key limitation is that the current validation is still preliminary: it was performed on a relatively small dataset from a single course context. Future work should test whether the same benefits hold for longer answers and other modalities such as code, mathematics, or other domain-specific responses.

Future Directions:

• Extending the cascade to longer and more complex responses
• Evaluating the approach across different subjects and answer modalities
• Studying how teachers use and adapt the feedback in real grading workflows
• Exploring richer semi-automatic interfaces that further reduce workload while preserving oversight

This blog post is based on research presented at the 3rd Workshop on Engineering Interactive Systems Embedding AI Technologies, 2025. For complete technical details, see the full paper.

This blog post describes research presented at the D-SAIL Workshop on Transformative Curriculum Design in Palermo, Italy (2025). Co-authored with S. Vanbrabant, D. Vanacken, and G. Rovelo Ruiz.

What We Did

This work focused on a problem many teaching staff now face: if students increasingly ask their questions to general-purpose LLMs instead of to instructors or teaching assistants, teachers lose an important source of feedback about where students are struggling. That makes it harder to adapt course materials, prepare targeted support, and maintain meaningful student-teacher interaction. INSIGHT was designed as a response to that problem.

The core goal was not to block AI use, but to create a privacy-aware, human-centered environment where students can use an LLM while their interactions still generate useful teaching insights. In other words, we wanted to explore how AI could help bridge the student-teacher gap instead of widening it.

This research builds directly on the ideas from my master’s thesis. In that earlier work, we explored how AI could support students and teaching staff with exercises through features such as LLM access, a dynamic FAQ, and teacher-side analytics. INSIGHT takes that direction further in a more focused proof of concept: it emphasizes modular deployment, explicit privacy choices, and the use of student questions as a signal for improving face-to-face support.

The paper identifies a clear tension in educational AI:

Opportunities:

• AI can help personalize learning and teaching support
• Students can get quick answers while solving exercises
• Teaching staff can reduce repetitive question load
• Interaction data can reveal patterns in student difficulties

Challenges:

• Student-teacher interaction may degrade if too many questions move to AI
• Teaching staff may miss cues about knowledge gaps
• Privacy becomes a major concern when educational data is collected
• Students may become over-reliant on AI tools if they are used uncritically

So the main research contribution was to design and present a modular prototype that gives students access to an LLM in a monitored, privacy-aware setting while equipping teaching staff with data-driven insights they can use to improve in-person teaching.

How We Did It

The project started with interviews and focus-group style discussions with teaching staff at Hasselt University. This happened in two phases: first through semi-structured interviews across different courses, and later by presenting an initial prototype during an internal university workshop focused on AI. These conversations shaped the design priorities of INSIGHT. The strongest recurring concern was that staff feared losing contact with students and, with that, losing insight into students’ understanding of the course material.

System Design: INSIGHT

We developed INSIGHT as a modular proof-of-concept system centered around an INSIGHT Core. As shown in the architecture diagram on page 3, this core combines a keyword extraction component, a sentence similarity component, and a course database, and connects them to both a local LLM and a user interface that can be adapted to different course contexts.

Core Components:

1. AI-Assisted Exercise Support

   • Students can ask questions to an LLM while working on exercises
   • The LLM is the main interaction method in the prototype
   • In this implementation, we used Llama 3.2 3B, mainly because it offers relatively fast responses and is feasible to run on consumer hardware
   • The modular setup makes it possible to swap in newer or more capable models later, depending on the needs of the course

A key design choice is that INSIGHT does not interfere with the LLM’s behavior through prompt engineering or artificial restrictions. The idea is that students should interact with it naturally, rather than being nudged into a constrained workflow. The system then analyzes those interactions afterward to generate useful signals for teaching staff.

1. Keyword Extraction

   • INSIGHT first needs to know which topics belong to each exercise
   • To do that, it uses KeyBERT to extract an initial keyword list from the exercise text
   • Teaching staff then review and refine those keywords manually in a mixed-initiative process
   • These reviewed keywords become the course’s topic vocabulary

The example shown in the keyword selection interface on page 4 illustrates why human review matters: automatic extraction is helpful, but imperfect. The paper explicitly notes odd outputs such as “tree tree” in the extracted keywords for one exercise, which demonstrates why final control remains with the teaching staff.

1. Dynamic FAQ Generation

   • Student questions are embedded in the same vector space as the topic vocabulary
   • The nearest keyword is used to assign a topic label to a question
   • Similar questions are grouped using the all-MiniLM-L6-v2 sentence embedding model and cosine similarity
   • Recurrent questions can then be turned into FAQ items
   • INSIGHT caches the LLM’s answer for grouped questions, and teaching staff can edit that answer before reuse

This makes the FAQ both dynamic and supervised. It grows from real student questions, but the teaching staff still verify and adjust the answers, which reduces variability and keeps the system human-in-the-loop.

1. Teaching Staff Insights Dashboard

   • INSIGHT collects and analyzes student questions to the LLM
   • It also tracks FAQ usage and exercise difficulty ratings
   • The teaching staff interface includes a dynamic FAQ and two visualizations

   • According to the interface shown on page 5, these graphs display:

     • how often each FAQ item is viewed
     • how frequently different course topics appear in student questions

These visualizations give teaching staff a fast way to spot recurring confusion, identify weak points in course materials, and prepare more targeted support during office hours or interactive sessions.

Modular Design

INSIGHT is designed to be:

• Flexible: The core system is separate from the user interface and LLM, so it can be deployed in different higher education courses without changing the core logic. The proof of concept uses data from UHasselt’s Algorithms and Data Structures course, but the architecture is intentionally broader than that one course.
• Privacy-aware: Students can choose an anonymous mode, allowing staff to see the data without attached names. In addition, the system uses Ollama to run the LLM locally, so queries can remain private and are not shared with third parties.
• Extensible: Because the components are modular, new models, interfaces, and analytics features can be added later without redesigning the full system.

This privacy design is especially important because the system deliberately collects interaction data. INSIGHT’s approach is to make that collection transparent and consent-based, rather than hidden or mandatory.

Implementation

Technically, the system consists of:

• a central INSIGHT Core
• a database storing course information and interaction data
• a keyword extraction module
• a sentence similarity module
• a local LLM
• separate teacher and student user interfaces

The student interface, shown on page 6, combines three functions:

• a chat window for asking the LLM questions
• access to the FAQ
• an exercise difficulty rating mechanism that gives explicit feedback to the teaching staff

The teacher interface, shown on page 5, allows staff to:

• review and edit FAQ items
• add FAQ entries manually
• inspect topic frequencies
• inspect FAQ view counts

Results

As a proof of concept, INSIGHT demonstrates that it is possible to combine LLM access, keyword analysis, dynamic FAQs, and teacher-facing analytics into a single educational support system that strengthens visibility into student difficulties instead of hiding them.

System Capabilities

INSIGHT successfully demonstrated:

1. Better Student-Teacher Connection The system is explicitly designed so that student questions to AI still become useful signals for the teaching staff. Rather than replacing face-to-face support, it helps staff understand where students need help before or during in-person interactions.

2. Data-Driven Personalization By analyzing question topics, FAQ usage, and difficulty ratings, teaching staff can:

   • identify which topics cause recurring confusion
   • adapt course materials over time
   • prepare more targeted explanations for students or groups
   • provide more personalized face-to-face support

3. Dynamic FAQ with Human Oversight Frequently recurring questions can be grouped, answered once, and then reviewed by staff. This reduces repetitive question load while still allowing teachers to verify accuracy and refine explanations.

4. Privacy-Conscious Monitoring The system shows that monitoring does not have to mean intrusive surveillance: students can opt into identified sharing or choose anonymous use, and local deployment reduces third-party data exposure.

Insights from Deployment

This paper presents INSIGHT primarily as a proof of concept, not a full classroom validation study. The demonstrated value lies in the kinds of insights the system can surface for teaching staff, especially in interactive course settings.

Based on the prototype, staff can derive insights such as:

• which exercise topics are producing the most questions,
• which questions recur often enough to justify an FAQ item,
• which FAQ items students revisit frequently,
• and which exercises students rate as more difficult.

Those signals can help answer practical teaching questions like:

• Which concept needs clearer explanation in the next session?
• Which exercise caused unexpected confusion?
• Which FAQ answers should be strengthened or expanded?

The paper also discusses important limitations. Students may avoid monitored systems and ask questions to outside LLMs instead, which could bias the teacher’s view of student understanding. That means the usefulness of INSIGHT depends not only on the tool itself, but also on whether students perceive value in using it and whether teaching staff actively act on the insights it provides.

Conclusion

INSIGHT shows that AI in education does not have to reduce human contact by default. If designed carefully, it can create a shared space where students get quick AI support and teaching staff still retain visibility into learning difficulties, common misconceptions, and emerging needs.

Key Contributions:

1. A modular architecture for integrating AI support into higher education courses
2. A mixed-initiative keyword analysis pipeline to map student questions to course topics
3. A dynamic FAQ system built from real student questions and supervised by teaching staff
4. A teacher-facing analytics layer that supports more personalized face-to-face teaching
5. A privacy-aware deployment model with anonymous mode and local LLM execution

What stands out most in this work is the design philosophy: INSIGHT does not try to stop students from using LLMs, nor does it assume AI should replace teachers. Instead, it accepts that these tools are already part of students’ reality and tries to make their use safer, more transparent, and more useful for actual teaching practice.

The paper also makes clear that technical design alone is not enough. Responsible AI use in education depends on dialogue, trust, and shared expectations between students and staff. INSIGHT supports that conversation, but it cannot replace it.

Future Work:

The next steps described in the paper are:

• conducting a cognitive walkthrough with new teaching staff interested in piloting the system,
• deploying INSIGHT in real courses at Hasselt University,
• empirically validating whether it improves support and learning,
• exploring knowledge tracing for more advanced personalization,
• and using the collected data to support adaptive learning and more inclusive learning experiences over time.

If we were to extend this line of work, the most important next step would be a real classroom study: not just to test whether the system works technically, but to understand whether students actually use it, whether teachers trust it, and whether it genuinely improves student-teacher interaction in practice.

This blog post is based on research presented at the D-SAIL Workshop, 2025. For complete details, see the full paper.

This blog post describes my master’s thesis work on using AI in education, supervised by Davy Vanacken and Gustavo Rovelo Ruiz at UHasselt.

What I Did

My thesis explored how AI—especially Large Language Models (LLMs)—can be integrated into education in a way that is useful, transparent, and responsible.

With the emergence of LLMs, students now have the opportunity to ask AI questions. However, teachers have concerns about the impact on teaching quality—particularly regarding the accuracy of AI-generated answers, the consistency of those answers, privacy, and the potential reduction in student-teacher interaction.

The goal of this thesis was to:

• Explore applications and challenges of AI (especially LLMs) in education
• Create a prototype demonstrating how these applications can be implemented
• Ensure that AI enhances rather than replaces student-teacher interaction

More specifically, the project was structured around three objectives:

1. Identify useful AI tools and their challenges in education through a literature review and exploratory experiments with LLM-generated answers discussed with teachers.
2. Develop a student-facing prototype that gives responsible access to an LLM, offers a dynamic FAQ, and recommends exercises based on earlier difficulties.
3. Develop a teacher-facing prototype that helps teachers manage the LLM, review frequently asked questions, inspect learning difficulties, and use data visualisations to improve course materials.

The broader research question was not whether AI should replace teachers, but how it can be integrated in a way that supports both students and teaching teams while preserving human oversight.

How I Did It

The research followed a structured process inspired by the ADDIE model: analysis, design, development, implementation, and evaluation.

1. Literature Review I conducted a broad review of existing AI applications in education, including machine learning, personalised learning systems, chatbots, expert systems, intelligent tutors, and virtual learning environments. I also focused on the main challenges of LLM use in education: unreliable answers, inconsistent outputs, issues around assessment, privacy, intellectual property, and the risk of weakening teacher-student interaction.

2. Experiments with LLMs Before designing the prototype, I explored how LLMs behaved when solving exercises and discussed those outputs with teachers. This helped clarify where LLMs were useful, where they failed, and what kinds of controls teachers would need to trust them in an educational setting. This phase also informed the role of prompt engineering in steering LLM responses toward course-specific expectations.

3. Concept Development Based on the literature and experiments, I defined four core design goals for the prototype:

   • Give students responsible access to an LLM
   • Build a dynamic FAQ that evolves from student questions
   • Recommend exercises based on earlier difficulties
   • Make the whole application transparent about which parts use AI and why

4. Prototype Implementation I built two applications using Windows Presentation Foundation (WPF) with C#:

   • one for teachers
   • one for students

   Both applications communicate with a central Python Flask server, which acts as the hub between the interfaces and the AI/ML components.

   The system combines several components:

   • An open-source LLM for student questions
   • KeyBERT for keyword extraction from exercises and questions
   • Sentence Transformers for grouping similar questions using sentence similarity
   • A collaborative filtering recommender for exercise suggestions
   • A database for courses, questions, ratings, and related interaction data

5. Design Decisions A key design choice was to rely on locally manageable, open-source models rather than a fully external commercial AI provider. In the prototype, the default model was GPT4All Falcon, chosen because it can run on relatively modest hardware and gives teachers more control. Teachers can also configure course-specific model URLs through Hugging Face-compatible APIs and provide custom instructions such as an opening prompt and instructions that are appended to every student message.

Results

The prototype demonstrated how multiple AI-assisted features can be combined into a practical educational workflow.

Teacher-side results

The teacher application gives control over:

• Which LLM students use
• How the LLM behaves, through course-specific instructions
• Which keywords are associated with each exercise
• Which FAQ items are added
• Which trends and difficulties appear in student data

Teachers can:

• Add a course and configure the LLM
• Input exercises and curate automatically extracted keywords
• Review grouped student questions
• Receive suggestions for new FAQ items
• Inspect exercise completion rates
• View perceived exercise difficulty
• Use visualisations to better understand where students struggle

This makes the AI layer adjustable rather than opaque.

Student-side results

The student application focuses on guided support:

• Students can chat with the LLM in a familiar interface
• They can select a course and optionally a specific exercise, which links questions to the right context
• They can browse a dynamic FAQ with previously answered questions and teacher-added hints
• They can receive exercise recommendations based on earlier ratings and difficulty patterns

An important addition is the option to use the system in an anonymous mode. This allows students to contribute data about difficult topics and common questions without attaching that data to their identity, while clearly warning them that anonymous use limits the teacher’s ability to give individual feedback.

Core technical outcomes

The prototype successfully demonstrated several innovations:

• Teacher-Customizable LLMs: Teachers can configure AI responses to align with their teaching style and course requirements
• Exercise Recommender: A collaborative filtering system predicts which exercises may help a student practise weak areas
• Dynamic FAQ: Student questions are grouped through sentence similarity, and commonly recurring questions can be proposed to teachers as FAQ additions
• Data Collection for Insights: Questions, ratings, completion data, and detected keywords help teachers understand learning bottlenecks

Key Insight: Rather than diminishing interaction between teachers and students, the collected data can help teachers maintain and even improve that interaction by making student difficulties more visible and by supporting more targeted feedback.

At the same time, the thesis also identified important limitations:

• LLM reliability is still a real issue
• The chosen LLM was efficient, but limited (for example, it only responded in English)
• The recommender suffers from the classic cold start problem, since recommendations are weaker when little prior data is available
• Some desirable features, such as stronger emotional intelligence or deeper reflection, would require far more computational resources or further model adaptation

Because this thesis focused on concept validation and implementation, it did not yet include a full user study with classroom participants. That evaluation is the most important next step.

Conclusion

This work shows that AI in education does not have to mean replacing teachers. Instead, it can be used to create a more supportive and data-informed learning environment—provided that the system is transparent, locally manageable, and designed around teacher control.

The thesis started with a broad analysis of AI opportunities and risks in education, then translated those findings into a working prototype for both students and teachers. The result is a concrete demonstration of how LLMs, keyword extraction, sentence similarity, and recommender systems can be combined in an educational setting while addressing important concerns such as transparency, privacy, and meaningful human oversight.

Several lessons stood out during the project:

• Control matters: teachers need ways to shape and constrain AI behaviour
• AI works best as a complement: not as a replacement for teacher-student interaction
• Transparency increases trust: users should understand which parts are AI-driven
• Local deployment helps privacy: avoiding unnecessary external data sharing is a major advantage

Future Work: The most important next step is a real user study comparing a control group and a test group over a longer period. Such a study should evaluate learning outcomes, retention, usability, transparency, student autonomy, and whether teachers still feel in control of the learning process. There is also room to improve the recommender, add more prompt-engineering support for teachers, and explore additional AI applications such as AI-assisted course design or tutoring.

If I were to extend this work, I would focus first on evaluating the prototype with real users, then on refining the recommendation logic and making the LLM support more accessible for teachers with little AI experience.

This blog post is based on my master’s thesis completed at UHasselt in 2024. For the complete technical details, please refer to the full thesis.