RFO-HITL: An AI agentic architecture for lecturer decision support in higher education

Abstract

In response to the challenges of transparency and accountability in integrating generative artificial intelligence (GenAI) into higher education, this study proposes the RFO-HITL (Rule-first orchestration with human-in-the-loop) architecture. This AI agent framework balances technological efficiency with human oversight by prioritising the adoption of explicit rule sets for decision-making, and limiting the role of large language models (LLMs). In addition, this RFO-HITL architecture is deployed using a no-code/low-code (NCLC) platform. In combination with the mandatory human-in-the-loop (HITL) mechanism. Based on the design science research (DSR) methodology, the architecture was systematically developed and evaluated using a synthetic dataset (N=100). Quantitative results demonstrate that the RFO-HITL architecture is promising in identifying at-risk cases, achieving an F1-score of 88.9%, and generating highquality student support content. A safety violation rate of 6% - mostly pertaining to length constraints and stylistic or tonal subtleties - was observed, thereby empirically underlining the significance of the HITL mechanism. This research primarily contributes to building the AI agent architecture, offering actionable design knowledge for the responsible implementation of GenAI in educational settings, thereby promoting transparency while preserving lectures pedagogical professional judgment.