Quality Assurance Automation of Canvas Courses: A Human-in-the-Loop Approach

The increasing demand for online education calls for cost-effective and scalable systems that help instructional designers and faculty ensure the quality of their courses. The QA Bot addresses this need by conducting rapid evaluations of courses against established standards, including Quality Matters, Online Learning Consortium, and Universal Design for Learning. The working prototype follows a hybrid approach that combines programmatic file extraction with LLM-based evaluation of observable course data. The project emphasizes accessibility by supporting smaller, high-performing models (e.g., Llama 3.1 8B; DeepSeek-R1 8B) that would enable an open-source release. Tool performance validation includes comparisons of course evaluations between human experts and the QA Bot using open-source and larger models available through commercial APIs. This paper presents a work-in-progress system that describes the tool's design rationale, integration into the learning engineering process, and a preliminary validation framework based on a pilot study.