AI-ASSISTED GRADING OF STUDENT-CREATED SCREENCASTS IN A COMPUTER PROGRAMMING COURSE: TEACHER PERCEIVED CHALLENGES AND OPPORTUNITIES

Abstract

In computer programming courses, assessing students' understanding often extends beyond code correctness to evaluating their ability to articulate and explain their work. To this end, screencast submissions, where students demonstrate their group programming projects, have become a valuable pedagogical tool. However, grading such submissions manually is time-intensive and subject to variability in human assessment. To address this challenge, we developed VideoAna, an AI-assisted tool designed to evaluate screencasts. This article outlines the software architecture of our tool, its evaluation methodology, and the results from its pilot implementation in a programming course. During development and deployment, we encountered and overcame several challenges, particularly in using optical character recognition (OCR) to interpret code displayed in screencasts and regular expressions (regex) to analyze and match key programming concepts. These issues included handling low-resolution videos and recognizing complex code structures. Despite these challenges, our tool demonstrated acceptable consistency in grading, significant time savings for instructors. This help instructors provide feedbacks with higher quality for students. This article will discuss the design and implementation of the tool, the lessons learned during its application, and practical strategies for addressing common technical and pedagogical challenges. Additionally, we will provide useful suggestions for other educators seeking to integrate AI tools into their assessment processes. Our findings highlight the potential of AI in assisting teachers to grade student-created screencasts in programming, and the potential of applying similar tools in other disciplines.