Self-Reported AI Use and Quiz Performance in an Introductory Programming Course: An Observational Study (Invited paper)

Abstract

Large language models (LLMs) are now widely available to students, but it is still unclear how their use during coursework relates to later quiz performance. This study examines AI use in a bachelor-level Python course, where learners completed programming exercises and follow-up quizzes. To examine how students’ self-reported AI use relates to perceived difficulty, confidence, perceived helpfulness, and quiz performance. We conducted a quantitative observational study with supplementary qualitative analysis using 170 anonymous exercise-level survey submissions collected after five programming exercises. Students reported their AI use, perceived difficulty, confidence, perceived helpfulness, and how they used AI. Three exercises included follow-up comprehension quizzes, providing 100 scored submissions. AI assistance during these quizzes was not ruled out. AI was used in 64.7% of submissions. In the scored subset, reported exercise AI use was associated with a 21.0-point lower mean than reported no-AI use on the observed-range 0–100 score, which is not percent correct. Exploratory subgroup differences varied by difficulty and confidence, with some small comparison cells. Higher AI-use intensity was associated with lower scores, while perceived helpfulness had a near-zero association with quiz performance. Open-ended responses suggested that higher-scoring AI-use submissions more often described debugging or checking their own code, while lower-scoring submissions more often described code generation. Student ratings of AI helpfulness should be interpreted cautiously, since they did not track quiz performance. For introductory programming courses, follow-up checks of independent understanding may be important when students use AI during exercises.