Abstract:

Sustaining student attention in school classrooms remains central to effective learning worldwide, yet educators continually face this challenge. This research introduces a novel multimodal engagement-monitoring system that leverages image recognition and speech analysis to capture real-time student attentiveness while minimizing interruptions to teaching.

The system integrates visual student detection and pose-based teacher gesture classification. It also tracks student facial orientation and recognizes Japanese speech to identify critical instructional moments and responses. Each component operates within a modular Python environment in the CiRA-Core IDE (Integrated Development Environment). This allows flexible system configuration and seamless scalability across diverse classroom settings.

What sets this solution apart is its ability to provide teachers with immediate feedback on student engagement and participation, enabling adjustments to facilitation and equal care for students. Experimental evaluation on classroom recordings, keeping track of student tasks, where seated students are observed on face orientation toward the teacher, writing or reading posture, and the number of hand raises per observation. At the same time, out-of-seat duration and length of time are observed to help teachers and schools understand the effectiveness of student and classroom behavior reinforcement techniques.

For this research, the system will first analyze a typical middle school class in Japan, and the second recording will feature students and teachers in the same middle school who have undergone PBS (Positive Behavior Support). Measuring the amount of time students are on-task or focused during the class compared to the total time the teacher requests attention. We will look to see whether the on-task ratio or the amount of student focus on the teacher's request for attention supports the intended PBS approach to supporting students and teachers.

By embracing emerging AI and multimodal recognition systems, this platform supports classroom analytics beyond traditional observation and offers actionable insights for both real-time adaptive teaching and long-term educational research. Its modular architecture enables users to access and adapt it to different teaching methodologies, learning environments, and student needs, thereby supporting inclusive and innovative educational practices.

This work demonstrates how cutting-edge educational technology empowers teachers to cultivate deeper learning, foster engagement, and respond to individual student needs.

Keywords:

Classroom analytics, Multimodal learning analytics, Computer vision in education, Teacher feedback, Student engagement.