Abstract:

We report ongoing research that assesses how well AI can evaluate teaching, which we define as “effective” to the degree it helps students learn. Our current research builds on a body of prior work in which we assessed how well human judges performed the same task. Under varying conditions (length of instructional sample; instruction documented as video, audio, and transcript; and judgments based on intuition alone, high-inference rubrics, and low-inference rubrics) human judges demonstrate significant limitations. Experts and non-experts did no better than chance when they relied solely on their intuitive judgment. Experts fared no better when using high-inference rubrics. However, experts and non-experts were more accurate than chance when they used low-inference rubrics, and just as accurate using transcripts of instruction compared to video. Machines are very good at performing low-inference tasks, and AI in particular is very good at “understanding” written text, such as transcripts. Is AI better at judging teaching effectiveness from transcripts than humans? If so, should human judges be replaced by machines? We provide data from a series of new experiments comparing AI with humans (using both intuition and different observation rubrics) that may help answer these questions. We then engage our audience in a discussion of the moral dilemmas introduced by the use of AI in evaluating teachers.

Keywords:

Teacher observation, evaluation, AI.