Abstract:

Immediate automated feedback supports student mastery of learning outcomes while improving student engagement and satisfaction. The answer-until-correct (AUC) feedback mechanism is one form of immediate feedback that can be produced relatively cheaply compared to elaborative automatic feedback. It has been shown that using AUC as part of a team-based collaborative learning exercise can improve participation and ultimately produce better learning outcomes. Further, it has been shown that incorporating AUC into an integrated testlet (IT) can successfully reproduce the results of a constructed-answer exam in the context of summative evaluation. The design of AUC-ITs has so far been discussed extensively in the context of post-secondary physics education. Here we detail the process of transferring the known best practices to the subject of software design, a more subjective discipline. We provide a general testlet structure that could be applied to other subjects in the future. The resulting AUC-ITs were used as a team-based learning exercise rather than as a summative assessment. An in-class observational protocol was used to produce a detailed report of the social and pedagogical effects of this activity. Finally the main contribution of this paper is to evaluate whether the hypothesized design structure of the AUC-IT is substantiated by the student response data. With increasing interest in using AUC-ITs as a learning exercise in both formal and informal educational settings, this discussion around design validation is especially timely. The design elements of a learning exercise should be understood, and successfully measured, prior to making claims about their effects on any learning outcome or other metric of interest. Here we present evidence for a so-called “engagement-learning" mechanism that supports the claim that learning will take place throughout AUC-IT participation, provided that meaningful and fruitful discussion is taking place. Here both conditions of "meaningful" and "fruitful" are given quantitative definitions which can be mined from the student response data. We provide a discussion around best practices for such a design validation study.

The contributions of the paper can be summarized as:
1. A general AUC-IT structure is provided which could be transferred to any subject domain. The structure is designed to measure multiple student misconceptions simultaneously, given a common problem prompt (e.g. a code segment or reading comprehension segment probing multiple misconceptions).
2. A novel methodology is provided to quantitatively evaluate whether the design structure of the AUC-IT (i.e. the assumptions of how learning is taking place throughout the AUC-IT participation) is true to the student experience of the AUC-IT, based on student interaction data.
3. Our analysis supports the assumption of an engagement-learning mechanism.
4. The social and pedagogical effects of the learning activity are qualitatively described in detail.
5. Specific AUC-ITs in the domain of software design are provided for further use.

Keywords:

Immediate feedback, design validation, software design.