Abstract:

The challenge of evaluating the impact of learning technologies is common and significant - for any learning technology to take root in the classroom there needs to be a robust and thorough analysis of the value of that technology and the feasibility of implementing that technology within a learning context. Such an evaluation has to take account of the impact on knowledge and skills acquisition, the usability/accessibility of the technology and the effect that a given technology has on learner motivation and engagement. A thorough evaluation also has to consider the impact on teacher role, the institutional impact of the technology and the practical aspects of technology uptake that can often hinder effective implementation. The NEWTON project, funded under H2020, is one of the largest TEL projects of its kind and integrates a range of innovative learning technologies including augmented reality, mulsemedia (multisensory media), gamification and remote virtual and fabrication laboratories. The project is currently evaluating the NEWTON platform and its connected components in a series of small and large-scale pilots across Europe. In this paper, we discuss the principal challenges of designing and implementing a large-scale TEL pilot, with a focus on a case study involving two schools in Salerno, Italy. We present the challenges of working with our Italian partner schools, developing content with them, proposing a very engaging learning experience (game/gamification-based), preparing the technology and supporting the teachers. We also consider evaluation design and the implications this has had on the Italian partner schools. Ultimately we shed light on how to run an effective pilot evaluation of new learning technologies in a way that genuinely involves and engages teachers and learners and provides robust impact data at the end of the process.

Keywords:

Pilot evaluation, TEL, technology enhanced learning, technology impact.