Abstract:

There is a clear disparity between the technology that we have at our fingertips – and that we use in our daily lives – and the adoption of that technology within our schools and our classrooms. There is now an increasing gap between the technological lives of young people and the learning experiences we can offer them. The problem, it seems, is not the availability of the technology but rather the translation of that technology into a learning context.

Part of the blame lies with the way in which we assess or evaluate the impact of new technologies before they reach the classroom. European businesses invest billions every year in the development of new learning technologies and yet comparatively few of these technologies ever end up in common usage in our classrooms. If teachers fail to adopt new technologies – or if students lack enthusiasm, engagement or connection with those technologies - this should be identified early on and remedies designed to address these issues.

The Newton Project – funded under Horizon 2020 – is developing a new approach to STEM learning that is based on the adoption of a range of new learning technologies (including augmented reality, virtual reality, Fab Labs, multi-sensory learning and gamification). As the project moves into the pilot phase, we are now setting out our approach to evaluation and impact assessment.

In this paper, we present our evaluation model. This model is an attempt to succeed where others before us have failed. The model considers a range of different aspects to impact assessment that, we believe, will tell us which technologies are likely to be adopted in the classrooms and laboratories of our schools and colleges across Europe.

Our model focuses on four fundamental elements: first, we look at knowledge and skills acquisition. What is the extent to which these new technologies have a genuine impact on the educational development of students and how can we prove that this is the case? Second, we look at student engagement and student motivation. What we know is that a more engaged, motivated student is likely to be a better learner. Our approach seeks to measure the level of engagement and motivation and demonstrate the impact this has on learning. Third, we explore the impact of the technology on teachers. How far do teachers embrace these new technologies? To what extent do these approaches make their teaching more effective? What are the barriers that might stop early adoption of these technologies within a classroom setting? And finally, we look at how the technology itself can utilise Educational Data Mining (EDM) methodologies and techniques in order to analyse data collected during teaching and learning processes to better understand student (and teacher) behaviour and the impact that the technology is having.

Although this is a field that has a long history across Europe, impact assessments are too often designed without rigour and with an aim to simply provide ‘good results’ rather than as a way of genuinely tested whether an educational approach actually works and is feasible for adoption in a wide range of learning settings. In presenting our theoretical model, we invite contributions from others who are similarly charged with assessing the impact of their learning technologies.

Keywords:

Impact evaluation, technology evaluation, STEM education, gamification, augmented reality, evaluation model.