EVALUATION OF SERIOUS GAMES: A HOLISTIC APPROACH

Digital games constitute a major emerging technology that is expected to enter mainstream educational use within a few years. The highly engaging and motivating character of such games has the potential to support the creation of immersive, meaningful, and situated learning experiences. To seize this potential meaningful quality and impact measurement are indispensible. Although there is a growing body of evidence on the efficacy of games for learning, evaluation is often poorly designed, incomplete, biased, if not entirely absent. Well-designed evaluation demonstrating the educational effect as well as the return on investment of serious games may foster broader adoption by educational institutions and training providers, and support the development of the serious game industry. In the European project RAGE (http://rageproject.eu/) a holistic and multi-perspective framework for serious game evaluation is developed and applied and is presented in this paper.

The primary objective of RAGE is to make available and accessible advanced software tools, methodologies and expertise for supporting serious game development and take-up: Reusable, and interoperable software components (“gaming assets”) for creation are made available, with the ability to undertake various kinds of data analysis (e.g. competence assessment, emotion detection) and game intelligence (e.g. competence-based personalisation, natural language understanding). These assets will be provided via an Ecosystem as a central access point and affinity social space. These games technologies will be applied and tested in learning game development for six selected application scenarios.

The RAGE evaluation framework forms the basis for analysis of the effectiveness of the RAGE technologies, and for ensuring that they reflect the needs of industrial and educational stakeholders. The assets will be evaluated from the industry perspective in respect of their software quality and their added value in the authoring process. In addition, the Ecosystem will be assessed in terms of its usefulness, usability, and overall impact for gaming stakeholders. Further, a contextual cost-benefit analysis will be undertaken in each of the (six) application scenarios. The evaluation perspective in addressing the “educational effectiveness” of the games will be on analysis of their benefit for learning, as well as users’ perceptions. Unobtrusive data tracking and sensoring will complement more traditional instruments (such as questionnaires), thus providing a wide spectrum of evaluation methods capable of obtaining in-depth insights of players’ behaviours and experiences.

The RAGE evaluation framework will give rise to scientifically sound, iterative, and mixed-method evaluations of the project outcomes and may serve as best practice for future serious game evaluation, in general. The tools and methods that RAGE produces will be of interest to the wider serious gaming research and industry communities as they strive for improving the quality of serious games.