Abstract:

Digital games constitute a major emerging technology that is increasingly entering mainstream educational use. Serious or applied games have a highly engaging and motivating character and enable immersive, meaningful and situated learning experiences. To take full advantage of the great potential of serious games a careful assessment of their quality and impact is necessary. Consequently, there is a need for sound evaluation methods and real-world testing via deployment in authentic educational environments.

The European project RAGE (http://rageproject.eu/) aims at enhancing the development and take-up of digital game-based learning for game industry and education by making available advanced game components – in order to enable easier, faster and more cost-effective development of applied games. The RAGE technologies and methodologies are applied and tested in the context of game projects for six selected application scenarios representing different target groups and markets. An evaluation framework has been elaborated that enables the systematic investigation of the RAGE components and games and their effectiveness.

This evaluation framework incorporates the perspectives of the different stakeholder groups of applied games in order to ensure:
(a) the added value and significance of game components for game developers,
(b) the relevance and pedagogical value of an applied game for training,
(c) the enjoyment of the game as well as learning effectiveness for learners.

We present the evaluation framework in detail and how it has been translated into and used in evaluation practice in an educational scenario with real-world users, by the example of the game project and use case of an interview simulation for job seekers. In the RAGE project the game ‘Job Quest’ was developed as a single player game simulation of a real job search experience, to serve as a training approach supporting face-to-face recruitment training in the human resource services industry. In Job Quest a range of RAGE game components were used and integrated in order to facilitate and accelerate game development (e.g. authentication, analytics) and to add unique functionalities (e.g. emotion detection). Empirical evaluations were carried out aligned with the phases of game development and deployment and incorporating the evaluation questions from the different stakeholders’ perspectives. The game components used in game development were evaluated by gathering game developers’ opinion on the software quality of the individual components and their added value for the game engineering process. In addition, feedback on the game and used components was elicited from training providers, to evaluate the technologies with respect to their added pedagogical value and use in the real-world training context. Actual learning experiences with the game were evaluated by collecting users’ assessments of usability, game experience, and usefulness for acquiring job search skills. Evaluation results from the different evaluation phases and target groups will be outlined.

Overall, the benefits of the RAGE approach and technologies for game development and educational application could be demonstrated and useful information and inspiration for future work could be obtained. The applied evaluation methodology proved as a suitable and holistic approach incorporating the evaluation interests and needs from different perspectives.

Keywords:

Serious games, applied games, evaluation.