Abstract:

For the last two decades, a rapidly growing amount of Open Educational Resources (OER) have become available in OER repositories (OERRs) for educators to re-use, re-publish and share within their communities. Regardless of the amount of educational resources available, many OERRs are often seen as unsuccessful (Mitchell & Lutters, 2006; Dichev & Dicheva, 2012). In consequence, finding models for sustainable collections is a key issue in repository research, and the main problem behind that is understanding the evolution of successful repositories (Sánchez-Alonso et al., 2011). Successes of OERRs have been studied widely (e.g. Ochoa & Duval, 2009; Tzikopoulos et al., 2007; Polsani, 2006) however, there has not been consensus in the community on how to measure the repositories success. Studies around the success of the repositories have also often focused entirely to the perspective of the OERR developers, forgetting that the users might have a different idea of a successful repository.

In this research we conducted a comprehensive literature review tacking the issue of repository success in the field of technology-enhanced learning. The literature review followed the steps defined by Kitchenham (2004) for conducting a rigorous literature review analysis. Our research questions were: How to measure OERR success? Which indicators, tools, instruments and metrics are currently used in practice? A total of 82 relevant papers from 1994 to 2015 were identified in the final analysis for the success metrics.

Our findings show that various metrics have been proposed by previous literature for the OERR’s (Lund & Hojsholt-Poulsen, 2010; Venturi & Bessis, 2006; Thomas & McDonald, 2007) success. Most repositories gather data on a collection of success indicators, which typically take into consideration 1) monthly or daily user rates/page views, 2)download counts or hits in the interface portal. However, these indicators might not be considering the life span and sustainability of the repository. OERR are often supported by project-style funding. When the project ends and the support of the community building activities end - the repositories start loosing users and downloads. Perhaps the most extensive quantitative study by Ochoa & Duval (2009) analysed LORs with metrics of 1) Content growth and 2) Contribution growth, lifetime & publishing rate, thereby taking into consideration the aspect of sustainability and business model over time. Previous studies (Clements et al., 2014) have also shown that success does not necessarily mean user engagement but that repository developers might see OERR successful if it has functioned as a test bed. In this research, we harmonise a definition of OERR success for both users and developers and suggest an overall OERR success measuring criteria based on previous studies. Our research can help developers, communities and future projects of OERRs in measuring the success of their repository.

Keywords:

Open Content, Open Educational Resources, Learning object repositories, Success, Metrics, Success indicators.