Abstract:

Multiple Choice Questions (MCQs) and other forms of objective tests have been extensively studied and evaluated not only as a cost-effective assessment method, but also as a way of engaging students in a “learning conversation” through regular formative feedback. According to the findings from a recent UK JISC project, assessment and feedback are two crucial components of the learning experience, which shape learners' understanding of the curriculum, and determine their ability to progress. However, creating a good objective test is not only difficult, but also a very time-consuming task, thus preventing its wider adoption. Increased use of the internet in education and the emergence of internet-based knowledge repositories such as Wikipedia, Wordnet and similar, have enabled the creation of tools and techniques for the automatic generation of objective tests (e.g. opensemcq.org, tao.lu etc.). The rapidly expanding body of knowledge in this area includes methods based on Semantic Web, Natural Language Processing (NLP) and pattern-matching techniques.

The aim of this paper is to introduce, discuss and evaluate an innovative approach to automated generation of MCQs from Semantic Web ontologies, which are increasingly used as a tool for organising knowledge from various domains, including biomedical, engineering, business, languages and others.

In the previously reported research an open source platform was introduced (opencemcq.org) which aimed at generating different types of questions for targeting specific levels of knowledge as defined in Bloom's taxonomy. In this work the quality of generated questions is evaluated based on the feedback from the domain experts (university lecturers) from different subject areas including business, law, computer science and life sciences.

The evaluation is based on more than 60 tests comprising 2794 questions, of which 202 were assessed. The preliminary analysis of the results indicates that the quality of questions varies across different types of question and different knowledge domains. In particular, the quality of questions testing the application of knowledge through identification of examples of specific concepts (e.g. “Which of the following items is an example of…”) is shown to be higher than of the other question types. The qualitative comments from the evaluators indicate that while the quality of questions is not yet at the level required for the immediate use, the questions provide a useful “seed” for further assessment design. The comments also suggest that concept-based tests are not the most natural method of assessing knowledge in some subject areas.

The findings from the evaluation will form a basis for further improvements of the opensemcq tool, in accordance with the design science research, the methodological stance adopted in this work.

Keywords:

E-assessment, semantic web ontologies, objective tests, evaluation.