Abstract:

Higher Education Institutions (HEI), require the presence of the students for evaluation purposes. They haven’t changed much regarding the evaluation process and keep using a well-known and reliable method to evaluate their students. The physically presence of the students is a reliable way to mitigate the possibilities of fraud.

The modern Learning Management Systems (LMS) can provide the necessary features to carry out the online evaluations activities. These systems and features can be use in a blended model, in which the students are required to be physically present, in a similar form to a traditional on site exam, and the exam is executed using a LMS. The exam room is equipped with PCs which are equipped with the necessary means to access the LMS and assure that a higher risk of fraud is not introduced by the usage of PCs and LMS as support for the exams. We call this model, the b-eval model.

To adopt this model, professors must design their evaluation process in order to incorporate the features and restrains posed by the PC and the LMS. In some scenarios, it is advantageous for the professor and for the students, as a proper designed exam can easily be executed by the students and instantaneously be evaluated and graded. Otherwise, the traditional process takes much longer as the professor must collect the paper exams, which he will later evaluate. The process can take several days or even weeks.
To change to the b-eval model the professor must consider the change in the specific practices required for the evaluation process, including: the exam design; the exam delivery to the students; the exam evaluation and grading.

In this paper we consider three typical scenarios, corresponding to typical practices, commonly used in HEI. For each scenario, it was considered a 20 questions evaluation, with a composition of types of answers, including: direct answer, multiple choice answer; and open written answer.

Of the three scenarios, the first is a test of 20 multiple choice questions; the second scenario is a mix of 8 multiple choice questions, plus 6 direct answers questions; and 6 written answer question; the third scenario is 3 multiple choice questions plus 17 written answer question. For the purpose of this study, we consider the number of 20 questions without regarding the test format or structure design.
Each scenario is evaluated regarding the effort and time necessary to design and create the tests as well as the effort and time necessary to evaluate and grade the tests.

At the end, we analysed and estimated the number of courses or Curricular Units (UC) at UTAD that could be grouped in one of the three proposed scenarios, and estimated the benefits of keeping the actual evaluation model versus the adoption of the b-eval model.

The b-eval model approach should introduce some other indirect and unaccounted benefits, including:
i) the possibility of running analytics on the data store of a set of exams, providing indicators regarding student performance, teaching methodology effectiveness, professors’ teaching success assessment;
ii) the development of a massive and rich set of evaluation questions, regarding specific subjects, that can be used to simplify and automate the production of exams, including the creation of the exam within specific parameters or bias, such as, a determined difficulty degree.

The model also poses some general compliances issues regarding the current higher education regulation and country laws.