Abstract:

There is an increasing demand for university evaluation to assure the quality of higher education. To maintain the quality of the evaluation system, it is necessary to improve the quality of the quality assurance system based on past evaluations. In Japan, as well as in other countries, the evaluation process relies on peer review based on universities’ self-assessment. Therefore, it is important to clarify and visualize what is going on as part of the judgment of the peer-review process for verification. However, few studies have been conducted on the peer-review judgment process. In this study, we analyzed differences in the ratings of education and research performance levels between self-assessments reports submitted by universities and evaluation reports reflecting evaluators’ judgment.

A part of the evaluation reports and self-assessments reports conducted for the national universities in FY2010 were used for the analysis. We examined the performance levels of education and research expressed in the analysis reports of 1,243 faculties and graduate schools that were evaluated. Moreover, the performance levels described in the self-assessment reports submitted by the universities were also analyzed in comparison with the evaluation reports.

Three-way factorial analysis of variance was conducted to compare the main effects of the types of evaluation, fields of academic study, and evaluation viewpoints as well as the interaction effects among them on the ratings of the performance levels of the faculties and graduate schools for educational and research evaluation data, respectively. The type of evaluation consisted of two levels (evaluation results and self-assessment), the fields of academic study included ten levels (e.g., social science, technology, etc.), and the evaluation viewpoints included five levels (e.g., educational system, learning outcomes, etc.).

For educational evaluation, all the main effects were statistically significant in the types of evaluation (p < .01), fields of academic study (p < .01), and evaluation viewpoints (p < .01). Significant interactions were observed in two pairs of factors—between types of evaluation and fields of academic study (p < .01) and between types of evaluation and evaluation viewpoints (p < .01). For research evaluation, only the main effects of the types of evaluation (p < .01) and the fields of academic study (p < .01) were significant, and no interaction was observed.

These results indicate that self-assessment is rated higher than the evaluation results both in educational and research evaluation. This could be due to two reasons. One is because of the characteristics of the university corporation evaluation that the result is reflected for the calculation in the assessment of management expenses grants. The pressure to acquire better evaluation results makes universities assess themselves better. Another possible reason can be the psychological process of the “better-than-average effect.” This phenomenon refers to the psychological tendency to overestimate self-evaluation compared with the average. These results provide new insights into building transparent evaluation methods in quality assurance agencies for higher education and universities as well as provide basic data that clarify the judgment process of evaluation between self-assessments and evaluation results.

Keywords:

University evaluation, performance-based evaluation, evaluation reports, analysis of variance.