Abstract:

Higher Education institutions are investing in the development of students' transversal competencies to respond to labor market demands. In such a context, the relevance of a measure to assess graduate students' job readiness has been highlighted. The Multiple Mini-Interviews (MMI) are widely used to assess transversal competencies (e.g., teamwork) of applicants to medical schools. An ongoing project is researching the feasibility of adapting the MMI to assess transversal competencies across different scientific fields.

This paper presents the pilot study conducted to evaluate the validity of MMI scenarios to assess transversal competencies’ profiles of graduate students. The study paid attention to the criteria used to assess the robustness of the testing procedures.

The examinees were 16 students from two Portuguese universities and different scientific fields (e.g., Economy, Education). The study also involved 19 interviewers and five standardized characters (SCs; trained persons to role-play with students in five of the MMI stations). The students were organized in two separate MMI circuits, with 10 stations designed to assess: problem solving, generation of novelty, open mind-set, learn to learn, positive attitude at work, teamwork, communication, leadership and market orientation. Interviewers rated students’ global performance and each competency on a scale from 1 to 10, when compared with other examinees.

After each MMI circuit, the examinees were interviewed in a focus group and were requested to fill in a questionnaire on the MMI relevance, adequacy, fairness, agreeableness, clarity of the instructions, duration, organization and possible bias.

To evaluate the MMI’s fitness to the purpose and ability to discriminate individual performances, the interviewers and SCs participated in a focus group. Interviewers also responded to a questionnaire on their perceptions about the station they were involved in (e.g., strengths and weaknesses, suggestions for improvement, competency that stood out).

The overall scores of the MMI showed that the scenarios discriminated participants according to the target competencies and that it gathered good reliability indices for internal consistency and interrater agreement. The analysis of the inter-rater reliability agreement among raters of the same competency in different stations evidenced, in some cases, lack of consensus in the ratings. A good degree of agreement is observed when different interviewers assessed the same student, in the same station, focusing on the same competencies.

The analysis of the students’, interviewers’ and SCs’ perceptions revealed that, overall, the MMI is potentially a method fit for the purpose of assessing transversal competencies necessary for the job market. In addition, students agreed that the instructions were clear and the length of the stations and the organization were adequate, and considered the MMI to be fair, agreeable to take part and unbiased.

This pilot study evidenced, overall, that this MMI-like method is very promising due to its adequacy and utility among HE institutions to assess reliably a wide range of graduates’ transversal competencies. This pilot study also made it possible to: identify which scenarios were more adequate for the assessment of transversal competencies and which needed clarification in their description and in the criteria for assessment; analyze suggestions for the SCs’ roles definition, and; provide practical knowledge of the MMI application procedures. Future development and validation studies are needed to confirm the fit of the MMI to assess transversal competencies in different work contexts. Currently, to overcome the COVID-19 demands on physical distancing, the MMI transformation to a digital format is in progress.

Keywords:

Assessment, graduate work, higher education, multiple mini-interviews, transversal competencies.