Abstract:

Assessments are a fundamental part of the educational process and are used to diagnose students’ learning difficulties. However, performing frequent evaluations and with fair judgments on assigning grades takes a lot of work and is time-consuming. An automatic question generator can contribute to the generation of several different assessments and assist the correction process, especially in combination with the Item Response Theory (IRT). IRT is a powerful mathematical tool to assess latent abilities in many cognitive areas. An artificial-intelligent assistant can help teachers to keep up with students’ learning trajectories.

Therefore, we propose the use of cutting-edge technologies both to aid teachers in formulating more accurate assessments and a better measure of the student’s abilities.
The strategy is to provide this robot with the capability to read documents from new domains and formulate evaluation queries.

To validate our proposal, we created 64 questions and performed two experiments with different students’ clusters. We analyze real students and compare the results with artificial students. In our experiments, we constructed artificial assessment items to understand the mathematical structure of the coefficients in specific scenarios.

We estimated the students’ IRT parameters and latent ability from the questions applied to the control class. Compared to the average number of correct answers, as is usual in evaluations, the estimated ability of the IRT correlates with 0.97, a quadratic residual error of 0.20, demonstrating to be close to the traditional assessment but more efficient in pedagogical terms since it describes the probability of success of the students examined. The results showed that it is possible to personalize evaluations and improve teachers’ tools to qualify and infer the students’ abilities.

Keywords:

Artificial Intelligence, Natural Language Processing, Educational Innovation, Conversational Systems.