Abstract:

The starting point for this work is a large number of semi-structured documents in which students summarize and evaluate lectures they have attended. The structure of these documents is largely uniform. They begin with five tables containing ratings for a total of fourteen predefined criteria, each assigned a score between 1 and 5. This is followed by a content summary of the lecture. Furthermore, the authors must justify their ratings, explaining the two highest-rated and the two lowest-rated criteria.

This paper presents a project aimed at providing students who create these documents with well-founded and, ideally, automated feedback. Typically, between 40 and 100 such documents are available for evaluation simultaneously.

To support this task, an AI model was developed in an n8n-based workflow environment, which returns three key pieces of information:
1. It verifies whether the textual justifications actually correspond to the assigned numerical scores, i.e., whether the two highest- and two lowest-rated criteria are indeed substantively justified.
2. Next, the system should check whether the content summary of the presentation is complete and covers the four essential aspects: objective, methodology, results, and conclusion.
3. Finally, the system should assess the linguistic quality of the text and identify potential errors in spelling, grammar, and punctuation.

To investigate the feasibility of this task, a study was conducted using several hundred documents. This study employed various large language models (LLMs), including models from Mistral, Microsoft's Phi, OpenAI's GPT, Meta, and Google's Gemma. The models were tested and compared within the n8n workflow.

This paper discusses the results of this study, particularly regarding the practical applicability of such systems for constructive feedback in a university setting. Several key aspects are analyzed: The incremental development of the n8n workflow and the prompts for the LLMs, which was necessary to achieve more precise and reliable results, is discussed. Furthermore, adjustments to the document templates are examined to enable automated and targeted evaluation of the contained information. The results of the implemented LLMs are then compared. Finally, both the advantages and limitations of these approaches are discussed.

Overall, this paper assesses the extent to which locally operated Large Language Models are suitable for providing students with sound and helpful feedback on their written work during practice and learning processes.

Keywords:

Artificial intelligence, Large Language Models (LLMs), automated feedback, learning analytics, text analysis, AI-supported assessment systems.