Abstract:

The emergence and widespread adoption of Large Language Models (LLMs), such as ChatGPT, have significantly expanded the possibilities to support computing education. In Software Engineering, recent studies suggest LLMs can assist a range of development activities, including requirements engineering, software testing, documentation generation, and task automation. Despite its potential, the literature still lacks empirical evidence supporting the effective integration of LLMs into teaching and learning, particularly concerning students’ perceptions of their use.

The present study investigates undergraduate students’ perceptions of usefulness, mental effort, and performance when using ChatGPT in Unified Modeling Language (UML) based software modeling activities. The goal is to generate empirical evidence that informs the discussion on the adoption and integration of generative AI tools in computing education. We employed a controlled experiment with two participant groups: an experimental group using ChatGPT, and a control group without access. Participants performed software UML modeling tasks, including creating class, use case, and activity diagrams. Quantitative and qualitative data were collected. Quantitative metrics assessed task performance in terms of efficacy and efficiency, while qualitative analyses were grounded in theoretical models of technology acceptance, specifically the Technology Acceptance Model (TAM), and the NASA Task Load Index (NASA-TLX) to measure perceived cognitive load.

Results indicate that students using ChatGPT reported higher confidence in task execution (88.2%), lower perceived mental effort, and reduced time pressure compared to the control group. The experimental group also demonstrated superior performance, evidenced by higher accuracy rates and improved conceptual understanding of UML modeling elements.

These findings suggest that ChatGPT can enhance student performance in UML modeling tasks, leading to higher accuracy, deeper conceptual comprehension, and faster task completion. Additionally, tool use was associated with greater confidence and reduced frustration, likely due to the immediate resolution of doubts, which mitigates stagnation in problem-solving. High levels of technological acceptance and strong intentions for continued use highlight the potential of generative AI as a supportive resource in computing education.

However, the results also raise an important pedagogical issue. The reduction in perceived cognitive effort may negatively affect long-term knowledge retention, which is a negative outcome. Therefore, generative AI, such as ChatGPT, should be integrated into educational activities as a facilitator and mediator of learning rather than as a replacement for critical reasoning and active knowledge construction by students.

Keywords:

Education, UML, LLMs, cognitive load.