Abstract:

The increasing use of large language models (LLMs) in academic and professional environments raises important questions regarding the reliability of AI-generated information, particularly in domains where accurate information retrieval is critical, such as software engineering risk management. This paper presents an empirical evaluation of LLM-generated news retrieval within the context of software engineering risk identification. The experiment was conducted in the course Software Engineering Risk Management of the Computer Engineering degree at the University of La Laguna (Tenerife, Spain).

This study provides two main contributions:
(1) an empirical evaluation of LLM behaviour in a structured information retrieval task related to software engineering risk events, and
(2) a reproducible process for analysing the reliability of AI-generated news outputs through manual verification of metadata consistency, thematic classification, and link validity.

A controlled protocol was designed to analyse the behaviour of an LLM system in a structured information retrieval task. A standardised prompt was executed using the same LLM version (ChatGPT 5.2). Each participant ran the prompt in a new conversation without prior context and transcribed the generated output without modification into a PDF document. The prompt required the generation of 10 news items related to risks in computing and software engineering, including title, year, category, source link, and summary. Each generated entry was analysed using a structured LLM output evaluation approach, verifying publication year validity, title accuracy, thematic classification, and link functionality.

Data collection took place between 19 February and 10 March 2026. A total of 9 participants produced 90 news entries that were manually analysed to construct a dataset for quantitative evaluation. Results show that 77/90 (85.6%) of the entries correspond to news published between 2024 and 2026. Regarding topic classification, 66/90 (73.3%) were related to software engineering, 22/90 (24.4%) to general computing, and 2/90 (2.2%) to both. Only 18/90 (20.0%) of the generated titles exactly matched the original news titles, indicating frequent paraphrasing by the LLM.

Keyword analysis of titles and summaries revealed recurring themes such as zero-day vulnerabilities, generative AI coding tools, major infrastructure incidents (e.g., CrowdStrike-related outages), supply-chain risks, and cloud infrastructure failures (e.g., Cloudflare or AWS). Additionally, 16/90 (17.8%) of the news items were explicitly related to generative AI technologies. Metadata validation showed that 80/90 (88.9%) contained a valid publication year.

All generated links were manually verified through a two-step validation process: checking link functionality and confirming correspondence with the referenced news article. In this dataset, 90/90 (100%) of the links were functional and correctly associated with the referenced news item.

Overall, the results indicate that LLM systems can retrieve recent and thematically relevant news related to software engineering risks, although limitations remain in title accuracy and thematic concentration. These findings suggest that while LLM-generated outputs can support risk awareness and information discovery, their results should be critically verified in academic and professional contexts.

Keywords:

Generative AI, Large Language Models (LLMs), Responsible AI, AI in Education, Computer Science Education.