Abstract:

With their popular public introduction, large language models (LLMs), especially ChatGPT, have come to the fore and been a topic of endless discussions from contributions to disruptions in the society. Surrounded by possible revolutionary claims, Generative AI (Gen AI) has been used for various educational purposes as well as supportive purposes such as using ChatGPT for the qualitative analysis. Due to rapid explosion of information, the volume of scientific articles has been on the rise. To provide peer-review for these articles before getting published, voluntary reviews from scholars in the relevant fields are needed by the academic journals. As this peer-review process is voluntary and takes considerable amount of time, the scholars must spare their time and energy for this process in addition to the other works they are responsible for in the academy. Therefore, it is getting difficult to find reviewers day by day. ChatGPT can act as a companion for the scholars and provide critical comments for the articles that the scholars need to review.

This qualitative study, mainly a document analysis, evaluates how ChatGPT can function as a reviewer for scientific articles and how well can ChatGPT evaluate a scientific article. For this reason, two scientific articles on online learning were selected (one submitted to a journal indexed in Emerging Sources Citation Index-ESCI and one submitted to a journal indexed in Social Sciences Citation Index-SSCI) and reviewed by ChatGPT and a scholar. First, the review process was tried on ChatGPT without any specific prompts. Then specific prompts were constructed based on journals’ review requirements. While doing this, one researcher evaluated two articles manually. These three review processes were compared. The results showed that the reviews done by ChatGPT were successful, and they included critical evaluations of the articles. It was found that the constructed prompts effectively aligned with the review requirements of the two journals. What is more, the evaluations of ChatGPT were in line with the evaluations of the researcher. This makes ChatGPT effective and efficient for reviewing journal articles and a possible companion for scholars.

The findings have implications for designing effective scientific article review process and writing necessary prompts to review academic articles, fostering scholars to review more articles in a relatively short time and enhancing their review performance. Overall, ChatGPT can help scholars regarding the review process and save them time and energy. As a result, they can spare more time for other academic purposes and become more productive.

Keywords:

ChatGPT, generative artificial intelligence, artificial intelligence, ai-based review, peer review, large language models.