Abstract:

Natural language processing (NLP) has made significant advancement in recent years, transforming how humans interact with technology. Large language models (LLMs) such as ChatGPT, Microsoft Copilot, and Google's Gemini have disrupted various sectors, including higher education, particularly since 2019. Prompt engineering, a technique to customise prompts entered in these LLM platforms, has gained increased attention. However, there is an existing gap regarding the application of this technique in information technology (IT) education. This paper presents a study aimed at bridging this gap by examining how prompt engineering can enhance LLMs for educational purposes in the IT domain. The study investigated how different prompt engineering techniques can improve code generation by LLMs, thereby assisting learning of students in higher education. We analysed the effectiveness of three different prompt techniques namely zero-shot, few-shot, and chain-of-thought for code generation. We used zero-shot prompt technique in GPT 3.5 to create code for 2 different kinds of code generation (function generation and snippet generation). We received accuracy of 62.7% for code function generation and 57.9% for code snipped generation. Next, we used few-shot prompt in GPT 3.5, which resulted in more accuracy of 78.5% for code function generation and 71.1% for code snippet generation. Finally, we used chain-of-thought prompt in GPT 3.5 that resulted in 69.7% for code function generation and 56.5% for code snippet generation. We also checked if there was correlation between our given performance metrics variables. Our results indicate that few-shot prompt technique that uses prompts with few instructions and examples is the most effective technique among the examined techniques to improve code generation. We also found that performance metrics - accuracy, precision, recall and f1-score were highly correlated with exact match score which signifies a strong measure for the desired output. The few-shot prompt technique can benefit students by facilitating faster learning by helping students quickly generate desired outputs without repetitive prompting. Further, it can reduce frustration by guiding students to use fewer instructions and examples. Learners can experiment with different prompt techniques to see how they affect code generation, which can help develop a deeper understanding of how code works and how to write it effectively. Hence, this study paves way for educators and researchers to encourage exploration and experimentation with personalised prompts and effective prompting styles to enhance students' learning experience and skills in using generative AI.

Keywords:

Prompt engineering, large language model, GPT, tech education, few-shot, zero-shot, chain of thought.