Abstract:

At the Faculty of Informatics of the University of Debrecen, we have been using the ProgCont system to evaluate programming tasks since 2011. Our students encounter this software in our programming competitions, in class and during exams.

The system works by compiling the source code submitted by the student, then typically running (testing) it several times and checking the results of the run. The tasks set out in the system specified in detail what input the submitted program should expect and also indicates the format in which the result should be presented. Of course, the functions and features of the system must also be demonstrated to the students so that they can work with it effectively. For this reason, we have put together several exercises to give elementary examples of how to use the system.

The importance of the system is growing year by year, and during the recent pandemic, we have tried to take advantage of the benefits of automatic assessment in the context of self-study. One of the difficulties is that the familiarisation with the system in distance education takes place with significantly less instructor supervision. The use of the system has to be learned independently.

In response to the latter challenges, we have embarked on a journey to improve the system's feedback. We aim to provide automatic help for incorrect solutions submitted, especially during stand-alone preparation, to help identify the source of the error. For this purpose, in many cases, we have reformulated the inputs for testing the programs submitted for each task and provided them with labels (annotations) that highlight the specificity of the input. The result of the evaluation of a failed program shows which tests with specific features the program failed. We would like to use this technique in the exercises to learn the ProgCont system too.

This article presents new annotations for 3X3 exercises that we recommend to students in the initial phase of getting acquainted with the system. The exercises themselves have been available in the system for several years, so we have been able to test the new annotations by re-evaluating the previous solutions with new annotated test cases. As a result, we are replacing instructor assistance lost due to distance education, providing developers with important information on how to adapt the system to make it easier to use, and supporting instructors by highlighting typical mistakes made by students; they could incorporate this knowledge into the curriculum when attendance education is restored.

Keywords:

ProgCont system, automatic evaluation, test case annotations, programming tasks.