Abstract:

At San José State University, we have developed CodEval, a code evaluation tool integrated with Canvas Learning Management System, that automatically evaluates students' work within a few minutes of the submission. Early feedback encourages students to submit their assignments early and gives them a clear idea of the quality of their submission. CodEval catches bad submissions early and does the tedious aspects of grading, such as compiling and running tests, leaving graders more time to spend on the qualitative aspect of grading. Instructors also get early insight into student success.

Students who receive immediate feedback on their programming projects are likely to start early. Students no longer have to wait for the assignment deadline to get results which encourages participation and engagement. Students can now go through numerous versions of assignment submissions until they are satisfied with the results.

This process also reduces the grader's work by ensuring that missing files or incorrectly submitted submissions are immediately caught. We have developed the assessment specification beyond current automatic assessment tools that only check program output. CodEval also checks for generated files, program arguments, exit codes, compilation errors, usage of key functions, and memory checks. Graders can now focus more on the aspects of grading that cannot be programmatically evaluated, such as whether the logic is used correctly or hacked to derive the desired output.

Before using CodEval, instructors would not have a clear view of the student's comprehension of the concept evaluated by the assignment until after the due date. CodeEval helps instructors identify and address the gaps in students' understanding and thus helps more students successfully complete the assignment.

Canvas LMS is a learning management system that has grown in popularity and is now the industry leader in learning management systems. Periodically, CodEval downloads new student submissions from Canvas. CodEval evaluates the submissions against the evaluation criteria written by the grader in a specification language that we developed. We use Docker to create the environment to evaluate the submission, thus safeguarding the host machine from any modification or data loss that may incur due to the student's code. Initially, CodeEval was used to evaluate assignments written in C for our Operating Systems class. We now also use it to evaluate assignments for another class using Java. Based on this experience, supporting other languages is a simple change to the evaluation specification.

CodEval is an easy tool for students, graders, and instructors and seamlessly integrates with Canvas. It is used in two classes with a total of 90 students and multiple coding assignments with promising results.

Keywords:

Computer science education, programming exercises, teaching assistant, code evaluation, learning management systems, canvas, docker.