Abstract:

Creative thinking is a valuable skill in professional and academic settings. Being able to quantitatively define and measure creativity is a fundamental step towards helping students improve it. However, in the context of computer programming, effectively measuring creativity is still an open problem.

In this paper, we present a framework based on clustering to assess the creativity of computer programmers from the code they wrote. In particular, we focus on measuring two dimensions of creativity:
(1) originality, i.e., how much an individual programmer's solution differs from other solutions to the same problem written by other programmers; and
(2) flexibility, i.e., how many substantially different solutions to the same problem an individual programmer is able to write.

First, we train a machine learning model to transform computer programs into code embeddings, which are numerical vectors summarizing the semantics of a program in a compact set of features. We then cluster the vectors corresponding to all the available solutions of a specific programming problem using density-based clustering algorithms. The obtained clusters represent groups of solutions that are semantically similar. We can then calculate an originality score for a new solution as the normalized minimum distance between that solution and all the cluster centers. If multiple solutions written by the same programmer are available, we can also determine a flexibility score for a programmer from the number of clusters that their solutions belong to.

We tested our approach using a dataset containing Java code submissions from four semesters of an introductory Computer Science course at Emory University. We used part of this dataset to train our machine learning model using the Code2Vec software; the remaining part was used for validation and testing. We experimented with multiple clustering algorithms, such as DBSCAN, HDBSCAN, and Mean Shift clustering. Our experiments indicated that Mean Shift was the most appropriate algorithm, as it produced a compact number of clusters with fewer noise points. We then calculated originality scores for students in our dataset for a specific programming problem. We observed that the distribution of the results is consistent with theoretical predictions.

Moreover, we generated additional solutions for the same programming problem using ChatGPT, an Artificial Intelligence (AI) system able to generate computer code. We calculated originality scores for these AI-generated programs, and compared them with the originality scores of the solutions written by human students. We observed that these AI-generated programs had higher originality scores than the average human student in our dataset.

Our experiments showed the viability of our approach towards a computational definition and measure of creativity in computer programming, at least regarding the originality dimension of creativity. As a next step, we are currently collecting new data to test our approach for the flexibility dimension of creativity, which requires us to collect multiple solutions written by the same student.

Keywords:

Creativity, Originality, Computer Programming.