Abstract:

Massive Open Online Courses (MOOCs) generate rich traces of learner activity, and clustering is frequently used to identify “at-risk” learners and learning gaps. However, many studies primarily rely on internal validity indices and visual inspection of clusters, with limited evidence that the resulting profiles are pedagogically meaningful. In this work, we report lessons learned from a multi-algorithm clustering study on a large introductory MOOC, where the main goal was to separate learners according to their risk of failing or not achieving the intended learning outcomes.

Using clickstream and assessment data, we compared several clustering designs that are widely used in learning analytics: K-Means on aggregated engagement features, phase-based K-Means on early/mid/late activity, density-based clustering with DBSCAN and OPTICS on Dynamic Time Warping distances, trajectory clustering with DBA-KMeans, and local outlier detection with LOF. All models were evaluated with classical internal validity metrics (silhouette, Calinski–Harabasz, Davies–Bouldin), but also with external criteria directly related to learning gaps: final grade distributions, certification rates, and the proportion of low-performing learners in each cluster.

Our findings show that many apparently “good” clusterings did not produce educationally useful groups. Several solutions with high internal validity collapsed almost all low-performing learners into every cluster (up to 97–100% low performers), or produced highly fragmented structures with only marginal differences in learning outcomes. These failures highlight how MOOC-specific data properties—strong class imbalance, sparsity, and heterogeneous engagement trajectories—systematically undermine naive clustering choices. Outcome-aware feature weighting in a hybrid K-Means model yielded clusters that aligned with distinct risk levels, while density-based methods provided complementary, fine-grained anomaly signals related to learning gaps.

Based on this experience, we offer concrete design guidelines for learning analytics practitioners who wish to use clustering for learning gap detection. We argue that internal metrics must be complemented with outcome-based validation, that temporal representations do not automatically yield actionable risk profiles, and that MOOC-specific adaptations are essential to move from attractive visual typologies towards clusters that genuinely support targeted pedagogical interventions.

Keywords:

Learning analytics, MOOCs, Clustering, Learning gaps, Outcome-based validation, Hybrid K-Means.