Abstract:

As digital learning environments generate increasingly rich behavioral data, there is growing demand for competency assessment systems that go beyond single-indicator evaluations. This study presents the design and development of a Multimodal Learning Analytics (MMLA)-based learner competency portfolio system applied to an online K-12 science education program. The system integrates three types of learning data — task-based rubric scores, LMS behavioral logs, and self-report survey responses — to measure five core competencies: scientific inquiry ability, creative problem-solving, critical thinking, self-directedness, and collaborative communication.

A mixed measurement framework is adopted in accordance with the nature of each competency. Task-based competencies are assessed using a criterion-referenced approach yielding 100-point scale scores averaged from session-level rubric evaluations. Log-based competencies are assessed using a norm-referenced approach yielding T-scores (M=50, SD=10): self-directedness is derived from a weighted composite of platform access interval (reverse-scored), e-book study frequency, and reflection submission count, while collaborative communication is computed from log-transformed post counts and activity participation records. Construct validity for each indicator is examined through correlation analyses with self-report survey data.

The indicator selection is grounded in prior empirical research conducted within the same program context. Lee, Park, and Sung (2021) demonstrated that time management — operationalized through login frequency and regularity — was the strongest predictor of academic achievement among self-regulated learning variables (β=.370, p<.001). Hong, Ham, and Lee (2025) further showed that time-series patterns of e-book access frequency distinguished three learner types with significantly different achievement outcomes (F=23.34, p<.001), confirming that login interval and e-book frequency capture the behavioral manifestation of self-directedness beyond mere access counts.

The system will be applied to the full cohort of enrolled learners, producing individualized visualized competency portfolio reports. Anticipated analyses include examination of ceiling effects in task-based competency distributions, T-score distributional properties for log-based competencies, and the relationship between self-directedness scores and task-based outcomes. Detailed application results and validity analyses will be reported in a forthcoming full paper.

This study contributes to accumulating empirical K-12 MMLA cases in a domain currently dominated by university-level research, and offers a methodological framework for mixed competency measurement design applicable across diverse online learning contexts.

Keywords:

Multimodal Learning Analytics, Competency Portfolio, Online Learning, K-12 Education, Behavioral Log Data, Self-Directed Learning.