Abstract:

Students’ online behaviour in two academic English subjects was examined to determine if there is a relationship between students’ first attempt at topic quizzes and their academic performance. For each subject, online quizzes were released weekly on the learning management system (LMS), with quizzes remaining available until the end of semester. LMS data relating to the online quizzes was obtained for the 3078 students who completed the two subjects in the 2015/16 academic year. The plot of the cumulative percentage of students completing each quiz across the semester showed that over half of the students completed Quizzes 1, 2 and 3 in the first two weeks after they were released. However, it took four weeks to achieve a completion rate above 50% for other quizzes. At the end of semester a sudden jump was observed for the completion of quizzes. By week 13, the completion rate for Quizzes 1 to 9 reached as high as 90% while it hovered around 70% for Quizzes 10 to 13, which were released together in week 10. Results showed that fewer students did the quizzes at the time they were released. The completion pattern also showed that many students put off completing quizzes, leaving them to the end of semester.

Decision tree analysis was used to identify at-risk students based on their pattern of quiz completion. In this analysis, students were classified into three groups based on their final subject grade: at-risk students, typical students and high performing students. Membership of these categories was then used as the outcome variable in the decision tree analysis. The tree where three variables (week of first attempt of Quiz 1, 3 and 5) were entered to predict the students’ grade category membership identified that the week of first attempt for Quizzes 1, 3 and 5 were significant predictors and the likelihood of being an at-risk student was higher if Quiz 1 or 3 (together with Quiz 5) was completed late rather than early.

These results have implications for monitoring future students’ learning progress by observing quiz attempt behavior and following up students who do not complete the quizzes or complete them late. Instructors could make use of these findings to develop a notification system to monitor students’ progress. A reference point for reporting on quiz completion could be selected by the instructor, reminding the students to do the quizzes to help keep them on track. The results can also inform setting of quiz deadlines to better align monitoring and intervention. Assigning earlier deadlines could help instructors to support at-risk students according to quiz marks and completion time. To better assess for learning, instructors should also highlight the importance of completing quizzes earlier.

To improve prediction accuracy, other variables could be examined such as marks for each of the quizzes or other student behaviours in the LMS. However, this study has shown that simple indicators like time of quiz completion can predict academic performance as measured by students’ grades in the subject. For this subject, time of quiz completion appears to be an “at risk” indicator that teachers can monitor to help improve outcomes for students.

Keywords:

Learning analytics, online learning, learning management system, academic English subjects.