Abstract:

Multiple-choice questions (MCQs) scale efficiently in large cohorts, yet feedback in practice often stops at a score. Learners may see which options are correct, but receive limited support to diagnose misconceptions, derive actionable learning goals, and regulate exam preparation. We present MC-Test, an open-source web platform for formative MCQ practice that integrates a didactic item model, a schema-bound LLM generation pipeline, learner-facing analytics, and pacing controls aimed at reducing rapid guessing.

MC-Test operationalizes Bloom levels 1–3 (recall, apply, analyze) and links each item to topic labels, learning objectives, and an item weight. Item generation follows a finite-state workflow: context assembly (course/topic/objective and target Bloom level), draft generation, automated JSON schema and semantic validation (e.g., required fields, unique options, valid answer index), and an optional repair loop that prompts minimal edits when violations are detected. The structured outputs enable automated checks and support planned psychometric analyses.

After each practice session, MC-Test provides brief rationales for every answer option, an optional mini-glossary, and exports of learning objectives and session reports (including PDF artifacts). Learner dashboards implement a diagnostic funnel from overview to action: a topic performance chart, a Bloom-level radar profile, concept mastery columns (understood/not understood/not attempted), and a topic×cognition heatmap that highlights both weak areas and avoidance patterns. For instructors and item authors, an item diagnostic view aggregates confidence×correctness patterns in a cumulative confidence matrix to flag potentially misleading items.

To mitigate impulsive clicking, MC-Test adds a configurable pacing layer treated as a testable design hypothesis rather than a guaranteed deep-processing intervention. A pre-answer cooldown temporarily disables submission based on item weight and text length. A post-answer reflection window delays navigation to encourage feedback inspection; durations scale with explanation length (standard +10 seconds; extended +20 seconds). A “panic mode” disables pacing when time becomes critical (e.g., <15 seconds remaining per item), prioritizing completion and fairness. All pacing parameters are course-level configurable to support accessibility accommodations.

MC-Test was prototyped with commercial LLM APIs and migrated to an on-premises backend using Ollama, keeping prompts and learning data within institutional infrastructure and improving cost predictability. Data minimization, audit logging, pseudonymous participation (e.g., Nobel/Turing laureate names), and role-based access control support privacy-by-design deployments. A first usability study (System Usability Scale, N = 20) yielded a mean SUS score of 70.38 (“OK”). Ongoing work evaluates behavioral effects using log-based indicators (response times, pacing interactions, and feedback-viewing time), assesses Bloom-label reliability via double-coding, and triangulates confidence patterns with classical item statistics to improve item quality over time.

Keywords:

Formative assessment, Multiple-choice questions (MCQs), Large language models (LLMs), Automated item generation, Learning analytics, Test-taking behavior (rapid guessing), Self-regulated learning, Privacy-by-design (on-premises AI).