Abstract:

Reading comprehension is a common form of exercise for language learning. Designing these exercises is time consuming, since it involves selecting a suitable reading comprehension passage, composing challenging questions and answers, as well as plausible distractors in the case of multiple-choice (MC) questions.This paper describes a web-based tool that assists teachers in authoring MC question items for reading comprehension using Large Language Models (LLMs).

While recent advances in generative AI has facilitated a number of question generation models, they have yet to fully realize their pedagogical potentials due to several limitations. First, the question category is not controllable. Many questions can be asked about a passage, ranging from straightforward ones based on information retrieval, to those requiring in-depth reasoning. Most question generation models are not trained to aim at any category. According to the internationally recognized PIRLS (Progress in International Reading Literacy Study) assessment standards, reading comprehension requires four distinct processes: retrieval, inferencing, integrating and evaluation. Our tool enables teachers to request specific PIRLS categories, mitigating the dearth of higher-order questions generated by current approaches.

Second, the distractors may not reveal student needs. Most distractor generation models are trained to find plausible distractors by maximizing semantic similarity with the correct answer. While these distractors may be able to attract students, they provide little insight beyond the fact that the student failed to pick the correct option. Adopted by Pearson and other educators, the Distractor Rationale Taxonomy posits that a distractor should diagnose the student’s weakness. In MC items for reading comprehension, distractors can reveal how the student misinterpreted the text, for example, whether he or she misunderstood a semantically ambiguous word, or failed to identify the text span that contained the answer. Teachers will gain insights into the student’s level of understanding and give more targeted feedback.

By addressing these limitations, our online tool offers a more controllable, learner-centric environment for developing multiple-choice questions for reading comprehension. Given the complex interactions between the question, the answer, and distractors, an LLM may not be able to directly produce an optimal output. We leverage Chain-of-Thought prompting to guide the model in making intermediate decisions, such as the inference that the student is expected to make, and the text span that forms the basis of the question and answer. A user study evaluates the efficacy of human collaboration at these intermediate steps, and the degree to which it increases model transparency and the quality of the generated items.

Keywords:

Large Language Models, reading Comprehension, language learning.