Abstract:

We present a conversational ITS (intelligent tutoring system) called Watson Tutor, which interacts one-on-one with students via natural language chat. One-on-one tutoring is an effective way to learn deeper knowledge, and the exchanges between human tutors and students can improve learning gains significantly. Our Tutor leverages ITS research and advances in natural language processing to create an interactive dialogue that mimics the questions and feedback responses of a good human tutor.

Students engage with Watson Tutor in a Socratic dialogue by answering or asking questions about a specific learning objective. Each student's understanding of the objective is tracked using a mastery score, which is updated when assessing their answers. The tutoring strategy is adaptive - students are guided through different concepts using questions, hints, pumps, etc., depending on their mastery. Since the Tutor is often anthropomorphized by students, we crafted its tone and personality to be genuine, engaging and nonjudgmental.

The framework for Watson Tutor is designed to scale up efficiently to a large number of domains. Our first large-scale application augments online textbooks on the Pearson Revel platform, across the domains of sociology, US government, and public speaking. Thousands of undergraduate students have used Watson Tutor on Revel in the fall semester of 2018, and more than half of the students surveyed indicated that it helped them to understand the content more than reading alone.

In this paper we introduce a new set of functionalities, which aim to improve learning interactions with Watson Tutor. These are based on transcripts and student surveys from pilot tests: 1) a feedback system for students to correct the Tutor when it makes a mistake, 2) interactive concept grouping activities to test student understanding, 3) text linking of relevant paragraphs to assist unprepared students.

We've developed a feedback system that allows students to vent their frustration in a positive way, when Watson Tutor makes a mistake interpreting or assessing student responses. Students can submit specific corrections by annotating the Tutor's utterances in the context of each dialogue turn. The corrections are reviewed by other students, and if approved, they are automatically used to retrain the Tutor's natural language understanding. In addition, the feedback system can also be used to flag rewrites of domain-specific content that causes issues for a significant number of students.

Watson Tutor can auto-generate concept maps for each learning objective using relevant concepts extracted from domain models. Clusters of related concepts are also identified and used to create visual concept grouping activities, in which students can drag-and-drop concepts. These activities enable students to self-test their understanding, and to receive feedback from the Tutor. This leads to more effective learning of concepts compared to passive observation of a concept map.

Text linking is a type of scaffolding that shows relevant paragraphs to students who are unprepared to answer a question. Watson Tutor displays these paragraphs embedded in the chat either after students give an incorrect answer, or if they request to see relevant text. Up to three paragraphs are extracted a priori for each learning objective from the textbook, using semantic search. We propose to validate the ranking of the paragraphs using student votes or subject matter expert reviews.

Keywords:

Conversational intelligent tutoring system, ITS, Dialogue based tutor, interactive learning.