Rimac: Improving a Natural-language Tutoring 

System that Engages Students in Deep 

Reasoning Dialogues about Physics

Motivations:

The Rimac project addresses two of the challenges that President Obama raised in his address to the National Academy of Sciences (Obama, 2009, included in Hamblin, 2009)—one educational, the other technological. The educational challenge is to increase U.S. students’ understanding of fundamental scientific concepts and principles, and their ability to use this knowledge to solve real-world problems.  Students’ failure to grasp basic scientific knowledge has been a persistent problem, especially in physics education.

President Obama's technological challenge is — in his words, to build “learning software as effective as a personal tutor” (Hamblin, 2009).  Several studies have established that effect sizes for one-on-one tutoring human tutors range from .4 to 2 sigmas relative to classroom instruction (e.g., Bloom, 1984; Cohen, Kulik, & Kulik, 1982; Corbett, 2001). Some of the most sophisticated intelligent tutoring systems (ITSs) have achieved effect sizes of .3 to 1.0 sigma, relative to classroom instruction on the same content (Corbett et al., 1999) so there is room for improvement. 

Previous Work:

As several cognitive scientists have noted, a critical first step is to determine what the key elements of human tutoring are that account for its effectiveness (e.g., Chi et al., 2001; Graesser et al., 2003). Research by Chi and colleagues (2001) revealed that it isn’t so much what tutors do during tutoring sessions that is important, nor what students do, but how (and how much) the student and tutor respond to each other’s conversational moves. In other words, the interactivity of human tutoring drives its effectiveness.  Of course, this important finding raises more specific questions: Exactly what is ‘interactivity?’ How is it achieved via language and dialogue? And what particular linguistic mechanisms predict learning?

In an exploratory phase we explored whether interactivity in dialogue is at least partly achieved through, and signaled by, linguistic cohesion—that is, lexical and semantic mechanisms such as word repetition, synonyms and paraphrases, and superordinate class (hypernym)/subordinate class (hyponym) relations (Halliday & Hasan, 1976). We used these relations to code a large corpus of dialogues between human tutors and physics students that took place after students solved problems in the Andes physics tutoring system (e.g., Gertner & VanLehn, 2000; Katz, Allbritton, & Connelly, 2003; Schulze et al., 2000; VanLehn et al., 2000, 2005). We found a positive correlation between “abstractive” dialog moves, in which the student or tutor repeated the other’s previous utterance but at a greater level of generality (a superordinate class relation), and pretest to posttest gains scores. We also found that tutor moves which repeated the student’s previous utterance but in a less abstract way (a class member relation) predicted learning.  Upon closer inspection, we noticed that superordinate-class and class-member spans signaled at a lexical level the existence of more sophisticated semantic connections between the student’s and tutor’s utterances—specifically, the dual processes of abstraction and specialization, respectively. Abstraction occurs when the tutor or student relates two or more narrower concepts together—for example, instantaneous acceleration and constant acceleration to their parent concept, acceleration. Specialization is the reverse, and typically occurs when the tutor (or student) distinguishes between related concepts.

Objectives:

We are developing a research platform that will ultimately allow us to test the hypothesis that abstraction and specialization not only predict learning, but cause it. Specifically, we will develop a system that will engage students in natural-language (NL) dialogues after they solve quantitative problems. The computer tutor will guide students, through natural-language dialogue, in answering deep reasoning questions. Like the human tutors that we observed in our exploratory study, the computer tutor will be designed to be sensitive to the level of abstraction of the student’s input at various points during the dialogue. The computer tutor will prompt the student to abstract or specialize, when appropriate, or will model these processes through its own dialogue moves.