Tech Details

For this conference, no tech is required, though you may want to bring your laptop.

Overview of the DDCI Process

DDCIs are designed to allow researchers to interview students at specific, relevant times (after a specific event) in their learning experience, but what those specific events might be varies by both the learning system and the research question. This means that DDCI research requires significant planning. Researchers must have access to the back-end of the learning system so that they can integrate the QRF server, which “listens for” events of interest, prioritizes them (according to predefined criteria), and then pushes them to the App.

Although QRF is open-source, it can only detect what is currently logged and/or modeled in the learning system. That is, when DDCIs were used to study epistemic emotions in Betty’s Brain (Hutt et al., 2019), researchers first had to develop models of these emotions that could detect them in real time from students’ interactions with Betty’s Brain, so that these could be used to trigger each interview.  However, QRF need not be triggered on complex, machine-learned models. For example, in research studying the development of STEM interest, researchers triggered on events like "student talked to an NPC" and "student added blocks to their Mars base."

The open-source code for QRF can be found here: https://github.com/pcla-code/QRF 

As Figure 1 shows, QRF integrates with existing student modeling technologies (e.g., behavior-sensing, affect-sensing, detection of self-regulated learning) to alert researchers to key moments in a user's experience. QRF ‘listens’ for events (e.g., interaction patterns) and identifies moments of interest, prioritizes them, and directs interviewers accordingly, allowing the interviewer to record DDCIs directly in the app along with relevant metadata (e.g. participant ID, time stamps, trigger information, etc). In addition to pushing triggers to the mobile phone, it also pushes them to a dashboard that allows fellow researchers to monitor the triggering events in real time.

Figure 1: Overview of QRF System

How do DDCIs compare to previous methods for contextualizing student interactions with online learning systems?

To date, there are two primary methods for obtaining real-time qualitative data about students' interactions with a learning software. One is to run various kinds of classroom observations, often with limited interactions between the researcher and the students. The other is the Think Aloud (TA) method, a technique grounded in information processing theory, where students are asked to verbalize their cognitive processes. The latter technique facilitates significantly more access to students' internal thoughts, but it has been criticized for increasing the cognitive load of the activity and for being difficult to implement in User Experience (UX) Design situations which cause frustration, which may be the exact time that researchers studying digital learning would like to hear the most from a given student. Moreover, it can be difficult to conduct in a regular classroom. Still, TA was created over concerns about alternative methods, including the ways in which memory–even when it is functioning ultimately–consolidates and clouds people’s interpretation of events. These concerns mean that TA can be preferable to alternatives like retrospective interviews.

DDCIs address some of the methodological concerns related to the TA approach while facilitating real-time access to students cognitive, affective, and metacognitive processes. By having a researcher direct questions towards students at optimal times, we offer several advantages over TA, including (a) we are less likely to either interfere with the totality of the learning process and/or to risk fatigue because we are only asking students to talk to us at specific times, (b) we are not asking them to read our minds (i.e. to figure out what information about their experience is relevant to our research questions), which subsequently (c) reduces the cognitive burden of the research process. That is, if a student’s verbalizations are incomplete, the researcher is there to ask follow-up questions that help us to distinguish between the student not having any information to share about their experience and the student not remembering to explain those details while also trying to learn.

Moreover, DDCIs increase the flexibility of the research process, as research subjects often require significant training to be able to understand what a researcher needs them to do during a Think Aloud protocol. In contrast, DDCIs allow the researcher to modify the questions that students are trying to answer during the research process. This flexibility has allowed us to increase the range of questions that we asked about students’ self-regulated learning , but it has also facilitated rapid design changes to a pedagogical agent that substantially improved students’ interactions with that system.

That said, the process of connecting with students across repeated interviews requires careful planning and practice. Researchers must identify appropriate times to conduct interviews (designing triggers for the QRF app), must design these triggers in a way that does not interrupt a given student too frequently, and then must execute these interviews in a way that builds trust with students while also asking interview questions that reflect their own research goals. In this tutorial, we will cover strategies required for all stages of a DDCI project, providing training to participants who are interested in conducting this–or similar research–on their own.