Strand C

Inquiry Rationale

Methodology Overview

I hope to focus my problem of practice on an exploration of whether a chatbot that uses natural language understanding can engage students in science-centered conversations. These dialogues between student and computer will be designed, prototyped, tested, and redesigned. They will be evaluated through the lens of whether or not such conversations can be developed into a useful form of assessment.

Chatbots can be designed in a very scripted way, prompting students for predictable answers. They can also be designed to dynamically move the conversation in a particular direction, based on the answers that a student provides. This creates an opportunity to probe student thinking in a more thorough way--one that mirrors what a live teacher might do. Natural language understanding allows a chatbot to understand the many different ways that a student might express an idea. Chatbots can be trained to understand a variety of slang or dialectic speech, allowing for the possibility of an equitable assessment vehicle.

The idea of using a chatbot as a form of assessment draws on several intriguing possibilities:

  1. Many students may be more comfortable expressing ideas in a form of texting than in a face-to-face conversation with peers or teachers. Chatting about their explanations may encourage clarity of thinking within a native ecology for modern teenagers.
  2. A chatbot can be programmed and trained to ask probing and clarifying questions, encouraging the student to more fully build out an idea.
  3. Having individual conversations with every student can be time consuming and impractical for teachers, but utilizing a chatbot can provide teachers with a means to have those conversations by proxy. This would be especially powerful if they are able to view transcripts and if the artificial intelligence yields actionable inferences about student understanding. 
natural language understanding: computer algorithms that can understand input in human language, either written or spoken.
machine learning: Computer programs that can "learn" because they are trained to recognize and respond to data in such a way that it can respond to new, but similar situations.

A Design Research Project

There are many Performance Expectations in the NGSS and the Nebraska science standards that are conducive to using conversation as a component of assessment. For the purposes of this problem of practice, I hope to identify a few of these and design conversation-based assessments with them. As a design project in a relatively unknown area of development, the scope is difficult to fully define at this stage. The challenge of this problem of practice lies in the technical task of constructing a piece of software that will perform in this way, as well as in identifying the possible forks and branches that may occur in such a conversation. A potential strength of the conversation-based format lies in the possibility of leveraging those branches to help make student thinking visible, possibly even helping the student to identify and clarify their ideas in the process. I see two possible pieces to this.

Component 1: Develop a general-usage chatbot to be used as assessment. In a natural, and conversational way, it will:

  1. Present a phenomenon, model, experiment, data set, or some other form of prompt. This could be done using an audiovisual tool or text description.
  2. Ask students to describe their observations of the phenomenon.
  3. Guide students through an assessment task. Possibilities include:
    • Coming up with a testable question
    • Analyzing a data set and making inferences
    • Making a claim and justifying it
    • Design or revise an experiment
    • Construct an explanation
  4. Further the conversation with natural commentary and clarifying questions based on student responses.
  5. Return transcripts and feedback to teachers about engagement and comprehension.

Component 2: Pilot the chatbot as a form of assessment. The researcher will:

  1. Ask a small group of teachers to use the format in the appropriate areas of study.
  2. Collect data from usage and evaluate the quality of the student interactions.
  3. Survey/interview the participants. Gather perceptions about
    • efficacy of the assessment method
    • perceived relative advantage
    • compatibility of the chatbot with their goals
    • complexity of chatbot usage
  4. Use transcripts, observations, and testing results to continuously improve the design throughout the project.