Identifying cognitive and metacognitive processes in SRL during complex problem solving in an OELE

Imagine a novice learner interacting with an open-ended learning environment (OELE) to solve a complex engineering problem. To learn effectively, a learner has to undertake several cognitive and metacognitive processes, e.g., activate prior knowledge, set meaningful goals for her/himself, implement several strategies to execute the task, monitor her/his emerging understanding, and follow advice from system-generated scaffolds, etc. When a learner undertakes these processes, she/he does it in a tacit/implicit manner which may not be overtly visible to us unless self-reported (e.g., “I should start from this goal first”). We call this plane the ‘invisible plane’ (Refer to the figure above), which can be interpreted if a learner verbally reports an account of her/his thinking. Several theoretical frameworks/models outline the various cognitive and metacognitive processes that a learner might undertake. For example, Pintrich (2000) outlines metacognitive processes such as ‘target goal setting’, ‘judgement of learning and comprehension monitoring’, etc. 

On the other hand, what is visible when a learner solves a complex problem in an OELE are the computer-generated traces, also known as log data. For example, in the figure above, the segment labelled 'observable plane’ represents that the learner ‘viewed the problem statement’ at time 0:01:12, which was followed by an action that represented 'selecting a sub-goal' at time 0:02:31. Since these actions can be observed and recorded automatically, we call this plane an ‘observable plane’. 

Our research goal is to identify the cognitive and metacognitive processes, such as the ones listed in the invisible plane, using the actions captured in the observable plane. To accomplish this goal, we have collected think-aloud verbalizations and trace data of learners interacting with a problem-solving OELE (i.e., MEttLE) across a synchronized timeline. As our solution approach, we:

• Developed a coding mechanism based on the theory and pedagogical design of OELE to identify the cognitive and metacognitive processes and validate them using think-aloud protocol-based research studies.

• Developed theory-pedagogical design and data-driven automatic mechanisms to detect the metacognitive processes automatically using computer-generated trace data and evaluate the solution by triangulating with ground truth. 

• Applied the automatic model to identify the metacognitive processes and analyse its impact on learners' complex problem-solving in an OELE.