Embedded Files
Process tracing methods are a set of techniques designed to identify the abstract knowledge representations of a person involved in a task. This is typically done during the preliminary phases of research to uncover the nature of expert declarative and procedural knowledge.
Below is an example of two process tracing methods in action:
Think-Aloud
Semi-structured Interview
Example
Research Question: How do expert Rocket League players make rapid and accurate game decisions?
Rocket League is a video game that could be best described as, “car soccer.” There are two teams, orange and blue, and the objective is for each player and their partner to put the ball in the opposing team’s net. All games begin with 5 minutes on the clock. Once that time elapses, the team with the most points wins the game. If no team has scored or the two teams are tied, the game continues until someone scores a point. This is a highly motor and cognitive task; Players have to push multiple buttons on their controller in close succession to control their camera, chat with other players (optional), and navigate their car around the field. Cognitive demands include perceptual skills, situation awareness, and decision making.
I chose to use the retrospective think-aloud technique to elicit knowledge about actions and decision-making strategies in-game. I decided to conduct the think aloud immediately following the end of the game to mitigate some of the problems that can be associated with concurrent think-aloud reports (e.g., task interference). The verbal report was facilitated with a video recording of the game, where the participant attempted to verbalize what he was consciously thinking about throughout the match. The recording was reviewed during the think-aloud to help the expert engage in information processing more similar to what occurs during performance.
Think-aloud Procedure:
Think-aloud Procedure:
Observe one expert player during a standard 2v2 player match of the videogame, Rocket League.
Record the game on video for later retrospective review by the expert. Take notes of major events, obvious rules and strategies used, but remain non-intrusive during the task.
Immediately after the game concludes, conduct a think-aloud with the SME.
a. Explain the think-aloud technique, emphasizing that the goal is to report internal thoughts throughout the game (i.e., verbalize the current contents of consciousness- what you are attending to).
b. Record audio of the expert think-aloud. Only step in when necessary to remind the expert to keep talking.
c. Identify general rules and strategies that were applied during the game from the think-aloud protocol.
Conduct follow-up semi-structured interview to address questions that arose during the think-aloud.
Semi-structured Interview:
Semi-structured Interview:
Why do you typically choose to hang out in the middle of the field?
It gives me an equal opportunity to either go in for the ball or to go back towards my goal and play defense if it looks like the other team is going for it. I can play offense or defense, depending on where the ball ends up. If it looks like the other team will have an opportunity to hit the ball towards our goal, I will start getting back.
About 45 seconds into the recording, you attempt to fly up into the air to play the ball. During the think-aloud you noted that was a mistake. Why was it a mistake and what led you to decide to go for the ball in that moment?
Because it was going to be close if both my opponent and I started flying for the ball at the same time. However, I decided to try anyway. He got there first.
When the face-off begins and you and your teammate are equally close, what information do you look for to determine if you should go for the ball or not?
The most obvious is the chat. If he doesn’t state that he is going for the ball in the chat, then I prepare to go. I creep forward like I’m going to go for the ball, while simultaneously looking to see if he is moving towards the ball. [follow-up: if you are both moving towards the ball, do you both of you continue for the faceoff?] I always back off if he looks like he is going for it. It never seems to happen the other way around.
You said that you’re always looking for boost when the time is right. How do you determine the right time?
In general, it is when I have no obligation to go for the ball and there is a boost available in an easily accessible area for me. For example, if my teammate is going for the ball, I know that I should not also go for the ball. That would be a good time to go for the boost. There are other times when I hit the ball up in the air or off the wall towards the center, and I know that I won’t be able to get to the ball and that the other players will be trying to go for the ball, so I go to take their corner boosts so that they can’t get it*.
*Note that boosts are used to accelerate the travel speed of the player’s car. Boosts have a reload period in which they are not available to be picked up again. Players have a limited amount of boost (i.e., 100 units) and must pick up more by driving over them on the arena floor. The six boosts along the boarder of the arena are higher value, meaning they fully reload boost.
5. Throughout the game you were often moving around in circles, never really sitting still. At certain points, you decided to move to a different area on the field. What information is helpful to you when deciding where to move next?
I’m often adjusting my positioning based on how I anticipate my opponent is going to hit the ball. Based on the way that they’re moving, their speed and their angle, and my experience of seeing the ball hit off of my car and other cars in previous games. Prior knowledge can help me anticipate where the ball is going next so that I can get there sooner.
6. Why do you decide to get on defense when you have no boost?
Typically when I have no boost that means my teammate (who theoretically does have boost) should go for the ball. I can’t do much to help if I am stuck in the middle of the field with no boost. It is typically the safest option to head back towards your net when you’re low on boost while your teammate plays offense and tries to keep the ball on your opponents’ side. You can replenish your boost safer on your side of the field just in case the ball comes down to your end. Boost is more likely to be available there because the opponent is usually not down there taking them.
7. You toggle your camera lock on and off throughout the game. When and why do you choose to do that?
I mostly do it when I’m going for boost to ensure that I actually collect the boost and don’t accidentally miss it. If you are moving fast and you miss the boost, it can throw off your rhythm and positioning and you’ll be stuck without boost. [follow-up: why is boost prioritized?] Boost is probably the most important part of the game because a large majority of the game is determining who can get to the ball first. If you don’t have boost you’ll be significantly slower getting to the ball. If you are trying to fly for the ball or attempt a save, the chances are much higher that you won’t be able to reach it. Pretty much all of my calculations of who is going to get to the ball first are based on the assumption that everyone else has full boost.
Analysis of the think-aloud protocol and post-task interview led to a deeper understanding of the game and the cognitive processes of experienced players. Results illuminated some general rules (e.g., if your teammate is not going for the ball, then you should go for it) and strategies (e.g., taking opponent team’s boost to slow them down when you are on their side of the field) that guide performance. Positioning and accurate go/no-go decisions are key to success in the game, which requires the player to have great situation awareness and quickly integrate information in working memory to select the best available action.
Things to remember:
Ideally, you would want to conduct this procedure several times, with multiple experts, to obtain data relating to a variety of game situations.
It is not always natural for a participant to talk about their their internal thoughts. It would be ideal to have them practice the think-aloud procedure with an unrelated task prior to beginning the real one.
A structured or semi-structured interview following the think-aloud protocol can be useful to follow-up with questions about initial observations.
Pros:
Easy to apply.
Rich information gathered from a single, or a few, individuals.
Allows researchers to uncover mental processes involved in a task, which are often not observable behaviors.
Cons:
Think-aloud is an introspective technique. Well-practiced actions and thoughts may be automatized, meaning the participant may not be conscious of what they are doing, which can make verbalization challenging.
Retrospective think-aloud is subject to limitations in participant memory, while concurrent reports are likely to interfere with the task. Supplementing reflections with review of video footage can be helpful for mitigating memory limitations.
References:
Shapiro, M. A. (1994). Think-aloud and thought-list procedures in investigating mental processes. In A. Lang (Ed.), Measuring psychological responses to media messages (pp. 1-14). Lawrence Erlbaum Associates.
Page updated
Report abuse