Abstract Ben van Buren and Colleen Macklin

Knowledge structures our perception of virtual worlds

“We can imagine that this complicated array of moving things which constitutes ‘the

world’ is something like a great chess game being played by the gods, and we are

observers of the game. We do not know what the rules of the game are; all we are

allowed to do is to watch the playing.”

– Richard Feynman

When we play a digital game, we rely on our perception in order to apprehend

hidden rules embedded in the dynamics of the game world. We are an experimental

psychologist and a game designer, who both construct virtual worlds (of different sorts,

and for different purposes) in our research and design practices. We co-teach a

research seminar on perception and game design, and we find the perception of virtual

worlds to be a venue for enriching and challenging exchange between psychologists

and game designers. Here we focus on an important theme that recurs in both of these

fields — that our perception of a ‘virtual’ object or system is always shaped by the

background knowledge of the perceiver.

Perception is a constructive process: what we see is determined not just by the

sensory data that arrive at the retina, but also by endogenous factors — often described

as biases, heuristics, or interpretations. These structuring factors are not random, but

rather reflect our previous encounters with the world. Thus, our perception of what is in

front of us is not fully real, in the sense that the mind plays a constructive role. But

neither is it fully virtual, because — even when we view an abstract work of art, or play a

fantastical videogame — the way in which that experience is structured reflects our

historical embedding in a complex and multifaceted real environment. Perception, then,

can be thought of as “half-real” (to borrow Jesper Juul’s term), an amalgam of the

“rules” of physics and sensory data with the constructed “metagame” (in reference to

Boluk and LeMieux, 2017) of the mind based on experience.

As introduced above, we both design virtual worlds: BvB designs behavioral

experiments to study the mechanisms of human perception, and CM designs games to

represent experiences, ideas, and emotional states. BvB’s lab has recently

demonstrated that a simple rotating stimulus cues spatial attention in a way that reflects

implicit knowledge of two physical principles: gravity and friction. CM has recently

developed a simple cloudspotting game to explore the perception of meaning and

pattern in the interplay between human players and machine-learning generated

content. Both projects mine the “half-real” nature of perception as both virtual and real:

in BvB’s research, implicit knowledge of physics is embedded in visual responses to

abstract stimuli; CM’s project explores the mind’s virtual patterning (i.e. apophenia) of

random data. Together, these projects illustrate how perception of virtual worlds always

bears the imprint of the observer’s surrounding reality.