This paper documents my reflections on the creation and navigation of three “data spatializations” developed as experiments in sensory embodiment/articulation of quantitative data. These explorations were spurred in part by Sarah Pink’s (2015) descriptions of Oswaldo Maciá’s installation work and Janet Cardiff’s sound walk pieces as examples of sensory articulations of research data. This prompting was further shaped by the present pandemic precluding the construction of physical, public installation work, as well as by my present research into the materialties of digital places, and some prior explorations using digital fabrication tools to generate “data materializations” (physical objects which embody different quantitative datasets). In the remainder of this introduction, I will lay out some theoretical considerations for “data spatialization” by first discussing some extant design theory on data materialization/physicalization, and then drawing connections to new-materialist and digital-materialist conceptions of materiality and sense-making with technology and art. This framework will then inform reflection on the three data spatializations themselves, first examining my experience of creating these digital artifacts/places through a sensory/material lens, and then examining people’s experiences in/with them.

In their discussion of their design-based data materialization work, Courtney Starrett, Susan Reiser and Tom Pacio (2018) discussed how data materialization prompts sensory experiences that data visualizations typically do not. Beyond the obvious modal conclusions to be drawn about which of the five senses in the Western sensorium are addressed by a visual diagram versus touchable object, Starrett et al. described how their materialized data objects, when exhibited, invited haptic participation from users. They noted that “[a] surprisingly high number of guests touched or picked up the silicone pieces off the table” (p. 382), and moving forward, Starrett et al. developed their projects with the understanding that these material objects don’t just communicate information or offer sensory experiences, but prompt sensory activity. Data spatializations may in turn prompt a greater (or at least different) variety of sensory activities by inviting locomotor and auditory action as well as visual and haptic. Starrett et al. differentiated their work from traditional data visualization (and “data physicalization” discussed by Jansen et al., 2015), by explicitly stating that they were less preoccupied with clearly communicating the source data and more focused on creating artifacts which may generate aesthetic and material interest. While my thinking-through-making of my data spatializations didn’t simply regard the source data as a means for getting at an aesthetically interesting form, Starrett et al.’s non-communicative focus does open up a space for other kinds of relationships, apart from transmission, that data artifacts can have with their source data.

In discussing the potentials of what they called “data physicalization,” Jansen et al. (2015) hewed closer to a conception of data artifact as a transmitter of information. They defined data physicalizations as “physical artifact[s] whose geometry or material properties encode data,” (p. 3), which can then be presumably decoded from the artifact. Nonetheless, they still highlighted the potential for data physicalizations to prompt novel sensory relationships with data. They described how data physicalizations invite “active perception” (p. 4) via motor engagement – e.g. exploring a small artifact by rotating it in the hand, or exploring a large artifact by walking through it. As with Starrett et al. (2018) above, artifacts (including places) that embody data are framed as prompting sensory activity rather than eliciting a passive sensory experience. Jansen et al. (2015) also described how, by engaging not just multiple modalities, but “intermodal” perceptions that integrate multiple traditionally delineated sensory modes, “[p]hysicalizations can take advantage of these additional sensory channels to convey a larger range of meanings than a simple visual display” (p. 4). While I’m less concerned about conveying meaning with my data spatializations, Jansen et al.’s observation points to how data materializations/spatializations may, by offering multi- and inter-modal sensory stimuli, elicit varied sensory experiences with/through/from data that may in turn become meaningful in different ways to different experiencers. Jansen et al. seemed to explicitly acknowledge this power beyond meaning when they observed that “[a]part from their perceptual and cognitive benefits, physicalizations allow individuals and groups to relate to data in new ways” (p. 4, emphasis added). Manning (2008) has postulated how thoughts are rooted in bodily feelings, actions, and relations, elicited through sensitive attention, and “transduce[d]” (p. 8) into complex connecting and transforming concepts through further feeling. Lakoff and Johnson (2003) argued that the metaphors that fundamentally structure thought are rooted in pre-linguistic analogies to bodily experiences. If thoughts are indeed rooted in bodily feelings, actions, and relations, then using a virtual space to juxtapose bodies and data in new and sometimes-impossible ways may yield new metaphors or other new structures of thought.

Manning’s (2008) “Creative Propositions for Thought in Motion,” referred to above, may help elaborate on Starrett el al.’s (2018) and Jansen et al.’s (2015) observations on the sensory potential of data materialization/physicalization. In that piece, Manning discussed artwork and the practices and forces that go into making it, meaning that to apply her thought here we need to recognize data artifacts as artworks. Manning wrote that “[o]ne work can have many dynamic forms, many concepts, many feelings or thoughts. There is no single point of identity for a work” (p. 9). Reflecting both Jansen et al.’s (2015) and Starrett at al.’s (2018) acknowledgments that making artifacts from data invites more than the simple transmission of that data, Manning speaks to how an artwork opens up space for myriad embodied experiences, potentially yielding the new relations to data Jansen et al. (2015) proposed. A seemingly coherent artifact is shaped by, contains, and expresses countless forces, its “oneness” coexisting with its “infinite multiplicity” (p. 14). However, Manning asserted that there is “[n]o relativism here” (p. 14)! The data spatialization-as-artifact’s opening up the range of potential relations with/to data is not simply a post-modern affirmation of multiple coherent human subjects operating from different positions (though critical readings that acknowledge positionality can have valuable political and material outcomes). Rather, what is acknowledged here is the way an artwork may invite participation between the complex forces that comprise it with the complex forces that comprise the viewer (and comprise the setting and cetera). This can be contrasted with other forms data may take, such as a chart or an academic write-up, which while also formed by myriad forces, typically foreclose play within them. Virtual places, such as the digital data spatializations I developed, may specifically facilitate Manning’s (2008) propositions to “[f]orget what you feel” and to “feel again” (p. 19) by allowing for physically impossible material spaces that may extend or defy previous embodied experiences. For instance, my data spatialization depicting CO2 emissions (see Figure 1) includes a cube that is volumetrically equivalent to the CO2 emitted by the United States in the year 2015. Each edge of the cube is 13,722.8m, making it an impossibly large form (Mt. Everest is 8,848m high, as a point of comparison). Participants in this spatialization will consequently experience something that may extend or analogize to prior experiences, but which strictly-speaking, they could not have experienced before, potentially eliciting new sensory experiences which may elicit new relations, or new pre-articulate thoughts, with respect to the data embodied by the shape.

Something that distinguishes data spatializations from the above-discussed data fabrications/physicalizations is that the sensory acts they elicit involve movement through a place. Jansen et al.’s (2015) discussion of the “active perceptions” activated by data physicalizations specifically noted that different scales of physicalization invite different sensory activities – e.g. rotating in the palm, versus walking through and around a place or artifact. In Walking Methodologies in a More-Than-Human World, Springgay and Truman (2017) wrote about the potentials of attending to affect when walking through a place. Their framing of affect comes not so much from a phenomenological standpoint as one that emphasizes the entanglement of myriad factors that comprise the place, the walker, and the their interaction. The “more-than-human world” noted in the title of their text acknowledges the agency of places, a “thing power” (Bennett, 2010, p. 6) that can also be extended to the materiality of digital places, the exertions they make, and the intra-actions they participate in. In their discussion of the impacts places can have on the bodies moving through them, Springgay and Truman (2017) address both “constructed” urban spaces and “uncultivated” or “natural” spaces, recognizing the problems entailed in the distinctions drawn between the “natural” and “un-natural,” and acknowledging the complexity of human and non-human agencies that construct these places. This again invites consideration of the embodied experience of walking through digital places, despite their undeniable-on-the-surface human-made-ness. The tools used to make these digital places, the hardware used to run them, the environments in which they are run, and the embodied sensory histories of both designer and walker all play a part in the experience of these “human-made” digital places. As Manning (2008) noted of the experience of the creation of human-made artifacts: “What we feel is the violence of the forces’ struggle for valuation within the work” (p. 12). The creator is less a producer of objects than a co-producer or wrangler of forces. I turn now to a reflection on the creation of these data spatializations.

Building the data spatializations entailed the use of a variety of digital materials, each possessing their own material qualities in the forms of invitations and inhibitions inscribed in them as virtual objects (Verbeek, 2006). These included a variety of digital file types, imagery, pictorial and textual data sources, and software tools. Unity, the software used to collect together and render participable the elements of each digital place, belongs itself to a class of software that self-identifies as a place: an integrated development environment, or IDE. This is a tool I have experimented with in the past, but largely in my capacity of developing curriculum to help students create virtual worlds using it, not for creating my own work. Creating these data spatializations has given me an opportunity to reflect on the sensory entailments of the tools/environments I ask my students to engage with, and the way(s) they exert force on creator and creation.

Madeleine Akrich (1992) and Bruno Latour (1992) advanced the concept of “scripts” by which technological material artifacts exert agency without activity. Peter-Paul Verbeek (2003) elaborated on this concept by highlighting the “invitations” and “inhibitions” that digital materials and artifacts contain in them which act on other entities (including human actors), influencing both action and experience (invitations and inhibitions also seem like an extension of Don Norman’s (1990) design concepts of “affordances” and “constraints”). In the course of creating these data fabrications, themselves digital material artifacts with their own agency, I often had to respond to, accede to, and abrade against the material agencies of the digital stuff I was working with.

Barad specifically articulated apparatuses as being “dynamic (re)configurings of the world, specific agential practices/intra-actions/performances” (p. 816) that (co-)produce phenomena. As a tool, an interface, and an environment, Unity unquestionably exerted agency over my own creative action, and prompted reflection on the agency it has exerted in the work of my students who have engaged with it. While it is a flexible and extensible system, Unity comes with a number of default material, lighting, and virtual camera settings that lend themselves to particular sensory qualities of its simulated surfaces. Unity Technologies also offers a free “standard assets pack” designed to help users quickly and efficiently put together basic projects without reinventing all of the attendant systems from scratch. This pack necessarily has design norms baked-in that reflect embodied and sensory assumptions. The included animations for the “standard” character, for example, will only work with humanoid models that have two arms and two legs, reflecting assumptions (assumptions that I would call ableist, looking through a critical lens) about what comprises a “standard” sensing human body. Unity’s units of measurement for objects in its systems, despite using the innocuous term “units,” in fact are equivalent to meters in the physical world, when framed by the standard physics and motion systems in the game. It was necessary to acknowledge this when developing the data spatializations for CO2 emissions and food deserts, both of which depend on evoking distances and scales from the physical world to derive potentially-sensorially-activating forms from the data. Within Unity’s editor, every object is pinned to a specific Cartesian x, y, and z coordinate, meaning that moving through a world made in Unity is invariably undergirded by a Cartesian foundationalist epistemology where one has a subjective, grounded perspective within an objective, concrete world (Phillips & Burbules, 2000). Ambiguity, uncertainty, or contingency in the structure of the world would need to be explicitly constructed or simulated within (or against) Unity’s systems.

One of the most impactful inscribed traits of Unity’s standard assets pack is that it implements a conventional video game interface for moving a body through virtual space, using either the W, A, S, and D keys in concert with the mouse, or using a handheld video game controller. There has been an increasing tendency in interaction design (self-identifing more regularly as “experience design”) to treat interfaces as transparent mediums through which our sensoria are unproblematically extended into virtual places (Bolter & Gromala, 2006; Lialina, 2012; Lialina, 2018). The receding of the interface as a salient agent in intra-actions involving humans and technology reduces both our critical awareness of and sensory sensitivity to the invitations and inhibitions inscribed in them. The interface is not simply a medium between two coherent things (the digital and the physical), but a co-constitutive agent participating in phenomena with other agents (broadly, the computer and the human user) within a larger system in which they all act. The interface is itself an apparatus, in Barad’s (2003) sense of the term, defined above. The default control interface embedded in Unity reflects a functional, tested solution for moving a virtual body through virtual space. However, it also inexorably places produced works which use that system within the idiom of a certain genre of “video games,” which people have different embodied histories with. These may be gender or age-related feelings of exclusion or alienation from this cultural form, stemming in part from commercial games being marketed for decades as toys for adolescent boys (Lien, 2013). They may also stem from lack of access to the specialized hardware for playing games, or any number of other histories that make the embodied knowledges associated with “video games” a rarity among, and fequently an obstacle to, most people. These design assumptions create a sensorimotor hierarchy: For users bodily familiar with “WASD” keyboard controls, or with a game controllers, the interface becomes a transparent extension into a sensory space, whereas for users with different embodied histories, the interface is more salient as a (possibly adversarial) agent participating in the intra-action between user, digital place, and interface. Different people’s different embodied histories with technology and digital culture will shape very different haptic experiences, that will impact their complex, inter-related sensory experiences of these virtual places. This relates, to a degree, to Starrett et al.’s (2018) discussion of the importance of the historical context of materials (in Starrett et al.'s case, the history of sterling silver dinnerware) when developing data-objects. The semantic and embodied connections people make to the digital or physical material with which an object is made can strongly affect the sensory experience of that object. With that in mind, I would like to briefly turn attention to sensory experiences within the data spatializations.

As of this writing, I have not yet had any responses to the informal survey I shared in a few online social spaces to solicit accounts of experience with the data spatializations. I will here briefly share my own accounts, as well as some thoughts on the role of digital materialities in the lack of participants.

My own experiences within the data spatializations are of course informed by my having made them and consequently having familiarity with not only the data sources informing them, but the myriad other agentic human and inhuman forces that informed their creation. However, buidilng a place and experiencing a place are not congruent sensory actions, and the experiences of moving through these spaces yielded some unexpected outcomes. In the food desert spatialization, walking the distance to the nearest grocer felt, as expected, interminable. However, an unexpected felt tension arose from the functional choice to use a Google Maps-style map to situate the body and the various food stores in the Richmond neighborhood depicted. Walking across the surface of a 1-to-1 scale map, like the perverse and useless map developed by the Cartographer’s Guild in Jorge Luis Borges’s short story “On Exactitude in Science,” seemed to exacerbate the sense of distance for me. Typically a Google Map is navigated quickly via mouse or touchscreen, with miles traversed in seconds. Having to walk across the same abstraction in real-time elicited for me a meaningful felt connection (or perhaps a disjunction) between the original map I had used to source the data and the material realities it had been trying to convey. Lev Manovich (2008) claimed that “data visualization moves from the concrete to the abstract and then again to the concrete” (p. 6). My experience walking through this particular virtual space reflected how data spatialization may push further this movement between abstraction and experience.

I am curious to elicit others’ accounts of sensory experience and action within these places, and will likely continue sharing the survey form past the writing of this piece. Toward the end of my last meeting with the sensory ethnography group in which I developed the data spatializations, a brief exchange occurred which surfaced some of the sensory-material realities of contemporary digital systems which may have contributed to the lack of participation. As noted above, there is an increasing tendency among interaction designers toward interactions with digital systems which are transparent, seamless, and elide the presence of interfaces, and also often the systems being interfaced with (Bolter & Gromala, 2006; Lialina, 2012; Lialina, 2018). One consequence is that the file systems of contemporary computers are no longer foregrounded, and the historic skeuomorphic visual metaphor of nested file folders is less often encountered, especially on mobile devices. In the past, downloading, selecting a folder, saving a file, and then opening or running it were part of the choreography[3] of computing, something that happened even when doing something as prosaic as opening an email attachment. These movements are less a part of contemporary computing choreographies, which are shaped by the invitations and inhibitions of contemporary design tendencies toward seamless, transparent interfaces. One outcome of this shift was foregrounded when one of our group members tried in earnest to download and run the spatialization, and encountered some unfamiliar technical hurdles. Users can tend to feel disempowered, and engage in self-blame when seamless designed systems suddenly become seam­ful (Norman, 1990). The self-blame can take the form of declaring one’s self “not a computer person,” situating one’s self within a sensory hierarchy wherein “non-computer people” are expected to have one kind of haptic experience with digital systems, and feel unwelcome or inadequate when presented with other experiences that are the exclusive territory of “the computer person.” This is a rhetorical gesture very similar to the common one encountered by art teachers whose students situate themselves as “not an art person” to foreclose feelings of inadequacy stemming from semantic and embodied associations with artmaking. In both cases, the person sources the lack or problem in themselves (they do not belong to the species of person with the innate knowledges needed for this sensory experience). However, they are not in a state of lack, but rather being acted on by variety of (f)actors that are disincentivizing uncomfortable material experimentation, play, and risk-taking.

Perhaps these data spatializations point to a larger question regarding sensory encounters with digital material: How might art educators foster more equitable digital choreographies, improvisational dances where the designer is less a prescriptive choreographer than a partner, co-producing the feeling-action of computing with the student/user as agent? By encouraging students to see themselves as extemporaneous and agentic participants in the computer’s affect-eliciting systems? By helping students develop creative skills and critical frames to develop their own digital systems that invite sensitive and engaged participation? These concerns lie beyond the scope of this reflection, but may scaffold future explorations through the sensory-material practices of digital making.

[3] Choreography is here used in the sense that Manning (2013) used it, to describe internalized, recurring, often but not always habitual, embodied performances. Choreography is cued by human and non-human actors in each other. As with varieties of literal choreographies, these cues may be prescriptive and routine, foreclosing present experience with the unfolding circumstance (Manning, 2019), or may be invitations to extemporaneous participation in the present.