3. Experimental Prototyping

The method used for exploration in Project Constructive is iterative development, evaluation, and assessment of experimental research prototypes. This method draws upon the assumption that in order gain understanding of game mechanics and their resulting dynamics when played it is in many cases necessary to build games (Mateas, 2005). This is increasingly important for achieving novel and innovative techniques, methods and approaches to game design. The method is described in Design for Results (2010) written by Elina Ollila and Mirjam P Eladhari, to appear in the journal Simulation and Gaming. An excerpt from the article is provided below. 

Mirjam gave a lecture on the topic as part of the course International Game Production Studies at Gotland University:



Below the Introduction section from Design for Results is provided.  An early version of the article, presented at Game Research Methods in Tampere 2010 is available here



Design for Results: Considerations for experimental prototyping and play testing using iterative game design


Mirjam P. Eladhari 

Gotland University, Sweden


Elina M.I. Ollila

Nokia Research Center, Finland


Introduction 

This article concerns development of experimental research prototypes, sometimes also called demonstrators or sketches. A commonly used development method is iterative design which, as described by Salen and Zimmerman (2001), is a play-based design process. A prototype is “played, evaluated, adjusted and played again, allowing the designer or design team to base decisions on the successive iterations or versions of the game. Iterative design is a cyclic process that alternates between prototyping, play-testing, evaluation, and refinement.”

In game research, methods from social sciences, humanities and technological fields are used commonly. In social sciences, the effects of games on people are studied, using the methodological body of the field. Common questions concern learning processes in games. Humanities oriented researchers commonly study the meaning and context of games, studying the games as artefacts, using the plethora of methods, such as critical analysis, of the field of humanities. In the fields of engineering and industry approaches often concern the understanding of design and developing of games. The focus is typically on how to make better games, other times games are used as drivers of technological innovations.

It is vital for the future of the field of game research that methods from other fields of study are adapted to the nature and essence of games, and that new methods are developed for this research field. It is important to discuss the methodological challenges in the field of research, where it is not uncommon that success criteria are imported from other fields, sometimes not considering the profound aesthetic nature of games.

Because of the cross-disciplinary nature of game research it is uncommon to use methods from only one approach. However, methods from different disciplines are often used at certain stages, and for different ends, in research projects. For example, in Eladhari's doctoral thesis (2009) the first chapters lays the grounds for the `why´ of the design work conducted from a humanities perspective, reasoning about the genre (virtual game worlds) approached. The middle chapters describe the 'how'; the prototypes built, using an engineering perspective in describing the technological implementation, and a cultural productions perspective in describing the game design. The final chapters concern the results of the research, describing the evaluations of the prototypes which are done using methods commonly used in the social sciences.

Our focus in this article is on how to in general, through the work process, plan for obtaining data from play tests that can yield answers to specific research questions. We provide a number of considerations that can be useful for researchers and prototype developers to take into account. The considerations and methods are based on our experiences with research projects, with a few exceptions where we have reviewed methods used by other researchers and added those into our “toolbox”. We also point out interesting directions for future research in this field.

In the beginning of the process of developing research prototypes, researchers take the initial stance in the research question they set out to explore. This governs the type of game and game prototype that can be useful to create, how it can be tested, and how significant data can be obtained in the process.

Early in the process the researcher’s questions often include the following questions:

• What types of game play dynamics and game play experiences can a certain, mechanic, feature, approach or method result in?[1]

 

• What are the qualities of the play experience, and how can these qualities be documented, analysed, interpreted and explained?

 

• How can it be decided if a certain game play feature results in something valuable, such as a new type of experience, a meaningful experience, or a ‘better experience’ in some other way, and if so, compared to what? Researchers may find themselves in situations where innovative results do not necessarily lend themselves to comparisons.

 

• What type of data can be useful to obtain in order to explore the research question?

 

We do not, in this article, focus on the design process of what makes a good game, a process described by among others (Fullerton, Swain, & Hoffman, 2004) and (Brathwaite & Schreiber, 2008). Researchers need to, in the design process, take into account what questions they aim to explore and stay focussed on these throughout the design and implementation work. It is, speaking from our own experience, easy to fall into a frame of mind where one aims to produce a good game, losing focus of obtaining research material. Ollila (2009) shows that excessive attempts on innovation[2] within a single game concept while investigating specific research questions can spoil the results. The very reason for the development of research prototypes is to find methods, features or approaches that can be used in other games, games which are specifically made to be fun[3], challenging and perhaps carry a message. The researcher needs to approach the design of the prototype both as a researcher and as a game designer.


Game design has been identified as a wicked problem space (Mateas & Stern, 2005). The phrase ‘wicked problem’ is used in social planning to describe problems where every attempt at producing a solution changes the understanding of the problems (Rittel & Webber, 1973). Mateas and Stern (2005) argued that even although studying existing games can lead to deeper understanding it is essential to also build them:

 

[...] if game studies is limited to analysing existing games and design spaces, it can be problematic to imagine or theorise about potential game features outside of these design spaces. Models about the nature of games and their features run the risk of being incomplete or wrong, simply because certain design spaces have not yet been explored.


For researchers and designers alike prototypes constitute important thinking-tools. The motivation for writing this article arose when the authors recognised a common need for a navigational aid in the wicked problem-space of game design, to use when planning for what type of prototype to build and how to test it, given the research question(s) and the resources (time, staff, and technology) at hand. Much of the available literature focuses on how to produce good playable games, but for the game researcher working with experimental game prototyping, the success criteria are different. It is vital to use the often scarce resources in the most efficient way to produce reliable communicable results that can be useful to others in related fields. Since the scope of the myriad of different research questions can be varied, along with ways to validate data, we here focus on the middle-stages; the prototyping, the playtesting and ways to obtain interpretable data from the tests. It is our hope that our contribution can be of use to others working with experimental game research prototypes.


Both testing methods and ways to analyse obtained data are in the field of game research prototypes heavily influenced by methods in the HCI field. An open question is whether the prototypes and the players’ experience of them can be analysed using HCI methods ‘off the shelf ’ or if something else entirely is required? For instance, Ollila (2009) shows that HCI methods can be applied, however, they need to be adapted to the field of game research.

Usability methods have been traditionally used to assess the usability factors, such as learnability and memorability, of user interfaces. More recently, focus has shifted to understanding User eXperience (UX) more broadly, including the emotional and social side, and the context of the user (see, e.g., (Roto, 2006)). User experience can be defined as “A person’s perceptions and responses that result from the use and/or anticipated use of a product, system or service” (UXIsoStandard, 2009). UX is a subjective experience of using or thinking of using a product or service in a certain kind of context, and it can be examined in various granularity levels. User experience consists of both functional and emotional sides, with more focus on the emotional side, and it changes dynamically over the time (Roto, 2006).

Game evaluations can be seen as consisting of three main areas: quality assurance, which typically focuses on the quality of the game software and seeing that all the functionality is in place; game usability testing, and playability testing which focuses on the game play. Quite often these activities focus on the game itself more than players’ context or culture and tests are conducted in laboratory settings (Ollila, 2009; Pagulayan, Keeker, Fuller, Wixton, & Romero, 2008). Players’ emotions are often assessed in interviews or analysing players’ voices, thinking-aloud-while-playing, body language, or facial expressions (Lazarro, 2008). Some researchers have studied player's reactions to games by recording their physiological signals during the play (Rajava, Saari, Laarni, Kallinen, & Salminen, 2005). In field (or beta) testing, players play the game in their natural settings, however, the data is usually gathered in interviews or via play diaries (Ollila, 2009), with the focus being more on the game itself than practices, or culture.

Many recent games could be better described as services than products. Many of these games are released at social platforms such as Facebook, or distributed as applications for mobile devices such as smart phones. When these kinds of games are released, they do not have necessarily a full feature set ready, and will evolve by time. In this case, prototyping is important as well – to create a game that is good enough for the players to get started with, however, after releasing the game, new features can be added and prototyped. This type of production allows developers to test their features on their actual target group of players in an earlier stage than what has been customary. In this article, we look at games in general, not particularly games as services, but acknowledge that this will be an important area of research for the future.

This article is organized in the following way. First, different methods of prototyping and types of prototypes are described, along with considerations regarding type of game that is developed. Then, various types of play-tests are described along with recommendations of which types to use depending on timing within the production cycle and availability of test-players. Also, an overview of potential methods of obtaining data from play-tests is provided.


Endnotes

[1] As described by Brathwaite (2008), a core mechanic (such as flipping over tiles or selling items to another player) of a game results in a core dynamic when it is played.
[2] Particularly innovation with creating a large set of completely new kind of, potentially engaging, game features that could make the game more fun to play, which may make it difficult to interpret the research data
[3] However, we do not suggest here that research prototypes should not be fun to play, which actually be another pit fall.

References

Brathwaite, B., & Schreiber, I. (2008). Challenges for Game Designers (1st ed.). Charles River Media.

Eladhari, M. P. (2010, September). Characterising Action Potential in Virtual Game Worlds applied with the Mind Module. Retrieved from http://urn.kb.se/resolve?urn=urn:nbn:se:hgo:diva-368.

Fullerton, T., Swain, C., & Hoffman, S. (2004). Game Design Workshop: Designing, Prototyping, and Playtesting Games. {CMP Books}. Retrieved from http://www.amazon.ca/exec/obidos/redirect?tag=citeulike09-20&path=ASIN/1578202221.

Lazarro, N. (2008). Why we pay: Affect and the Fun of Games: Designing Emotions for Games, Entertainment Interfaces and Interactive Products. In A. Sears & J. A. Jacko (Eds.), . Lawrence Erlbaum Associates.

Mateas, M., & Stern, A. (2005). Build It to Understand It: Ludology Meets Narratology in Game Design Space. Retrieved from http://users.soe.ucsc.edu/~michaelm/publications/mateas-worlds-in-play-2007.pdf.

Ollila, E. M. I. (2009, March). Using Prototyping and Evaluation Methods in Iterative Design of Innovative Mobile Games.

Rajava, N., Saari, T., Laarni, J., Kallinen, K., & Salminen, M. (2005). The Psychophysiology of Video Gaming: Phasic Emotional Responses to Game Events. In D. C. Suzanne & J. Jennifer (Eds.), . University of Vancouver. Retrieved from http://www.digra.org:8080/Plone/dl/display_html?chid=06278.36196.pdf.

Rittel, H., & Webber, M. (1973). Dilemmas in a General Theory of Planning (pp. 155-159). Elsevier Scientific Publishing. Retrieved from http://www.uctc.net/mwebber/Rittel+Webber+Dilemmas+General_Theory_of_Planning.pdf.

Roto, V. (2006). Web browsing on mobile phones-characteristics of user experience. Helsinki University of Technology.

Salen, K., & Zimmerman, E. (2001). Rules of Play: Game Design Fundamentals. MIT Press.

Standard. (2009). ISO FDIS 9241-210..