Mixing Methods :: Fishing for Information

A workshop at UX Week 2010 in San Francisco, August 2010. Organised by Elizabeth F. Churchill (Yahoo!) and Les Nelson (PARC). More information on UX Week here: http://uxweek.com/ and on the workshop here: http://uxweek.com/talks/14185

Workshop Description

All theories have self-prescribed limits. All models are incomplete. All methods of analysis accentuate one perspective at the expense of others. The real power of any method comes when it is used in conjunction with others. Mixing methods becomes especially important when trying to introduce technology into a real setting. Actual circumstance involving individual idiosyncrasies and collective behaviors brings more complexity than any one method can handle. Artful triangulation across multiple methods can give an efficient handle on that complexity.

In this workshop, we discuss how different user-centered methods can be mixed and merged to drive inspiration, innovation and validation of interactive experiences. We first cover a basic framework of socio-technical design, highlighting areas of active research in the human-computer interaction community.

We will present a number of case studies where we combined methods (ethnographic field work, prototype development and evaluation, focus groups, data mining and field experiments) to derive innovation possibilities and then how we drove the invention process to develop new products. From these case studies, we will illustrate an observation to innovation framework and show how to adapt established methods to address emergent questions. We encourage participants to come to the workshop with their own case studies and/or questions about methods they have used.

We address the following questions of interest to innovators, designers, engineers, marketers, and managers using practical examples:

  • What methods are available at the different stages in understanding  a new interactive experience?
  • Which of the many meanings of the word ‘prototype’ might work best in different circumstances, and at what cost?
  • How do you identify the blind spots in one method and recognize alternative ways to illuminate those gaps in understanding?
  • How do you undertake conversations across disciplines, each with their preconceptions, strengths, and limits?

Workshop Objectives

This is a workshop, not a tutorial. So, although we have plenty of material to share with you, please come prepared with stories, experiences and, if you like, artifacts for a show-and-tell. Our plan is to work together to create new understandings and to broaden and deepen our collective knowledge. Here are our objectives:
  • Share experiences of using different methods
  • Explore limits of existing methods and draw up ideas for methods, e.g., use of personal mobile devices, sensors, the Internet as laboratory, scaling, global reach
  • Share resources and build friendships

Workshop Outline

Below is our tentative outline for the workshop. However, given it is a workshop, this is more of a guideline than a plan - we will adapt as needed and desired by you, our attendees.

  • Introduction to workshop and overview - 30 minutes
    • What is innovation?
    • What role do formative and summative methods for understanding user experience play in driving innovation?
    • What methods are available at the different stages in understanding  a new interactive experience?
    • Which of the many meanings of the word ‘prototype’ might work best in different circumstances, and at what cost?
  • Round the room introductions - 20 minutes
    • Your name, your affiliation and job title, your background
  • Method Madness Part 1 - 10 minutes
BREAK - 30 minutes
  • Method Madness Part 2 - 30 minutes
  • Exercise in groups, draw on your own experiences - 45 minutes
    • Build a Questions and Method Scope Grid 
    • Consider questions that have gone unanswered - what new methods are needed?
  • Report back, synthesis, new method methods and open questions - 30 minutes
  • Close-out open questions and next steps


There are many excellent texts available that detail methods that are routinely used when designing and evaluating innovative technologies. Here are some of our favourites:
  • Observing the User Experience: A Practitioner’s Guide to User Research Mike Kuniavsky, Morgan Kaufmann, 2003 An excellent text book for practitioners and researchers alike.

  • Research Methods in Human Computer Interaction. Jonathan Lazar, Jinjuan Heidi Feng and Harry Hochheiser, Wiley, 2010.
  • Designing Social Interfaces: Principles, Best Practices and Patterns for Designing the Social Web Erin Malone and Christian Crumlish -  http://www.designingsocialinterfaces.com/ - make sure to check out the wiki

  • A Project Guide to UX Design: For user experience designers in the field or in the making. Russ Unger and Carolyn Chandler. Peachpit Press 2009
  • Sketching User Experiences. getting the design right and the right design. Bill Buxton, Morgan Kaufman, 2007
More general texts that detail methods which are not specifically about technology design:
  • An introduction to Qualitative Research. Uew Flick, Sage Publications, 1998.
  • Virtual Methods. Issues in Social Resaerch on the Internet, Edited by Christine Hine. Berg Press, 2005.
  • Mixed Methodology. Combining Qualitative and Quantitative Approaches. Abbas Tashakkori and Charles Teddlie. Applied Social Research Methods Series. Volume 46. Sage Publications, 1998.
  • Qualitative Interviewing. The Art of Hearing Data. Herbert J. Rubin and Irene S. Rubin. Sage Publications, 1995.
  • Tricks of the Trade. How to think about your research while you're doing it. Howard S. Becker. The University of Chicago Press, 1998.
  • Ethnographer's Toolkit. Edited by Jean J. Schensul and Margaret D. Lecompte. A 7-volume paperback set that details all aspects of ethnographic research.

  • Writing Ethnographic Fieldnotes. Robert M. Emerson, Rachel I. Fretz and Linda L. Shaw. University of Chicago Press, 1995.
  • Image Based Research. A Sourcebook for Qualitative Researchers. Edited by Jon Prosser Routledge, 1998.
  • An upcoming book that deals with upcoming methods and needs for new methods from a social theory point of view - Inventive Methods: The Happening of the Social Edited by Lury Celia, Nina Wakeford. Taylor and Francis, due out 2011

  • Programming Collective Intelligence. Toby Segaram. O'Reilly Press, 2007

Case Studies from our work

In the workshop we will summarise a couple of projects we have worked on together that led to prototypes and/or products. Here are the relevant references from those case studies - more details can be found on the Paper Abstracts & Videos page:
  • The foundation for our approach is described here: Design Through Matchmaking: Technology in Search of Users. Sara Bly and Elizabeth Churchill. In Interactions, pp. 23-31, March-April 1999. 

    This paper was a reflective piece on the work we and our colleagues were doing; we discussed how infrastructure and imagined-use are intertwined and meet in the design of interactions and interfaces and argued, therefore, that both should be considered when thinking about iterative improvement of an existing design or innovating to develop a new design. We used the term 'matchmaking' to suggest how reflective and nuanced this process can be. However, this paper was written a while back, focusing on work technologies and looking mostly at the instance of a technology already prototyped being iteratively designed or extended for use other than its initial conception. We have been updating these ideas to a more general arena, and thinking about the matchmaking process more generally as a way of thinking about we can think more broadly about technological and human capabilities and desires in concert.

  • These are the technology design papers from which our case study discussions for our presentation are drawn:
    • E.F. Churchill, J. Trevor, S. Bly, L. Nelson, and D. Cubranic. Anchored Conversations. Chatting in the Context of a Document. In CHI 2000 Conference Proceedings, ACM Press, pp. 454-461, 2000
    • E. F. Churchill, L. Nelson, and L. Denoue Multimedia Fliers: Informal Information Sharing With Digital Community Bulletin Boards In Proceedings of Communities and Technologies, Amsterdam, The Netherlands, September 2003, September 5, 2003.
    • E.F. Churchill, L. Nelson, L. Denoue and A. Girgensohn The Plasma Poster Network: Posting Multimedia Content in Public Places. In Proceedings of INTERACT 2003, Ninth IFIP TC13 International Conference on Human-Computer Interaction, Zürich, Switzerland, 1 September 2003.
    • L. Nelson, L. Denoue and E. Churchill AttrActive windows: active windows for pervasive computing applications. In Proceedings of the 2003 international conference on Intelligent User Interfaces January 2003
    • Yamada, T., Shingu, J., Churchill, E.F., Nelson, L., Helfman, J., and Murphy, P. Who Cares? Reflecting Who is Reading What on Distributed Community Bulletin Boards. Proceedings of the Association for Computing Machinery Symposium on User Interface Software and Technology, Santa Fe, New Mexico, 2004.
    • Elizabeth Churchill, Les Nelson, Laurent Denoue, Jonathan Helfman, Paul Murphy Sharing Multimedia Content with Interactive Displays: A Case Study Presented at DIS 2004, Cambridge, MA, August 1-4, 2004. YeTI, the technology described in this paper, won Best Utility and Honorable Mention in the Professional category in the DIS2004 International Design Award competition, held in Cambridge, MA, August 1-4, 2004.
    • Churchill, E.F.; Goodman, E.; OSullivan, J. MapChat: Conversing in Place, CHI 2008, 05/04/08, Florence, Italy, 2008

Web resources

There are literally thousands of excellent blogs and lists of methods that are used in the design of innovative technologies. Here are some we have referred to recently:


The gadget spec URL could not be found
Subpages (1): Paper Abstracts & Videos