Saturday 6 July 2024, 2:00 pm to 5:30 pm
Meagan Flus (University of Toronto) | meagan.flus@mail.utoronto.ca
Sharon Ferguson (University of Toronto) | sharon.ferguson@mail.utoronto.ca
Yuval Kahlon (Tokyo Institute of Technology) yuvkah@gmail.com
Gregory Litster (University of Toronto) | greg.litster@mail.utoronto.ca
Christopher McComb (Carnegie Mellon University) ccm@cmu.edu
Alison Olechowski (University of Toronto) | olechowski@mie.utoronto.ca
Team dynamics, and the study of teamwork in design teams, is complex. As such, researchers have employed a variety of methods to understand how design unfolds in team settings, such as interviews [1], protocol analysis [2], and analyzing digital trace data [3]. While each of these method and subsequent data analysis has strengths, they also have limitations. For example, protocol analysis and other qualitative methods are subject to coding bias [4], whereas methods such as Natural Language Processing may lose the richness of a dataset. We therefore argue that a triangulation of analytical methods may be necessary to understand the complex nature of teamwork in design. A new challenge arises, however, in acquiring the resources to conduct such comprehensive and diverse data analysis. Open access design team data permits a collaborative approach to data analysis – researchers can investigate complementary research questions using a large
variety of methods on the same dataset, uncovering findings and gaining insights not possible by a single researcher.
The free availability and distribution of research outputs is quickly becoming a norm in many fields; for example, access to tools like ChatGPT by OpenAI have revolutionized how we use artificial intelligence globally. The design community has begun recognizing the
demand for publicly available datasets. Perhaps most notably, the Design Thinking Research Symposium (DTRS) series hosted symposia in 1994, 2004, and 2014 centered around the concept of research with common datasets and analysis [5]. The shared
context permitted an open discussion on methods and findings, allowing for comparisons on the same data. The popularization of open access tools and the focus on shared datasets of past design workshops have encouraged more discussion on open access design data, as evidenced by an upcoming ASME Journal of Mechanical Design special issue on cultivating datasets in engineering design [6]. However, the future of open access design data remains elusive. In light of the popularization of open access data, we argue that a revisited conversation on use of open-source design team data is necessary.
Provided the success of past implementations of shared datasets and analyses, yet noting the significant time gap since a workshop of this nature has been hosted, we propose a workshop to simulate the open access model with the specific goal of improving design team research. Attendees will apply different data analysis methods on the same design team dataset, provided by the organizers, with the objective of exploring best practices of open access design team data. We hope to start a conversation about what open-access data looks like in the design community and how this might help to uncover the complexities associated with collaborative designing.
The workshop will involve a short initial presentation followed by time dedicated to small group activities and finally, a large group discussion.
The workshop will begin with a brief overview of activities, followed by introductions from all attendees, sharing their experiences and interests as they relate to the study of design teams. The organizers will then give a short presentation, detailing the current state of open-source design team research and reflective prompts on how open-source design team data could impact our field. We will then discuss design research analytical methods, specifically those suited for studying design teams. Finally, we will present attendees with a sample data set – transcripts from a design team collaborating on a design project – and provide an overview of the contents.
Once attendees are familiar with the dataset, they will split into small groups. Depending on the size of the workshop, we will either have two groups or four, with half of the participants applying a qualitative method for data analysis (e.g., thematic analysis) and the other half will use natural language processing on a pre-formatted dataset (e.g., topic modeling). We will survey attendees before the workshop on the skills that they are familiar with in order to place them in groups where some have used a method before, while others can learn a new method in a low-stakes setting. Each group will determine a research question suitable for their method. They will then conduct analysis to whatever extent possible in the timeframe of the workshop. Finally, the groups will give a short “lightning talk” of their process, findings, and insights to the larger group.
After a break, during which the organizers will prepare a summary of the small group findings, all attendees will gather for a full group discussion. The main objective of this discussion will be to explore how different methods of analysis help researchers gain unique insights on design teams. We will explore the benefits and drawbacks of opensource design team data, then prepare a set of best practices and next steps.
In line with the topic of open access data, we wish to make the insights from this workshop available to those not in attendance. Post-workshop, the organizers will prepare a report to be disseminated to the wider design community (e.g., a Design Research Note on open access design team data) and a broader audience (e.g., a Medium article or a podcast).
• Introduction (10 mins)
• Background presentation (15 mins)
• Small group data analysis (90 mins)
• Break (15 mins)
• Group discussion and wrap-up (90 mins)
The workshop will be open to all researchers, practitioners, and students who are interested in studying cooperative design and learning about open-source data.
Before attending the workshop, participants will be asked to complete a Microsoft Form where we will collect information on their objective(s) for attending the workshop, data collection method(s) and analysis previously used, and any other information that will be insightful in planning details for the workshop (e.g., methods to discuss during the workshop). Since a significant portion of the workshop will be dedicated to applying analytical methods in teams of experts and novices, we will give participants an opportunity to either share their experiences or learn a new method. For this reason, the workshop will be open to participants with varying levels of expertise.
[1] M. Flus and A. Hurst, “Experiences of Design at Hackathons: Initial Findings from an Interview Study,” Proc. Des. Soc., vol. 1, pp. 1461–1470, Aug. 2021, doi: 10.1017/pds.2021.407.
[2] G. Litster, A. Hurst, and C. Cardoso, “A Systems Thinking Inspired Approach to Understanding Design Activity,” in Design Computing and Cognition’22, J. S. Gero, Ed., Cham: Springer International Publishing, 2023, pp. 161–177. doi: 10.1007/978-3-031-20418-0_11.
[3] S. A. Ferguson, K. Cheng, L. Adolphe, G. Van De Zande, D. Wallace, and A. Olechowski, “Communication patterns in engineering enterprise social networks: an exploratory analysis using short text topic modelling,” Des. Sci., vol. 8, p. e18, 2022, doi: 10.1017/dsj.2022.12.
[4] E. Babbie and L. W. Roberts, Fundamentals of Social Research, 4th ed. Nelson Education Ltd., 2018.
[5] N. Cross, “A brief history of the Design Thinking Research Symposium series,” Design Studies, vol. 57, pp. 160–164, Jul. 2018, doi: 10.1016/j.destud.2018.03.007.
[6] “CALL FOR PAPERS: Design by Data: Cultivating Datasets for Engineering Design - ASME JMD.” Accessed: Mar. 04, 2024. [Online]. Available:
https://asmejmd.org/2024/02/14/design-by-data-cultivating-datasets-for-engineeringdesign/, https://asmejmd.org/2024/02/14/design-by-data-cultivating-datasets-forengineering-design/