Mixed Methods Data Collection & Synthesis

Collecting and interpreting qualitative and quantitative datasets collected during the feedback process.

PURPOSE:

The idea is to use both qualitative and quantitative forms of data collection to better engage stakeholders that would otherwise have difficulty participating or feel pressured away from expressing their true feelings about design vision.

DESIGN PHASE:

Synthesis will take place after data has been collected. The data collection methods will be selected early in the design process, and the synthesis will take place upon receiving feedback.

Vision & Goals
Site Analysis
Program Development
Design Alternatives
Feedback & Synthesize
Final Master Plan
Implementation

MATERIALS NEEDED:

Access to an online survey platform
A variety of multiple choice and short answer questions for your stakeholders
Note-taking materials

PREFERRED FORMAT:

This activity is suitable for stakeholder groups that are literate and have access to internet and/or in-person presentations given by design teams. The survey activities should be distributed through an online format to reach a broader audience, but paper versions are also an option. Surveys and their questions should be as clear and concise as possible.

TIME REQUIRED:

The process of completing the survey is asynchronous, but a deadline for completion should be given to participants to encourage timely participation. A timeline for completion should be determined based on the number of participants you hope to have and the deadline of the design project. The survey should be as concise as possible, and ideally less than ten minutes.

SUGGESTIONS:

Do come prepared to your design-alternatives presentation with a series of baseline questions you would like to have answered immediately after the presentation by your stakeholders.
Do have the survey finalized and the URL ready for distribution before your presentation so attendees can complete the survey during or immediately following the presentation.
Do encourage all stakeholders to take the survey so that individuals who feel intimidated have an opportunity to share their opinions.
Do have a data analysis plan in mind prior to creating the survey. Understand that, for certain aspects of the design, not all choices will be made by defaulting to the majority's preference, especially when working with a small sample size.

Overview

The intention of developing a survey that includes both quantitative and qualitative data collection (mixed methods) is to help the designers/investigators understand the reasoning behind stakeholder vision and design preferences. This engagement strategy also helps stakeholders continue to think about the various alternatives that have been presented to them, encouraging participants to reflect on concepts and what makes their preferred alternative appealing to them. This method is most effectively used in design following the presentation of design alternatives to stakeholders. In the case of the 2021 Brooklyn Cemetery team, the mixed methods approach used qualitative data to help explain the data generated from a quantitative study. This strategy is particularly useful if unanticipated findings emerge, or when the designers/investigators feel they will benefit from having additional context to understand their stakeholder's rationale. With this activity, each participant is given a link to an online survey that consists of both multiple choice and short answer questions. Each individual is asked to select their preferred design alternative and is additionally given the opportunity to explain this selection in writing in the subsequent question. This leaves the designers/investigators with results in the form of both a popular vote on the design alternatives, as well as descriptions articulating the reasoning behind the choices made from each participant.

Organization and Process

Steps

To engage participants in a survey, steps need to be taken to both encourage participation, and ensure the necessary questions for the investigator are being addressed adequately. Steps to create and distribute a helpful survey include:

Examine your rationale for using mixed methods research

Research that is best suited for mixed methods is typically research that would benefit from having multiple perspectives of the research problem. Ensure this method is well-suited for your needs, and ensure you have the time to analyze both quantitative and qualitative research results. This is largely dependent on the number of investigators, the number of potential survey respondents, and the timeline of the project. In the case study outlined below, a version of the sequential explanatory strategy was implemented.

Design your survey (convergent parallel strategy)

The convergent parallel strategy is a popular mixed methods design approach. It is characterized by the collection of quantitative data in a first phase of research followed by the collection and analysis of qualitative data in a second phase that builds on the results of the initial quantitative results. Weight typically is given to the quantitative data, and the integration of the datasets takes place when the quantitative results require additional understanding and can be supplemented by the secondary qualitative data.

Determine what it is you want to learn from your survey, and ensure each of the questions you ask can provide you with the insight and guidance you will need to proceed with your design project. A combination of multiple choice questions and short answer questions for design research is a great mixed methods survey approach. This combination allows for respondents to articulate their thoughts about a preference they have selected.

Distribute your survey

After presenting design alternatives to your stakeholders, the survey can be distributed electronically in the form of a URL or a QR code. Survey hosting websites such as Google Forms and SurveyMonkey create links automatically. QR codes often need to be formed through an additional website. If you stakeholder demographic is generally older, consider avoiding the distribution of a QR code. Paper versions of the survey should also be made available if you are interacting with stakeholders in-person, but analyzing results is much faster if everything is conducted through an electronic platform. Encourage your participants to remind the other stakeholders to complete the survey. Establish a closing date for the survey so that stakeholders will be more likely to complete the survey in a timely manner.

Analyze your survey results

Review the answers put forth by your survey respondents starting with the quantitative data. Determine if certain categories have clear popular majorities and be prepared to discuss these results at your next stakeholder meeting. For questions that resulted in close numbers, review the qualitative responses and empathize with the rationale provided behind each participant's decision. These responses can help determine if there was a miscommunication between the designer and the stakeholders, and if this is the case, be prepared to address this at the next stakeholder meeting. If there doesn't appear to be a misunderstanding, but rather, each participant has a unique and distinct preference, be prepared to offer your expert opinion at the next stakeholder meeting. What is your preference as the designer based on your knowledge of the subject and the feedback provided by the survey? Be able to explain these factors at the next stakeholder meeting and prepare to ask attendees to re-answer some of the questions. If the stakeholder meeting is conducted on Zoom or other video chat platforms, participants can answer electronic poll questions in real time.

Determine what the path forward for the design is based on the survey results and your expert opinion, and prepare a presentation for final feedback from your stakeholders.

Consult your stakeholders and go ahead with your design!

Schedule your next stakeholder meeting after the survey response deadline. Present the quantitative results to your stakeholders, and list out some of the rationales provided by respondents. Clearly state your plan for pushing the design forward while reiterating that the design decisions are primarily guided by the opinions expressed in the survey. Ensure that your stakeholders are satisfied with the direction of the project.

Examples

Figure 1. Example of a mixed method survey provided from Brooklyn Cemetery Design Project (Fall 2021). This particular question has a quantitative aspect (the multiple choice question about a preferred road surface material) and a qualitative aspect (the short-answer question that gives respondents a chance to explain their choice).

Figure 2. Quantitative and qualitative results to the question asked in Figure 1. The written responses provide context to the selections made in the multiple choice (quantitative) portion of the question.

Case Example

Flowchart depicting a generalized approach to the convergent parallel design method. Graphic derived from chart published by Subedi, 2016.

This method was applied in October 2021 by the College of Environment + Design MLA students as part of the development of the Brooklyn Cemetery Implementation Plan. Brooklyn Cemetery is a historic burial ground for black Americans located in Athens, Georgia. Tasked with picking up the project from where the Fall 2020 MLA students left it, the 2021 design team developed a project scope that would allow them to progress the work performed by the previous class, building upon concepts including the development of roads, trails, and stormwater conveyance systems, and improving upon a memorial concept initially put forward by David Evans, MLA 2021. The 2020 class had previously completed introductory meetings, community priority input meetings, design concept presentations, and closed design revision review meetings with primary stakeholders. Prior to implementing this method, the 2021 class had met with their primary stakeholder, developed a scope statement that was approved by the primary stakeholder (effectively serving as the program development phase), made several site visits and completed a site analysis, internally iterated several concepts for each of the scope statement items, and presented these concepts in an online-in-person hybrid meeting with the stakeholders. Immediately following the stakeholder meeting, the mixed methods survey and a recording of the presentation was distributed virtually to all stakeholders to solicit input on the next round of refined design revisions. The method was chosen to: receive input from the stakeholders who were unable to attend the meeting in real time; provide a private channel for stakeholders to communicate with the designers so each respondent could openly express their thoughts without fearing judgement from their peers; determine if there were clear preferences for any of the proposed design alternatives; give each respondent the opportunity to explain the reasoning behind the decisions.

Summary of quantitative survey results. These results informed the design team on items including road surface material selections, headstone style and material recommendations, and developing a guide for the bronze artist (to be commissioned at a later date) to communicate the stakeholder vision.

The design team selected this format having already scheduled the design alternatives presentation and knowing that some of our stakeholders would likely be unable to attend the meeting. We wanted to find a way to give voice to those who were unable to join in the conversation in October, and in the process of researching methods that would allow us to do so, we concluded that a mixed methods survey would accomplish this goal in a simple yet effective way. The survey was developed using the Google Forms platform. The team included at least one multiple choice and one short-answer question for each design alternative that was presented at the meeting. The link to the survey was distributed through the Zoom chat to participants that attended the presentation via Zoom. The link was additionally distributed to stakeholders through the Friends of Brooklyn Cemetery listserv.

The survey responses gave the design team an understanding of the stakeholder design preferences. The feedback that was given at the time of the in-person presentation was valuable, but after reviewing the survey results, it was clear that not all of the stakeholders felt comfortable expressing their opinions, concerns or questions at the time of the meeting. The survey provided a safe space for the stakeholders to reach out to the design team, and honestly express their thoughts and preferences without fear of judgement from their peers. If the team had only listened the feedback given in-person at the presentation, we would have overlooked several stakeholder concerns because not all of the participants were equally vocal.

Lessons Learned

The mixed method survey strategy is most successful when:

The design team has the time needed to synthesize the amount of qualitative data being collected
Open-ended qualitative feedback would be a valuable supplement to popular vote counts
Speakers and attendees have reasonable access to online survey platforms and have experience with a video call/conferencing services such as Zoom

Survey design approaches that can encourage participation include:

Grouping survey questions into topics
Questions should appear in a logical order
Surveys should appear short and/or questions should be concise and clearly stated

Doyle, Louise, Anne-Marie Brady, and Gobnait Byrne. "An overview of mixed methods research." Journal of research in nursing 14, no. 2 (2009): 175-185.

Subedi, Dhanapati. "Explanatory sequential mixed method design as the third research community of knowledge claim." American Journal of Educational Research 4, no. 7 (2016): 570-577.

Page updated

Google Sites

Report abuse