Methods

Methods

Adapting the Most Significant Change To Capture Forward-Looking Visions 

iDE adapted the Most Significant Change (MSC)(2) methodology to capture forward-looking changes our potential iDE-powered entrepreneurs and clients would like to see in their lives over the coming years. MSC is traditionally used as a participatory evaluation method that involves generating personal accounts of change and then engaging different community-level stakeholders to determine which changes are the most significant and why(3). Our adaptation of it focused on the most significant changes iDE clients would like to see in their lives and communities in the short and long term and why. This concept underscores the significance of allowing communities to define and determine their priorities and approaches based on their unique needs, perspectives, and aspirations. It implies that individuals, families, and communities have the agency to make decisions aligning with their own agenda and vision and leverage their knowledge and resources to support one another. 

We chose the MSC methodology because of its ability to uncover whether our ideas as an organization around what prosperity means are aligned with how our clients see prosperity. Asking a diverse set of clients open-ended questions allowed us to find out. A focus on participatory methods - such as co-created design, customers defining success metrics, and sales metrics - allow us to put client choice at the forefront of what we do. We also knew this method would surface outcomes and context-based learnings we may not yet have thought of to inform our technical interventions, and most importantly we chose it for its participatory nature, which allows for clients and other stakeholders to lead the analysis process at the local level. In addition to MSC, we qualitatively analyzed the raw data to capture and aggregate client visions of prosperity by major themes across all our country programs to inform iDE global strategy. 

2. Davies, R. and Dart, J. (2005). The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use”. (Link) 

3. “Most Significant Change.” Better Evaluation, Global Evaluation Initiative, 3 Nov. 2021, https://www.betterevaluation.org/methods-approaches/approaches/most-significant-change 

Participatory MSC Training

In an effort to conduct training on MSC for all the country offices, the MERL team in conjunction with the Innovation Lab/Design team developed a course. The course included a virtual component with reading materials and weekly reflection sessions led by Henok Begashaw (HQ-MERL) and Fatima Shehata (HQ-Design). The virtual training occurred during January - February of 2023 in preparation for the in-person Inclusive MERL Summit planned for March 2023 in Lusaka, Zambia.

The virtual training sessions culminated with the in-person gathering for the Inclusive MERL summit in Lusaka, Zambia in the first week of March 2023. During this summit, all MERL leads from every iDE country office besides Honduras and Nicaragua were represented. The in-person gathering allowed for more detailed training including prompt development, role-playing, and pre-testing with clients of the HTTG project in the Central Province of Zambia. Following the pre-testing all the MERL staff were able to reflect on what worked and what did not, resulting in the development of a training and implementation draft plan for each of their country programs. Please refer to the MSC prompt used for this activity in Annex A.

Story Collection

To collect stories capturing individuals’ definitions of prosperity, we developed a general interview guide that iDE’s global MERL teams adapted on a country level for their data collection contexts. The original interview guide (attached in Annex A) included questions about changes in clients'  lives over the next year, five years, and ten years, and questions that sought to understand the significance of these changes. The interview guide also included further probing questions relating to definitions of organizational goal terms such as "prosperity" and "resilience", the domains of change (such as work, health, family, community, etc.), iDE sectors (agriculture, diet, water, sanitation, and gender), barriers to achieving the stated changes, and perceived confidence, power, and skills to achieve those changes.

Sample

This adapted MSC process was implemented in nine iDE countries. Stories of client visions of change were collected through 249 interviews in communities where iDE works, determined by the country teams based on logistics and project locations. The sample for collected stories of change in Bangladesh, Cambodia, Ethiopia, Honduras, Nepal, Mozambique, and Vietnam each included approximately ten females who identified as entrepreneurs, ten females who did not identify as entrepreneurs and ten males from either category. In Ghana and Zambia, the teams leveraged an MSC visioning story collection that was part of the baseline for iDE’s multi-country agriculture project called Her Time To Grow due to the similarity of the target sample, prompts, and objectives, which included all women. 

As this was a study meant to build our understanding around how the people iDE works with think about prosperity, we used a purposive sampling process to ensure we created an inclusive sample and therefore captured visions across the diverse life experiences iDE clients represent. Beyond gender and entrepreneurial status, iDE’s global Gender Equity & Social Inclusion (GESI) team came together to recommend that each country draw the sample to be inclusive of the following parameters: 

• Name

• Age

• Main ethnicity and/or religion of your household 

• Disability, as defined and measured by the Washington Group Short Set on Functioning (WG-SS)

• Marital status

• Highest level of education achieved

• Government poverty classification if applicable (ie - Cambodia: IDPoor status, Vietnam: Poor/Near/Non, Ethiopia: if on CBHI or PNSP lists, etc.)

Story Selection Process

These collected stories were then shared with participatory Selection Committees at either the country level or regional (sub-country) level, as decided by each country MERL team in consultation with their GESI representatives, typically depending on the geography and demography of each country. These Selection Committees invited various stakeholders, including iDE clients, private sector actors, community leaders, local government officials, and other actors (FBAs, latrine producers, agricultural extension agents), who determined which story was “the most significant,” per the MSC methodology. The process of determining the "most significant" story involved first defining clear criteria of what constitutes a “most significant story” that included factors that the community classified as most valuable. In MSC, this process is considered “analysis by selection” because values are identified by the Committees by discussing stories and then determining selection criteria. The primary inclusion criteria for these Selection Committee members was to identify those stakeholders that have strong knowledge of the lived experience of the storytellers.

Teams in each country facilitated the Selection Committees, adapting the number of committees and processes to their contexts. Fifteen Selection Committees were conducted in various locations around the world, with the locations shown in Figure 1. Detailed descriptions of each committee are detailed in Annex C.

Additional Qualitative Analysis

Following the MSC process, collected stories and summary reports of selection criteria from committees were analyzed qualitatively by iDE’s HQ MERL team using Dedoose software (a software for analyzing qualitative data including text). One researcher coded all of the MSC stories, and a second researcher coded the Selection Committee reports and did an additional review of a subsample of the MSC stories. We utilized content analysis, applying both inductive and deductive coding approaches. Definitions of the codes identified and applied are available in Annex B. Qualitative content analysis complements the MSC methodology by providing a structured approach to identifying, organizing, and interpreting patterns in the data at the global level, which both supports a nuanced understanding of visions of change from storytellers and allows for global themes to emerge. 

There were three primary types of codes applied to stories and discussed throughout the report: Themes, Domains, and Definitions. A Theme is a recurring idea that emerged in stories, such as “Personal Transportation,” “Labor / Employment,” or “Nutrition.” Themes can be nested within one another, for example “Natural Resource Management '' and “Climate & Weather Impacts” are sub-categories nested within the “Environment” Theme.  Domains, which included Household, Business & Entrepreneurship, and Community, are areas in which the described changes take place. Definitions coded areas where respondents defined the key terms prosperity and resilience.

Our MERL teams across the country offices translated and transcribed 249 stories which were analyzed in Dedoose, with storytellers representing both men and women, various age groups, and those who self-identified as entrepreneurs and non-entrepreneurs. Although additional stories beyond the 249 were collected, we could not transcribe or translate all of them into English and then analyze them at this aggregate level due to time and other resource constraints. For example, 45 stories overall were collected from Vietnam and 30 of those were included in the qualitative analysis. However, all stories selected as Most Significant were fully transcribed, and all transcribed stories were incorporated into qualitative analysis. The demographic composition of stories analyzed in Dedoose is detailed in Table 1. In addition to the stories, we also analyzed the reporting forms completed by our Monitoring, Evaluation, Research, & Learning (MERL) teams in each country detailing the MSC selection process and criteria from the Selection Committees.

Limitations

While the combination of MSC and qualitative content analysis provides a participatory and rigorous approach to assessing definitions of prosperity through the eyes of the clients, certain limitations should be considered.

First, the sample size, approximately thirty per country, is not representative of the areas covered by story collection. Therefore, the findings should be considered as highlighting a diverse set of life experiences present in the iDE context sampled, rather than a complete representation of the perspective of the communities where the samples were selected. Given that this was an inductive process meant to help us learn about how a diverse set of our clients define success in their lives, it was not meant to test any theories or hypotheses about the places we work.

Second, is a consideration of bias. Several forms of bias are present in the data collection and analysis. An important distinction is needed however when considering participatory methods. These methods aim to incorporate the perspective, and thus bias, of storytellers and selection committee members. Our purpose was to amplify the voices of those who are not usually involved in the measurement process. Therefore, we are seeking out a wide range of biases from these community stakeholders. These aspects of bias are intentionally included and are the aim of the MSC method as well. However, there are other biases that were not included intentionally which should frame how we interpret the data. Stories from clients may be impacted by acquiescence bias (the tendency to agree or affirm) and social desirability bias (when respondents provide answers that they perceive as favorable) rather than expressing true beliefs or behaviors. An example in this case could be the vast majority of people affirming, “Yes, I have the skills and confidence to achieve these changes” when asked about their confidence in achieving change, because of a tendency to agree and/or a desire give a favorable answer to the researcher asking the question. Additionally, as many individuals interviewed have existing relationships with or awareness of iDE programs, sponsorship bias (the inclination to provide responses that align with perceived expectations to maintain support, connection, or benefits from iDE) may have distorted responses. Sponsorship bias may be especially present where MSC was embedded within ongoing iDE projects. Bias can also manifest in qualitative analysis when researchers, consciously or unconsciously, bring their perspectives to the interpretation of data, potentially shaping the findings. 

The third limitation to acknowledge is the relatively recent adoption of the MSC methodology by by iDE teams. The iDE MERL and GESI teams are in the process of learning to effectively facilitate participatory processes and gather and analyze qualitative data for MERL purposes, which has implications for data consistency and quality. At the same time, this study was an opportunity for iDE MERL teams around the world to receive training and gain practical experience in participatory methods. 

Finally, due to the large amount of data and numerous themes present in the 249 MSC stories, not every theme is included in this report. If you are interested in more information or further analysis of specific findings in the report, you are welcome to contact measurement@ideglobal.org for additional details.