2. Case Studies

As is true for M&E in general, the more straightforward the data is to be collected, the easier it will be to design and use a mobile phone application for the M&E. For example, much health related data is easy to measure; we can quantify the number of cases reported, number of clinic visits, instances of infection, rates of recovery. The same applies to humanitarian aid; the amount of food distributed at X point, tents set up, water collection points are relatively easy to track. So when searching for potential applications of mobile phone based M&E, the majority of case studies currently using mobile M&E will be for either humanitarian aid or public health.

We are therefore profiling a humanitarian case study, a health care related case study, and a case study which is neither, to illustrate that there are potentially broader applications of mobile based M&E.

But basic M&E points must still be followed:

1. What data is to be collected?

2. How will the data capture fit within the rest of the M&E analysis?

3. How will the data be analyzed, and who will be doing the analysis?

4. How will the process be managed, and by whom?

An excellent compendium of case studies of mobile phone for development can be found on the MobileActive site, click case studies

Case Study I: Humanitarian Aid

McRam: Multi-Cluster Rapid Assessment Mechanism is a project the UN has been using in Pakistan for humanitarian emergencies since 2008. To go to the site click here.

PROJECT

The project was designed to address the difficulties in time-lag in gathering accurate post-emergency data, to permit more accurate targeting of aid. The project started in Pakistan in 2008, after the earthquake, and has continued to be used. However, as the McRam field manual notes, "PDAs do not do anything magical to the questions.The key to a good assessment will always be having well designed questions and well trained people to ask them." The field manual is a comprehensive guide to setting up the questionnaires, to read the guide itself, go to McRam field manual.

The McRam tool was developed with the collaboration of the humanitarian community; NGOs, UN agencies, and INGOs. The tool is called multi-cluster because one team goes to each site, and asks ALL the relevant questions. That is, previously, a WATSAN (water and sanitation team) would go on site, and then a nutrition team might go, and then a livelihoods team, each aid sector sending their own team of assessors. By combining all the various sector questions into one form (VERY long) this approach helps with:

• • avoiding assistance fatigue...telling the same story over & over before receiving aid

  • quicker data gathering

  • quicker issuance of reports resulting in more targeted aid

Teams were trained for two days on the questionnaires themselves, and one day on the PDAs themselves. The initial use of McRam in 2008 revealed that it was necessary to create separate teams for men and women, reflecting the different roles and responsibilities each hold within Pakistani culture.

In a meeting the group had with with UN needs assessment expert Kimberley Lietz, she noted that programs which have recipient based use of data collection can "cause problems regarding the data. It is unclear how many people respond, which can skew the results. So we are finding that it makes more sense to use the technology for practitioners, i.e. those in the field."

Herbert Tatham, Assessment Needs Coordinator for UNOCHA in Pakistan, who has been working with McRam noted that most of the lessons learned to date are still "for internal uses only, and not yet available to the public". However he said that they are keen to expand the use of mobile phones for needs assessment and M&E, as they appear to be bringing good efficiency and accuracy gains for their program.

LESSONS LEARNED

• 1. The first two assessments revealed the necessity of creating separate questionnaires for men and women. Women, due to their restricted environment, and lower literacy rate, required a questionnaire reflecting their movements and areas of responsibility.

  2. The tool is only as good as the questions and the analysts. Analysts need to be well trained to gather the collected data, interpret it and translate results into clear reports.

  3. Projects using mobile phone based data collection have to incorporate technical IT trainers as part of their programming.

Case Study II: Health Care Delivery: Project Mwana

PROJECT

Project Mwana is a program developed in conjunction with UNICEF Innovation, Ministry of Health for Malawi and Zambia, frogdesign, and Zambia and Malawi UNICEF offices. The immediate goal of the project is to increase mothers' visits to clinics significantly by January 2012 in rural Zambia and peri-urban Malawi by leveraging mobile technologies in innovative ways. The longer-term goal is to develop a communication system that can be scaled across many different countries in partnership with other UNICEF country offices. The software uses SMS (text messages) to deliver HIV test results for infants directly to health clinics and enhance communication within the clinics. Using SMS in this way means mothers are informed about their child’s HIV status 1–2 months earlier than would otherwise be possible. If necessary, the child can then be started on critical treatment. The project includes an SMS component for results delivery and as well as a customized web dashboard for reporting and administration.

The project elected to have trainers use their own cellphones, thereby avoiding the necessity of training them on a new piece of equipment.

Time spent on lab tests in the field, samples sent to the capital, and results received by clinics in the field shorten from several months to two weeks and in some cases more quickly than that. This encouraged the project to widen the potential uses, to track visits, send SMS messages to mothers to remind them of visits.

This data was also sent directly to the Ministry of Health in both countries, enabling the Ministry's to gather real time information about rural health.

LESSONS LEARNED

• 1. The project stressed having users enter data on their own phones. This ensured easier training and comprehension for clinic workers, and wider potential implementation.

  2. Self or peer training became easier

  3. More training needed at central and Ministerial level for analysts to analyze, graph, interpret and make use of the information which is being gathered.

For a more complete study of the project, blogs about it, and a slide show, click Project Mwana

Case Study III: Fair Trade Compliance: Mexico

PROJECT

Yael Schwartzman started a project in Mexico in 2008, designing a program to use smart phones for monitoring of free trade compliance. She worked with a consortium, CEPCO of 2800 coffee producers in Oaxaca. Conversion from paper monitoring to mobile phone based monitoring resulted in 38% reduction in inspection time and 69% reduction in evaluation time, when compared to the earlier paper-based approach. These efficiency gains would reduce the cooperative’s yearly expenses by $10,100, or 48% of the total internal inspection costs. A paper she wrote describing the project in detail can be found by clicking here

In discussion with Yael, she said that their software system was developed prior to the development of Open Data Kit a popular open source software which is increasingly used in the mobile phone M&E application world.The project created an application which was used on a Nokia smart phone. The focus was to increase efficiency in the fair trade monitoring process. Yael commented that "technology has improved since our pilot, and we are looking into more use of ODK in future projects."

LESSONS LEARNED

• 1. Users reported that the application did not allow for open-ended comments and observations. If their thoughts/observations were not captured by the questionnaire, there was no way to include any additional information.

  2. Battery failure was an issue, particularly when in remote locations with no access to electricity to recharge the phone. Users learned to bring additional batteries into the field.

  3. Using the phones to take pictures or videos could cause additional complications with the instrument, which may or may not be a limitation of the phone used in the project.

  4. Concern about the potential for cheating/hacking.