Validating quantitative data triangulation model
The AMFm evaluation team developed a theory of change to depict the causal pathways through which AMFm interventions were intended to work.
This theory of change was used to target the collection of quantitative and qualitative data that would be used to prepare case studies for each country.
The GEF conducts original evaluative research, including theory-based approaches, during the evaluation to assess progress toward desired impacts in the face of sparse data.
It also uses qualitative methods and mixes emerging evidence with available quantitative data through systematic triangulation with the ultimate goal of identifying evaluation findings.
The evaluation matrix then feeds into a triangulation matrix.
“We categorize the evaluative evidence in three major research areas of perceptions, validation, and documentation,” said Carugi, explaining that perceptions, while not always reflecting reality, need to be accounted for in these evaluations (see Figure 8-1). brainstorms question by question to populate the matrix and discuss which findings are real and which need further analysis.
The study used outlet surveys at baseline and endline, household survey data for some of the countries being evaluated, and documentation of key contextual factors.
All of these methods are deployed within the context of an evaluation matrix that the GEF develops for each evaluation.
For example, Madagascar only met one success metric out of six, but the data from the case study revealed some process and contextual reasons for the country’s relatively poor performance.
Orders for medication were low to begin with because of low confidence among importers, and the information and communications campaign was limited because in-country regulators decided that direct-to-consumer advertising was inappropriate and banned the practice 1 month into the program.
In the triangulation process, the evaluation team FIGURE 8-1 Three major areas of evaluative evidence in the Global Environmental Facility country program evaluations, as presented by Carugi. “After the triangulation brainstorming meeting, we go back to the drawing board and try to confirm or challenge the key preliminary evaluation findings and try to identify what we can do to fill in the missing key preliminary evaluation findings,” said Carugi.
DEVELOPING A DEEPER AND WIDER UNDERSTANDING OF RESULTS As described previously, the AMFm evaluation, said Catherine Goodman, senior lecturer in health economics and policy in the Department of Global Health and Development at the London School of Hygiene and Tropical Medicine, was a before-and-after study that did not have control areas.The development of standardized protocols, procedures, and methodologies for triangulation have helped address this criticism.TRIANGULATION IN THE GLOBAL ENVIRONMENTAL FACILITY Carugi described the use of triangulation in the GEF, which is a partnership among 183 countries with international institutions, civil society organizations, and the private sector to address global environmental issues while supporting national sustainable development initiatives.Carugi said the GEF’s country portfolio evaluations are all conducted in a standardized way using standard terms of reference and questions for comparability purposes.