Setting up and conducting an evaluation framework for PPD can effectively follow a generic three-step
approach.19
D.2.1. Step One: Design of the M&E project brief
A project brief for the M&E exercise consists of the following elements:
(a) Background information about the PPD, and the context under which the M&E exercise takes place.
(b) Defining the basic questions that need to be addressed by the M&E framework, the main indicators
and how they can be verified;
(c) A description of the approach, methods and tools for the M&E framework. The following methods
and tools provide a generic menu:
| Methods | Possible Tools |
|---|---|
| Desk-study interviews (with internal and external stakeholders) Questionnaires (internal and external stakeholders) Observation (Site-visits and observing meetings) Focus group meetings (for analysis and/or validation and discussion of findings) Debriefing and feedbackReporting |
|
From the menu above, a minimal set of methods and corresponding tools needs to be selected to ensure that data can be collected and analyzed on input, output, outcomes and impact. In addition to this, some methods and tools are also needed to acquire data on the process.
The elements in bold in all cases need to be part of the M&E framework; the other elements can be selected depending on available budget.
(d) A budget that balances the available resources for M&E and the needs of the M&E framework. The final two phases are iterative. If the methods and tools selected surpass the available budget, new and more restrictive choices have to be made on the methods and tools to be used.
Once the right balance is found, the following elements can be added to the project brief:
1. A project management framework with a description of tasks and responsibilities for the internal
stakeholders to conduct M&E on the PPD;
2. A description of competencies and tasks for an external consultant to co-conduct the M&E, and a
description of the reporting format, deliverables, time-path and available resources for the M&E
process in a ToR.
Sample ToR instructions are provided in Annex D1.
This matrix outlines the main competencies an external consultant should have, and the main tasks
that are associated with each:
| Behavioral competencies | Tasks related with competency |
|---|---|
| Communication skills Capable of bringing across information and ideas in understandable language, verbal and written. |
Interviewing Facilitation of meetings Report writing |
| Analytical skills Able to analyze a complex problem, situation or process and understands its causes. |
Conduct the M&E, using specific approaches, methods and tools |
| Cross-cultural awareness Capable of understanding and mediating different cultural values and practices – in the public, private and civil sectors – in multi-stakeholder settings. |
Facilitation of multi-stakeholder processes with participants of different sectors. Feedback and discussion of results of M&E exercise, and jointly developing with stakeholders follow-up actions in such a way that all stakeholders feel ownership of the process. |
| Creativity and enterprise Can recognize possibilities, come up with solutions and invent new methods when necessary. Willing to take calculated risks. |
Develop recommendations that are relevant and applicable in specific development contexts and acceptable for different stakeholders involved. |
| Knowledge and Experience related Competencies | Tasks related with competency |
| PSD experience | Analyze M&E data and interpret in light of broader knowledge and experience in private sector development. |
| Previous work in developing countries and emerging markets |
Analyze M&E data and interpret in the light of knowledge of social economic development processes and pro-poor economic growth. |
| Knowledgeable about PPDs | Analyze M&E data and interpret them in the light of relevant public-private dialogue and partnerships processes. Development of recommendations through cross-fertilization of relevant PPD experiences. |
| Technical expertise on M&E approaches and frameworks |
Collect data, using sound M&E approaches and instruments. Correctly analyze data and develop conclusions and recommendations, showing a clean chain of reasoning. |
When the tasks for the consultant in the M&E process are clear and the desired competencies have been
identified, the final element can be put in place:
3. Selection, recruitment and briefing of external consultants.
The following matrix is helpful for developing the contents of the project brief.
- Define level of involvement of stakeholders in the M&E process
| Design Matrix: Planning research and developing a project brief | |
|---|---|
| Background information |
|
| Basic questions |
|
| Indicators and sources of verification |
|
| Basic approach |
|
| Specific methods and tools |
|
| Task-division in M&E process |
|
| Time-line |
|
| Requirements of consultants |
|
| Budget |
|
D.2.2. Step Two: Conduct the data-collection
Step Two usually will take the bulk of the time and resources in the M&E process – particularly when one wants to evaluate the process and impact of a PPD, not all information will be readily available.
Under step one, a first selection of basic approaches and tools was made. In this step, specific instruments will have to be selected to ensure the M&E framework is coherent and reliable; guaranteeing reliability of objective (hard) data and sufficiently cross-checked inter-subjective (soft)
data.
The following is a list of possible data collection methods, with a brief indication of the strong and weaker points of each of those methods.
- Expensive
- Data usually not
statistically significant
- Expensive
- Detailed
- Reliable
- Can track relationships overtime
td>
- Detailed
- Reliable
- Can track
relationships over
time
| Data collection | Description | Suitable for evaluation | Advantages | Disadvantages |
|---|---|---|---|---|
| Sample survey | Collection of data using questionnaires with a fixed format. Subjects: households (social-economic survey); a sector (farm management survey); or an activity (enterprise survey). |
|
|
|
| Case Studies | Collection of data through regular interviews (e.g. once every month or fortnight ) for a longer period (often a year). |
|
|
|
| Tracer studies | Determination of effects of intervention over a longer period |
|
|
|
| Record keeping | Data collected and recorded by interviewee |
|
|
|
| Rapid appraisal | Combination of data collection and instant appraisal in the field |
|
|
|
| Visual observation | Observation of activities/results/ issues |
|
|
|
| Individual interviews | Collection of data through interviews with informers, “open” interviews or “structured” interviews, with questionnaires as part of a sample survey |
|
|
|
| Group interviews | Group discussion with informants. The interviewer acts as discussion leader |
|
|
|
In section 3, some of these instruments – the ones which are particularly relevant for M&E of PPD processes – are described in more detail.
D.2.3. Step Three: Writing the process evaluation and impact assessment
The final step in the M&E process is to draw up the report on the PPD process. In general, an impact
report should:
address the process;
Process evaluation and impact assessment reports on a PPD ideally should also contain information on:
In sections 4, 5, and 6, each of these aspects of PPDs will be dealt with in detail.
The final report should be written in two stages. First, a draft final report should enable stakeholders to provide final comments on the findings and analysis – but not the conclusions and recommendations, which are the sole responsibility of the evaluator. After this feedback, the final report can be written.
D.2.4. Dissemination
Evaluation reports can contain a wealth of information from which many other actors can learn, so it is worth considering whether and how to disseminate these findings more widely than with the direct stakeholders of the PPD.
For the purposes of wider dissemination, the report might need to be edited to avoid confidential information being disclosed to third parties, and to ensure the report is understandable for readers who do not know the context of the dialogue.