Public-Private Dialogue

A website of the OPEN PRIVATE SECTOR platform
Created by the World Bank, moderated by CIPE.

Created by the World Bank, moderated by CIPE

  • Facebook
  • Twitter
  • YouTube
  • HOME
  • LESSONS LEARNED
  • COUNTRY CASES
    • Africa
    • Asia
    • Eurasia
    • Latin America & the Caribbean
  • TOOLS
    • PPD Charter
    • PPD Handbook
    • Monitoring and Evaluations
      • The Evaluation Wheel/The Reform Process Table
      • The PPD Reform Tracking Tool
      • Monitoring and Evaluation for Business Environment Reform: A Handbook for Practitioners
      • The PPD Performance Indicators for Donors
    • Operational Resources
      • Manual for Trainers on PPD and Good Governance
      • Quick Guides to Integrating Public-Private Dialogue
      • Key Documents for Setting up PPD Projects
  • WORKSHOP 2015
    • Past Workshops
      • Workshop 2014
      • Workshop 2011
      • Workshop 2010
      • Workshop 2009
      • Workshop 2008
      • Workshop 2007
      • Workshop 2006
  • eLEARNING
  • BLOG
  • ABOUT
    • Contact
You are here: Home / Tools / PPD Handbook / D. Monitoring and Evaluation Framework for PPD / D.2. Project management guidelines: three steps toward evaluating PPD

D.2. Project management guidelines: three steps toward evaluating PPD

Setting up and conducting an evaluation framework for PPD can effectively follow a generic three-step
approach.19

D.2.1. Step One: Design of the M&E project brief

A project brief for the M&E exercise consists of the following elements:

(a) Background information about the PPD, and the context under which the M&E exercise takes place.
(b) Defining the basic questions that need to be addressed by the M&E framework, the main indicators
and how they can be verified;
(c) A description of the approach, methods and tools for the M&E framework. The following methods
and tools provide a generic menu:

Methods Possible Tools
Desk-study interviews (with internal and external stakeholders)
Questionnaires (internal and external stakeholders)
Observation (Site-visits and observing meetings)
Focus group meetings (for analysis and/or validation and discussion of findings)
Debriefing and feedback
Reporting
  • Checklists
  • Open and closed questions techniques
  • Most Significant Change Interview format
  • List of open and closed questions and control-questions
    (clusters and constructs)
  • (Self-) assessment questionnaires
  • Statistical software to process information
  • Written and digital channels to collect and process
    information from questionnaires
  • Models and formats to structure information
  • Facilitation techniques
  • Visual formats to organize and present data in focus group
    meetings
  • Facilitation techniques
  • Feedback and confronting techniques
  • Format for M&E reporting
  • Matrix to analyze information requirements of stakeholders
    in the M&E process

From the menu above, a minimal set of methods and corresponding tools needs to be selected to ensure that data can be collected and analyzed on input, output, outcomes and impact. In addition to this, some methods and tools are also needed to acquire data on the process.

The elements in bold in all cases need to be part of the M&E framework; the other elements can be selected depending on available budget.

(d) A budget that balances the available resources for M&E and the needs of the M&E framework. The final two phases are iterative. If the methods and tools selected surpass the available budget, new and more restrictive choices have to be made on the methods and tools to be used.

Once the right balance is found, the following elements can be added to the project brief:

1. A project management framework with a description of tasks and responsibilities for the internal
stakeholders to conduct M&E on the PPD;

2. A description of competencies and tasks for an external consultant to co-conduct the M&E, and a
description of the reporting format, deliverables, time-path and available resources for the M&E
process in a ToR.

Sample ToR instructions are provided in Annex D1.

This matrix outlines the main competencies an external consultant should have, and the main tasks
that are associated with each:

Behavioral competencies Tasks related with competency
Communication skills
Capable of bringing across information and
ideas in understandable language, verbal
and written.
Interviewing
Facilitation of meetings
Report writing
Analytical skills
Able to analyze a complex problem,
situation or process and understands its
causes.
Conduct the M&E, using specific approaches,
methods and tools
Cross-cultural awareness
Capable of understanding and mediating
different cultural values and practices – in
the public, private and civil sectors – in
multi-stakeholder settings.
Facilitation of multi-stakeholder processes with
participants of different sectors.
Feedback and discussion of results of M&E
exercise, and jointly developing with stakeholders
follow-up actions in such a way that all
stakeholders feel ownership of the process.
Creativity and enterprise
Can recognize possibilities, come up with
solutions and invent new methods when
necessary. Willing to take calculated risks.
Develop recommendations that are relevant and
applicable in specific development contexts and
acceptable for different stakeholders involved.
Knowledge and Experience related Competencies Tasks related with competency
PSD experience Analyze M&E data and interpret in light of broader
knowledge and experience in private sector
development.
Previous work in developing countries
and emerging markets
Analyze M&E data and interpret in the light of
knowledge of social economic development
processes and pro-poor economic growth.
Knowledgeable about PPDs Analyze M&E data and interpret them in the light
of relevant public-private dialogue and
partnerships processes.
Development of recommendations through cross-fertilization
of relevant PPD experiences.
Technical expertise on M&E approaches
and frameworks
Collect data, using sound M&E approaches and
instruments. Correctly analyze data and develop
conclusions and recommendations, showing a
clean chain of reasoning.

When the tasks for the consultant in the M&E process are clear and the desired competencies have been
identified, the final element can be put in place:

3. Selection, recruitment and briefing of external consultants.

The following matrix is helpful for developing the contents of the project brief.

  • Define level of involvement of stakeholders in the M&E process
Design Matrix: Planning research and developing a project brief
Background
information
  • Short history of the PPD process
  • Describe the context of the PPD
  • Describe the stakeholders in the PPD, including the specific stakes they
    have in the process
  • Relevant available lessons learned (from previous M&E data)
Basic questions
  • Define the information requirements of stakeholders involved in the PPD
  • Define the information requirements of initiators of the M&E process
    (facilitators and/or donors)
  • What is the program doing? Define a small set of generic “W” questions:
    What? Who? Where? When? Why? How?
Indicators and
sources of
verification
  • Presentation (first outline) of generic indicators (output, outcome and
    impact level)
  • Indication of main sources of verification (available and/or to be
    developed in the M&E exercise)
Basic approach
  • Focus on monitoring, evaluation or both?
  • Focus on process, results or both?
  • Focus on outputs, outcomes, impact or all?
  • Focus on hard, soft data or both?
Specific methods
and tools
  • Select basic methods and tools: desk-study, surveys, interviews, focus group
    meetings and/or others
Task-division in
M&E process
  • Describe responsibilities and tasks of relevant persons involved in the M&E
    exercise
Time-line
  • Define the timeline for the M&E process
  • Describe the main steps of the M&E process and indicate deliverables at
    the end of each step
Requirements of
consultants
  • Set out behavioral competencies, knowledge and experience-based
    competencies, and other formal requirements
Budget
  • Budget
  • Sources of funding

D.2.2. Step Two: Conduct the data-collection

Step Two usually will take the bulk of the time and resources in the M&E process – particularly when one wants to evaluate the process and impact of a PPD, not all information will be readily available.

Under step one, a first selection of basic approaches and tools was made. In this step, specific instruments will have to be selected to ensure the M&E framework is coherent and reliable; guaranteeing reliability of objective (hard) data and sufficiently cross-checked inter-subjective (soft)
data.

The following is a list of possible data collection methods, with a brief indication of the strong and weaker points of each of those methods.

  • Expensive
  • Data usually not
    statistically significant
  • Expensive
  • Detailed
  • Reliable
  • Can track relationships overtime

td>

  • Detailed
  • Reliable
  • Can track
    relationships over
    time
Data collection Description Suitable for evaluation Advantages Disadvantages
Sample survey Collection of data using
questionnaires with a fixed
format. Subjects: households
(social-economic survey); a
sector (farm management
survey); or an activity
(enterprise survey).
  • Assessment
  • detailed data
  • Statistical
    significance
  • Relibale
  • Time consuming
  • Expensive
  • Information is static
Case Studies Collection of data through
regular interviews (e.g. once
every month or fortnight )
for a longer period (often a
year).
  • Assessment
  • Description
  • Detailed Data
  • Dynamic Information
  • Very Reliable
  • Very time consuming
Tracer studies Determination of effects of
intervention over a longer
period
  • Assessment
  • Description
  • Detailed

  • Reliable
  • Can track relationships over time
  • Expensive
Record keeping Data collected and recorded
by interviewee
  • Assessment
  • Data collection is
    cheap
  • Very detailed
    information
  • A lot of data to
    process
  • Danger of bias of
    interviewee
Rapid appraisal Combination of data
collection and instant
appraisal in the field
  • Description
  • Appraisal
  • Quick Results
  • Relatively cheap
    because of
    concentration on
    essentials
  • Can only be applied by
    experienced experts
  • Reliability and validity
    of information cannot
    be guaranteed
Visual observation Observation of
activities/results/
issues
  • Description
  • Judgment
  • Cheap and quick
  • Observation is
    free of bias
  • Interpretation is
    difficult; can be a
    tendency to over interpret
  • Can only be applied by
    experienced experts
Individual interviews Collection of data through
interviews with informers,
“open” interviews or
“structured” interviews, with
questionnaires as part of a
sample survey
  • Description
  • Judgment
  • Facilitation
  • Provides opinion
    of individual
    interviewees
  • Time Consuming
  • Bias of interviewees
    sometimes difficult to
    assess
Group interviews Group discussion with
informants. The interviewer
acts as discussion leader
  • Description
  • Facilitation
  • Provides a clear
    insight into
    differences of
    opinion and how
    these are
    considered
  • Discussion may
    lead to further
    analysis
  • Neutral position of
    interviewer sometimes
    difficult to maintain

In section 3, some of these instruments – the ones which are particularly relevant for M&E of PPD processes – are described in more detail.

D.2.3. Step Three: Writing the process evaluation and impact assessment

The final step in the M&E process is to draw up the report on the PPD process. In general, an impact
report should:



  • have clear separation of findings, conclusions, and recommendations;>/li>
    
  • deal with inputs, outputs, outcomes, and impacts;
  • tackle questions of relevance, efficiency, effectiveness, and sustainability;
    address the process;
  • 

  • identify lessons to be disseminated; and
  • 

  • balance between judgment, for accountability purposes, and learning.
  • Process evaluation and impact assessment reports on a PPD ideally should also contain information on:

  • the organizational effectiveness and process development of the PPD;
  • economic impacts of the PPD; and
  • reform process impacts.
  • In sections 4, 5, and 6, each of these aspects of PPDs will be dealt with in detail.

    The final report should be written in two stages. First, a draft final report should enable stakeholders to provide final comments on the findings and analysis – but not the conclusions and recommendations, which are the sole responsibility of the evaluator. After this feedback, the final report can be written.

    D.2.4. Dissemination

    Evaluation reports can contain a wealth of information from which many other actors can learn, so it is worth considering whether and how to disseminate these findings more widely than with the direct stakeholders of the PPD.

    For the purposes of wider dissemination, the report might need to be edited to avoid confidential information being disclosed to third parties, and to ensure the report is understandable for readers who do not know the context of the dialogue.

    • Download >> PDF (146k)

      Table of Contents

      D.1. Introduction

      D.2. Project management guidelines: three steps toward evaluating PPD

      D.3. Methods and Instruments for data collection

      D.4. Evaluation of organizational process and evolution of the PPD

      D.5. Evaluation of the political economy impact of PPDS on reform processes

      D.6. Evaluation of regulatory and economic outcomes and impact of PPDS

      D.7. Concluding Remarks

      Rights & Permissions

      The material in this work is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The sponsors of the work encourage dissemination of its work and will normally grant permission promptly.

      For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the World Bank, DFID or the OECD Development Center.

      ALL RIGHTS RESERVED
      Small and Medium Enterprise Department
      The World Bank Group

      Latest Tweets

      • Loading tweets...

      Follow @PPDialogue

      In Partnership With

      CIPE World Bank Group International Finance Corporation DfID GIZ OECD

      About | Legal | Privacy Policy | Terms Of Use

      All Rights Reserved © 2016

      Copyright © 2023 · Genesis Sample Theme on Genesis Framework · WordPress · Log in