D.4. Evaluation of organizational process and evolution of the PPD

This is the first of three sections presenting three different frameworks, tools, and sets of indicators to analyze PPD.

This section focuses on organization process and evolution. The following one looks mainly at the
impact of PPDs on political economy reforms relevant to private sector development. The next looks at more specific outcomes and impacts of a PPD on regulations and economic changes in the environment of the PPD. Ideally they should all be used in combination to obtain a complete overview of the overall results of PPDs.

D.4.1. Results matter, but so do processes: mitigating the risk of mixing subjective and objective information

Monitoring and evaluation of PPD processes is often focused on results. However, the process of
reaching those results – or not reaching them – is equally important.

More insight should be obtained into process aspects of dialogue and partnerships, to identify aspects that need improvement to increase the likeliness of achieving anticipated results.

It is not easy to address process-related aspects in evaluation with traditional logical framework oriented instruments, because processes are experienced and perceived differently by the various stakeholders involved.

Therefore, to some extent evaluation of PPDs also becomes measurement of the subjective opinions and perceptions of people involved in the dialogue. The challenge is to move forward from measurement of individual subjective opinions towards measurement of inter-subjective and crosschecked indicators.

Another related challenge is to deal with aspects that lack written information. Though these might be possible to observe, they may easily be misinterpreted by the consultant or task manager conducting the exercise. This calls for participatory instruments that can generate more objective information on people’s opinions, providing sufficient security for people to express these opinions openly.

A possible approach, presented in this section, is based upon the acceptance that measurement of opinions – provided they are sufficiently cross-checked – can generate sufficiently reliable and useful information on PPD processes.

In addition, other methods and instruments need to be used to pair the inter-subjective opinions with objective and quantifiable data on indicators.

D.4.2. An instrument for evaluating the process and evolution of PPDs

The main aspects of PPDs presented here are based on the PPD Charter (chapter C, above). The table below shows which aspects can be evaluated with the instrument presented later in this section.

Summary Table: Indicators covered with the evaluation wheel (see below) and the appropriateness of use of indicators for benchmarking purposes

Main aspects (following the PPD Charter)	Useful for benchmarking
Mandate and Institutional Alignment What were/are the objectives of the PPD and what was/is its mandate towards the government and the private sector? How does it fit with current institutions?	Yes
Structure and participation How is the PPD structured, does it enable balanced and effective participation?	Yes
Champion(s) Has the PPD identified champions, and how has it tried to leverage them over time to impact the effectiveness of the dialogue process?	Yes
Facilitator(s) and management Did the PPD engage suitable facilitators? How has their role been defined? Have they managed to effectively ensure cohesion and performance? What conflicts did they manage, and how did they resolve these?	Yes
Outputs What outputs does the PPD produce, and under what internal processes? Have outputs from the PPD contributed to agreed private sector development outcomes in the shape of structure and process outputs, analytical outputs or recommendations?	Limited
Outreach and communications Has the PPD communication enabled a shared vision and understanding through the development of a common language and built trust among stakeholders?	Yes
Sub National Has the dialogue been conducted at all levels of decision making down to the most local possible level involving micro-entrepreneurs, SMEs and local stakeholders?	No
Sector-specific Have sector-specific or issue-specific public-private dialogues been encouraged?	Yes
International role Does the PPD represent and promote national and regional interests of both public and private actors in international negotiations and international dialogue processes?	Limited
Post Conflict/Reconciliation /Crisis-recovery Has the PPD contributed to consolidate peace and rebuild the economy through private sector development in post-conflict and crisis environments – includingpost-natural disaster?	No
Development partners Has the PPD benefited from the input and support of donors? How has the donor agenda impacted the decisions of the PPD?	Limited

This approach required caution. Structures and outputs are diverse in different PPD contexts;
international role and post-conflict factors are not always relevant; and the involvement of
development partners can range from absent to crucial. These elements of the charter, then, only have a limited usability for comparison and benchmarking.

The other aspects of the PPD charter, however, have potential for comparison and benchmarking. The indicators presented here under each aspect have been designed in such a way that they can generate comparative data.

The “evaluation wheel” instrument presented below provides the opportunity to cross-check data on similar aspects of different PPDs, or the development of one PPD over time. It generates information for facilitators or program managers to improve their insight into the PPD process, cross-check in focus groups, and use in future desk study.

D.4.3 The “Evaluation Wheel”

The evaluation wheel has been developed to evaluate different aspects of PPDs in a visual way. The number and type of aspects can be customized according to the M&E needs of the PPD and the needs of different stakeholders involved. The wheel presented in this section includes all twelve elements of the PPD charter.22

In this hypothetical example of an evaluation wheel, some aspects of the PPD being analyzed are valued relatively highly – such as structure and participation, outreach and communication. Others are less well developed, such as dialogue at the sub-national level and effective facilitation.

The evaluation wheel can be customized according to context. We recommend, however, using a
standardized set of indicators to compose the evaluation wheel.

D.4.4. Use of the evaluation wheel for comparison and benchmarking

For each of the 12 process aspects represented on the wheel, below are presented two objectively verifiable indicators indexed on a scale from 1 to 10. The average index between different indicators for a single process aspect gives the final score to be plotted on the wheel.

The following matrix presents several indicators that can be objectively verified by the evaluator(s) through interviews and desk study.

#	Operational Process Indicators	Index measurement	Technique to gather information
Mandate and institutional alignment : Average scoring on all indicators on a scale from 0 to 10
1	Existence of mission statement and capacity of participants to explain this mission statement	Non-existence=0; existence (in coherent written document)=10 Percent of respondents who are able to recite the substance of the mission statement; none=0; all=10.	Desk studyInterviews (minimum of 5 interviews with stakeholders)
2	Degree of anchorage of the partnership into existing public institutions, as per its mandate	Percent of participants with decision-making power in their home institutions (none=0 and all=10) Mandate formally accepted and signed by relevant public institutions (none=0; all=10)	Desk study Interviews
Structure and participation: Average scoring on all indicators on a scale from 0 to 10
3	Existence of rules and regulations in the partnership, including formal mechanisms in place to balance power	Non-existence of documents with rules and regulations=0; complete set of clear rules and regulations=10 Equal participation of each stakeholder group (in number and level representatives): unequal or stakeholder groups missing=0; exactly equal=10	Desk studyDesk study Interviews
4	Degree of participatory decision making	Percent of decisions reached by consensus or vote during partnership meetings (none=0; all=10) Active contribution of all different stakeholder groups in developing proposals (none=0; all=10)	Desk study InterviewsInterviews
Champion(s) and leadership: Average scoring on all indicators on a scale from 0 to 10
5	The presence and clear involvement of champions who are recognized as such by stakeholders	Existence of at least one champion in each of the participating stakeholder groups (none=0, all =10) Percent of respondents that identify the same champion(s) (all mention different champions=0, all mention the same one(s)= 10)	Interviews
6	Continuity of involvement of champions in dialogue or partnership	Turnover rate of champions. (high, staying on only a few months=0 to low, continuous presence=10)	Interviews
Facilitation and management: Average scoring on all indicators on a scale from 0 to 10
7	Quality of facilitation of the PPD	Existence of Terms of Reference for facilitators and other members of the Secretariat? Nonexistent= 0; coherent written document=10 Percent of respondents who indicate that facilitators perform well	Desk StudyInterviews
8	Quality of management arrangements (responsibilities, tasks, structure, arrangements etc.)	Existence of task descriptions for manager(s), and – if there is more than one manager – clear division of tasks (non-existence=0, clear description/division=10) Timely availability of project plans and timelines for all stakeholders (no and not for all=0 and yes for all=10)	Desk StudyDesk study Interviews
Outputs: Average scoring on all indicators on a scale from 0 to 10
9	Amount and kind of economic and/or reform proposals in relation to planning	Number and kind of economic and/or reform proposals (none=0, as planned =6, exceeding planning=10)	Desk study Interviews
10	Degree to which dialogue or partnership has innovated or changed existing institutional structures.	% of respondents of external organizations who indicate the PPD has had influence on activities of their organizations. (none=0, all=10) Appreciation expressed by external stakeholders on the performance of the PPD (no knowledge at all/low appreciation=0; detailed knowledge and high appreciation=10)	Interviews with external stakeholders
Outreach and communication: Average scoring on all indicators on a scale from 0 to 10
11	Quality and frequency of communication between different stakeholder groups	Distribution of time between listening and speaking of participants of different stakeholder groups in meetings of the PPD (extremely unequal=0 and very equal=10) Number of misunderstandings or disagreements in communication that are clarified (none=0, all=10)	Observation of meetings Interviews
12	Amount and kind of outreach and communication activities to civil society and media	Amount of money spent yearly by the partnership on media and communication as a percentage of the total budget of the PPD (no budget=0, total amount (needs to be customized to situation)=10) Amount of (written, verbal, television) external communication messages (none=0, total amount (needs to be customized to situation)=10	Desk-study Interviews (internal and external stakeholders)
Monitoring: Average scoring on all indicators on a scale from 0 to 10
13	Quality of reporting and documentation on activities of the partnership	Number and frequency of monitoring reports (on a scale from 0 to 10) % compliance of reporting with qualitative targets set for monitoring (not at all=0, entirely=10)	Desk Study
14	Degree to which monitoring results have resulted in changes in planning and targets	Percent of follow-up actions on recommendations in monitoring reports (no recommendations followed up=0, all recommendations followed up=10)	Desk study Interviews
Sub-national: Average scoring on all indicators on a scale from 0 to 10
15	Existence of local and regional structures or consultation mechanisms for the dialogue or partnership	Consultation of PPD (through formal structures/channels) at further decentralized geographical levels (no at all=0, many channels and all relevant levels=10) Percent of respondents at the level of local target groups (indirect beneficiaries of the PPD) who are satisfied with the performance of the PPD (none=0, all=10)	Desk study Interviews Interviews with beneficiaries and target groups at the local level
16	Existence of activities of the PPD at other levels (local, regional or national) through ad hoc activities or dedicated programs or working groups	Number of activities at other levels than the dialogue and partnership itself (none=0, many and at many different levels=10)	Interviews (internal and external stakeholders)
Sector Specific: Average scoring on all indicators on a scale from 0 to 10
17	Degree to which the dialogue or partnership addresses specific problems of participants	Number of (sub)sector or issue specific working groups in the PPD (none=0, all relevant subsectors=10)	Desk study Interviews
18	Capacity of the dialogue or partnership to generate concrete solutions to specific problems of participants	Number of (sub)sector or issue specific proposals generated (none=10, at least one per year for each (sub)sector or issue=10) Quality of these proposals rated by the evaluator(s) (on a scale from 0-10)	Desk study InterviewsDesk study
International Role: Average scoring on all indicators on a scale from 0 to 10
19	Presence and participation of participants in the dialogue or partnership at international forums and conferences	Number of international events in which representatives of the PPD participated (none=0, all relevant international forums and conference=10) Number of presentations on the PPD for audience as a percentage of total events in which was participated (none=0, all=10)	Desk study Interviews
20	Active consultation and contacts made by international actors to learn from the dialogue or partnerships	Number of international actors who made inquiries with the PPD (none=0, regular inquiries by different international actors (at least 10 inquiries of 5 different actors)=10)	Desk study Interviews
Post-conflict – reconciliation: Average scoring on all indicators on a scale from 0 to 10
21	Capacity to put conflicts on the agenda of the dialogue or partnership and resolve them	Existence of an internal communication strategy to mitigate conflicts (not existent=0, existent (written and coherent)=10) Percent of conflicts that have been peacefully resolved within the PPD according to respondents. Average % of all respondents (none=0, all=10)	Desk studyInterviews (participants in the PPD)
22	Contribution made by the dialogue or partnership to conflict resolution and peace building in its external environment.	Existence of an external communication strategy to mitigate conflicts in the direct external environment of the PPD (not existent=0, existent (written and coherent)=10 Number of relevant conflicts in the direct context of the PPD positively influenced by the PPD, according to external stakeholders. (no influence at all=0, in all conflicts positive contribution noticeable=10)	Desk studyInterviews (external stakeholders)
Development Partners: Average scoring on all indicators on a scale from 0 to 10
23	Degree of dependence of the PPD on financial support of development partners (DPs)	Amount of financial support from DPs as a percentage of the total costs of the dialogue of partnership (total budget provided by DPs=0, more than 50% of budget provided by own resources=10)	Desk study
24	Degree of autonomy of the agenda of the PPD from agendas of development partners	Number of points on the agenda that were promoted by DPs as a percentage of total issues on the agenda. (all points promoted by DPs=0, no points promoted by DPs=10	Desk study Interviews

The design of the evaluation wheel is such that it enables a clear and easily readable picture of a dialogue or partnership, which can be used for comparison of process-evaluation results over time in the same PPD or to compare different PPDs on the 12 elements of the PPD charter.

See next section and Annex D4 for how to assess each of the indicators.

D.4.5. Use of the evaluation for in-depth analysis of a specific PPD

The evaluation wheel is useful for providing more in-depth understanding of the dynamics of a PPD in a particular context, and analyzing differences in appreciation of the PPD in question by stakeholder groups participating in it.

For this purpose, the indicators in the matrix above are verified through the aggregation of individual assessments on each of the 12 elements of the PPD Charter from all stakeholders participating in the dialogue or partnership. These can be cross-checked by focus groups, interviews and desk studies.

See Annex D5 for a questionnaire for collecting individual assessments. Annex D6 provides an evaluation wheel template.

It is possible to represent on the same wheel the appreciation of private sector stakeholders only (represented by the blue area on the below example), government representatives only (the red line), etc.

It is also possible to produce a single wheel with the overall aggregated data of all stakeholders. This wheel would thus present the inter-subjective overall assessment of the PPD as perceived by all stakeholders,and can be usefully compared with the first wheel composed on the basis of objective
indicators.

The evaluation wheel is a powerful visual tool to enable discussion and more in-depth analysis of a PPD in focus group meetings with different participants in the dialogue or partnership. Different aggregated scores of individual assessments for each relevant stakeholder group can easily be compared using the picture above.

In particular, scores that show big differences between different stakeholder groups, and those aspects that are scored low by all stakeholder groups, will need further discussion.

Aspects that show a low scoring of all stakeholder groups will likely require improvements in the design and the process of the PPD. Aspects that show differences in appreciation of different stakeholder group will require improvements in information provision to specific stakeholder groups or changes in set-up, rules and regulations to enable a more balanced participation of all stakeholder groups.

By comparing wheels over time – for instance, with intervals of a year – task managers can analyze if the PPD is developing towards or away from a balanced participation and appreciation by different stakeholder groups.

Public-Private Dialogue

Created by the World Bank, moderated by CIPE

D.4. Evaluation of organizational process and evolution of the PPD

D.4.1. Results matter, but so do processes: mitigating the risk of mixing subjective and objective information

D.4.2. An instrument for evaluating the process and evolution of PPDs

D.4.3 The “Evaluation Wheel”

D.4.4. Use of the evaluation wheel for comparison and benchmarking