The issue: Like any other activity in which time and money are invested, PPD needs to be monitored and evaluated. But PPD poses particular problems for M&E. One of the most valuable benefits it can bring – an improved atmosphere of cooperation and trust – is long-term and hard to quantify. And while it is a sign of success for a donor-initiated dialogue if participants take ownership of the process, that also makes it harder to set concrete targets in advance.
PRINCIPLE VII: MONITORING AND EVALUATION
Monitoring and evaluation is an effective tool to manage the public-private dialogue process and to demonstrate its purpose and performance.
- While remaining flexible, user-friendly and light, the monitoring and evaluation framework adopted by a PPD should provide stakeholders with the ability to monitor internal processes and encourage transparency and accountability.
- Definition of inputs, outputs, outcomes, and impacts will be enhanced with designation of appropriate indicators with periodic review from stakeholders, which will rely on the collection of reliable data.
- Monitoring and evaluation techniques enable better overall planning, can ignite potential advocacy, and provide both internal and external motivation to promote more effective implementation.
- To this effect, PPDs should develop a baseline assessment to measure their effectiveness in order to enable the partnership to better measure how it is achieving its goals over time and delivering on its envisaged benefits.
C.7.1. Separating monitoring from evaluation
While this principle of the charter deals with M&E, this chapter addresses only internal monitoring which a PPD can do itself. A separate section of this handbook – Section D, below – deals in much greater depth with how the PPD process can be externally evaluated.
Internal monitoring goes hand-in-hand with future planning. It tracks inputs, outputs, and intermediate outcomes. Evaluation, in contrast, is a time-bound exercise which focuses more on the outcome and impact level and aims to show the degree to which the overall objective or goal of the program has been realized.
Below are presented some suggestions taken from international good practice on how PPDs can usefully monitor their activities and thereby learn information that can be helpful in adjusting modes of operation and planning for the future.
C.7.2. Monitor inputs such as human resources and budgets
It is helpful to monitor whether inputs into the PPD process – notably financial and human – are being utilized as envisaged. That implies close tracking of the budget against financial expenditure, and regular reviews of whether staff members are performing along the lines envisaged in their ToRs.
Evidence of divergence between foreseen and actual results may indicate either a need to revisit spending or use of staff time in practice to bring it back into line with what was originally envisaged, or a need to revisit future budgets and ToRs to bring them into line with lessons learned in practice.
C.7.3. Track the impact of each working group’s work through a progress matrix
A simple but highly effective tool that has been used with particular success in Cambodia, Vietnam, and Lao is a progress matrix to track the progress of issues being studied by each PPD working group.
The matrix has five columns. The first describes an issue. The second summarizes the private sector recommendations on the issue. The third is for tracking progress on this issue in working group meetings. The fourth tracks the current status of proposals related to this issue in the PPD forum as a whole (e.g. firm commitments, sub decrees signed, laws submitted to parliament), and the fifth column is for marking achievements.
As is often the case with an issue there may be several recommendations to overcome the issue or its various aspects. These are reflected in the various rows.
A progress matrix can be used in a precise or loose way.
If it tracks pinpointing recommendations that are extremely specific, the matrix provides a sure means to measure the progress of each proposal, which is essential to the process.
But it must be taken into consideration that a PPD process is often a politically sensitive mechanism. Being too prescriptive within the progress matrix or focusing too early on desired outcomes (e.g. laws or regulation passed) may substantively undermine the work of the project management team if the progress matrix is viewed as an accusing tool. The government could see it as an implicit message saying “you have not done enough yet, and we’re pointing a finger at you on the specifics”. For those who have to build and establish the relationships that will in the medium to long term provide the desired outcomes, the progress matrix may then be used is a more flexible way, showing progress on issues, but for instance consider as “implemented” issues that were not resolved, but where the government provided a firm, non-yet-negotiable position on.
C.7.4. Track the selection process of reform proposals generated by PPD
Tracking the progress of reform proposals from initial suggestions through to concrete
recommendations for reform – if this is an intended output of the PPD – provides useful insights into the effectiveness of the suggestion filtering process and also serves to promote transparency and build the legitimacy of final proposals by demonstrating how they came to be adopted.
| BULLDOZER PHASE II – FIRST PLENARY SESSION | Forms distributed by committees | Proposals Received by committees | Proposals preselected by committees | Selected Proposals sent to Bulldozer Board | Selected proposals by Bulldozer Board | Proposals vetted by IMF, WB, EC, USAID, OHR | Proposals selected in First Plenary Session for inclusion into final book of 50 | Proposals on hold for further review |
|---|---|---|---|---|---|---|---|---|
| Northwest | 500 | 29 | 27 | 8 | 7 | 6 | 6 | 1 |
| Northeast | 700 | 40 | 20 | 10 | 5 | 4 | 4 | |
| Banja Luka Region | 600 | 70 | 40 | 10 | 4 | 2 | 2 | 2 |
| Sarajevo Region | 450 | 132 | 32 | 12 | 9 | 7 | 5 | 3 |
| Herzegovina | 200 | 15 | 10 | 4 | 3 | 1 | 1 | 2 |
| Central Bosnia | 1200 | 80 | 25 | 10 | 3 | 1 | 1 | 4 |
| Central Bosnia | 3650 | 366 | 154 | 54 | 31 | 21 | 19 | 12 |
| 100% | 42% | 15% | 8.5% | 5% |
The example above shows how this technique was used and translated into easy-to-follow table format in the Bosnia Bulldozer initiative, which adopted the methodology of distributing forms to entrepreneurs in different regions to solicit reform proposals.
The suggestions were assessed by local-level committees before being sent on for consideration by national-level experts, who then referred selected proposals to experts from the international community for advice on their practicability and desirability. The final selection of reforms was presented to the PPD stakeholders in plenary session for their official adoption.
This table shows the process of winnowing by which only the strongest 5% of initial suggestions for reform proposals received made it through to be adopted as recommendations.
C.7.5. Track steps in proposing reforms and having them implemented
Each PPD works slightly differently depending on its domestic circumstances, but it is usually possible to devise a step-by-step framework showing the ideal progression towards implementing private sector development reforms.
The below ladder of reform shows five steps taken by a private sector forum, from gathering
suggestions to presenting final proposals for government consideration, and five further steps which the government is then expected to take, from analyzing proposals to verifying their implementation.
Using such a ladder to keep track of progress in issues, in the case of the private sector forum, and in implementation, in the case of the government, helps to identify delays and bottlenecks in the process.
The upper half of the ladder is especially helpful as it can be translated into a tracking table to monitor government progress on reform proposals, with each proposal generated by the private sector forum being assigned a score from one to five on the basis of how far it gets along the government part of the ladder.
C.7.6. Use monitoring to promote accountability and transparency
Internal monitoring mechanisms provide a PPD with an opportunity to promote the values of transparency and accountability. In certain political contexts, the very act of monitoring and distributing the results regularly to stakeholders can play a valuable role in setting an example to government and corporate partners.
Finally, an important element of internal monitoring is understanding what is a success in the local context. Where low levels of trust exist, it is important to demonstrate that the dialogue itself is a successful outcome. Clarification of the government’s position on a bureaucratic process can be as important, in the short term, as the passing of critical legislation after PPD dialogue will be in the future. In this context, the writing and maintaining of accurate records that respect the solemnity of the PPD process is important. Accurate, agreed minutes, are also an important ongoing mechanism for increasing transparency and providing a mechanism from which to launch the next round of consultations.
But while the above tools have their utility, capacity often varies greatly within the government and private sector in developing economies. It is not always practicable to over-evaluate and prescribe a strict adherence to procedures as outlined above. As stated, each country has its own practices and it is important that the PPD process finds the right path towards achieving its stated goals.