Regulatory and economic outputs, outcomes and impacts of PPDs can be evaluated based on the logical framework approach. As objectives, anticipated results and activities are likely to be different in each PPD, the approach presented here focuses upon providing the general logical framework model. The specific sets of indicators will need to be customized to each PPD.

This framework focuses on “hard,” i.e. objectively verifiable outputs, outcomes, and impacts of PPD-processes. As such it complements the process-oriented framework and the framework for measuring the (often indirect) impact on reform processes, set out in the above two sections.

D.6.1. The structure of the logical framework

The logical framework identifies five basic levels in the evaluation exercise, which are related in a chain. This figure presents the five levels (in the first box, two levels are combined) and gives some examples of generic indicators related to each link in the chain.

As has been mentioned, M&E is not a stand-alone activity that can be designed retrospectively. Rather, it is closely related to the program cycle. The logical framework is a powerful visual tool to illustrate the relation of M&E with planning and it shows that indicators need to be developed in the planning stage of the PPD process.

D.6.2. The different stages of the program cycle

Indicators are used at several stages in the project or program cycle (see figure). Since characteristics of these stages differ, the functions of indicators may also differ.

D.6.2.1. The identification stage

During the identification stage, the plan for the project or program is being formulated in general terms and, normally, planning indicators are not yet formulated at this stage. Nevertheless, indicators may already play a crucial role in getting more precise information on the context of the intervention and on the problems to be tackled. During this identification stage, task managers should try to reinforce the quality of the situational analysis of the PPD by using specific indicators.

D.6.2.2. The formulation stage

During the formulation stage, the initial plan needs to be worked out in more detail. Generally, several indicators will now be formulated. At this stage, the indicators are called either planning indicators or Objectively Verifiable Indicators (OVI’s) as different persons will come up with the same information.

If the plan is worked out in a logical framework matrix, the planning indicators or OVIs are presented in the second column of this matrix.

Indicators are generally formulated for the overall objective(s), the project purpose and the results. This provides sufficient level of detail for the formulation stage.

Together with the indicators, it is also necessary to identify the sources where the information referred to in the indicator can be found. These are called Sources of Verification (SoV) and included in the logical framework matrix in the third column.

Identification of these SoVs at this stage is highly recommended, since discussions on where to find the information or how to collect it often lead to reformulation of the
indicator. If the information referred to in the indicator cannot be obtained, the indicator becomes useless and a new one should be formulated.

The development of indicators during the formulation stage is crucial. First of all, when objectives are still vague, planning indicators are necessary to obtain information about target groups, timeframe and baseline and target values. Without this information, resource allocation and budgeting as well as operational planning remain almost impossible.

For example, an objective such as “increased worker productivity” means little without specifying the exact nature of productivity, and with how much it is supposed to increase, for how many workers, and within what timeframe.

Second, it is also necessary already at the formulation stage to define how to track progress towards the objectives during implementation.

See, for example, the indicators formulated to track progress towards the achievement of the MDGs. With the formulation of clear targets, the MDGs clearly indicate what is to be achieved in terms of development by 2015. The identified indicators are important signals (direct as well as indirect ones; see also below) indicating changes in relation to MDG 1.

As explained above, a complete plan will include information on target group, timeframe, baseline and target values. This information can be included already in the objectives or is further specified in the indicators. The completed plan, including the indicators, will now provide a solid basis for the management during implementation and thus also for monitoring and evaluation.

D.6.2.3. The implementation stage

During the implementation stage, monitoring and evaluation activities provide managers with
particular information, for which – again – indicators are used. Obviously, the planning indicators formulated at the level of results, purpose and overall goal will be the starting point for monitoring and evaluation.

However, during implementation, managers will also be interested in other aspects of the intervention, such as depletion of budget or specific bottlenecks caused by external factors.

In this context, some organizations distinguish input indicators, output indicators, and so on, referring to the type of monitoring or evaluation they are used for (see figure):

D.6.3. Different indicators and focus of monitoring

Moreover, a manager might want to monitor management aspects that are not even mentioned in the
logical framework (such as human resource management issues, reporting frequency, etc.). Based on the specific information needs of managers, a variety of monitoring indicators can be identified.

For evaluation – and, as a consequence, also for indicators for evaluation – the same principle is applicable. Depending on the evaluation criteria – such as relevance, effectiveness, efficiency, sustainability, and impact – the evaluator’s information needs and focus of the evaluation indicators will have to be further defined.

Again, planning and monitoring indicators will be a good basis for evaluation activities. For example, indicators of achievement of goals will facilitate the appreciation of impact, while indicators of results will provide information to evaluate effectiveness.

Monitoring indicators on budget depletion and on use of resources will facilitate the evaluation of efficiency and indicators related to context may well support appreciation of sustainability.

D.6.4. Type of indicators

We can distinguish two types of indicators:
1) Direct indicators, which refer directly to the subject they have been developed for.
2) Indirect indicators, which only refer in an indirect way to the subject.

D.6.4.1. Direct indicators

These indicators directly address the subject of interest, and are often suited to operational and technical subjects. Nonetheless, these might not be easy to measure. A good example of a direct indicator is “the proportion of the population below $1 per day”.

D.6.4.2. Indirect indicators

Indirect indicators, also referred to as proxy indicators, refer in an indirect way to the subject of interest. There can be several reasons to formulate indirect indicators:

• The subject of interest cannot be measured directly. This is particularly the case for more
  qualitative subjects, like behavioral change, living conditions, good governance, etc.
• The subject of analysis can be measured directly, but it is too sensitive to do so – for example, level of income or, in the context of an HIV/AIDS intervention, “safe sex.”
• The use of an indirect indicator can be more cost-effective than the use of a direct one. As such, indirect indicators are very typical management tools.

Generally, managers are not looking for scientifically reliable data but for management information. An indirect indicator may well represent the right balance between reliability of information and the efforts needed to obtain the data.

Indicators are defined to provide unambiguous information about interventions and their achievements, but are never an end in themselves. In addition to the reasons outlined above for
defining indicators, the use of meaningful indicators in the evaluation exercise can lead to management information on the PPD.

D.6.5. Standardized indicators, e.g. Doing Business

In addition to customized indicators (see 6.6 below), a limited number of standardized indicators can help to enable comparison and benchmarking of PPDs in different contexts. We advise evaluators to refer to the list of indicators in the Doing Business reports issued every year by the World Bank (www.doingbusiness.org). The reports present a number of objective measurements of the costs of regulatory frameworks and procedures for entrepreneurs.

From the table with Doing Business Indicators above, the evaluators should pick those aspects on which the PPD to be evaluated claimed to have had an impact. The related indicators under the relevant aspects should be added to other customized indicators that have been identified in the design of the logical framework of the PPD process.

The list of Doing Business indicators is also relevant as a reference for the set-up of new PPD processes. During the planning stage, Doing Business indicators can already be included in the project design and the development of the logical framework.

D.6.6. Developing customized indicators

Considerable efforts to formulate standardized indicators are being undertaken, such as the Doing Business Indicators. Nonetheless it will usually be necessary to develop some customized indicators. This section offers some suggestions.

D.6.6.1. Formulation of indicators

Because indicators for planning, monitoring or evaluation serve slightly different purposes, clarity on the status of the project or program in the life cycle is useful before starting to formulate the indicators. Formulation of good indicators cannot be hurried. It is good practice to go step-by-step and worry about the precise formulation of the indicator at a later stage. The following steps may be of help:

1. What
Brainstorm on the variables – what are the objectives, in terms of numbers of standards.

2. How much
Define the magnitude of the desired change.

3. Who
Clarify the target group. Often specific information
on who belongs to the target group is necessary, for
example in cases when gender specificity is required.

4. Where
Specify information on the intervention area, if this
does not yet become clear from step 3.

5. When
Define the timeframe.

The brainstorm on variables (step 1) may lead to a number of different options. Especially when indirect variables are identified, consider the likely validity, accuracy, sensitivity and cost-effectiveness of identifying these variables when deciding which will best serve the information needs of the involved managers.

Cost-effectiveness, of course, also needs to be considered for direct indicators and may be a reason to prefer indirect indicators.

In defining the indicators, emphasis should be placed on trying to ensure that indicators:

• are relevant to the types of technical assistance and facilitation most frequently undertaken in PPDs;
• are valid with respect to reflecting the underlying concept that is meant to be measured,
• are reliable in terms of minimizing potential measurement error, and
• are practical for facilities and regional offices to use given the effort required to collect necessary data.

D.6.6.2. From inputs to impacts

When customizing indicators, clarity is needed about different terms:

• Inputs refer to the resources used in program activities
• Activities are the actions taken or work performed in particular projects using specified inputs. These include assessments, advisory services, training, and public awareness campaigns.
• These activities are intended to result in outputs such as reports, advice, training events, and media coverage.
• In turn, these outputs are expected to yield certain outcomes in terms of changes in knowledge, behavior and performance among beneficiaries in the target population.
• Finally, it is anticipated that programs will generate development impacts including higher productivity, greater income, and economic growth.

D.6.6.3. Choose only indicators relevant to the specific PPD

The design of the PPD process will dictate which indicators are relevant and which are not. From the relevant indicators, select those for which data are available at a reasonable cost.

D.6.6.4. Consider carefully the target groups for indicators

Outcome and impact indicators are used to measure different characteristics of PPDs and the results they anticipate for relevant stakeholders. These could be, for example, individuals, financial institutions, BMOs, or private enterprises, and they can be either direct or indirect beneficiaries of the PPD.

D.6.7. The limitations of indicators

It is necessary to bear in mind that changes in indicators do not necessarily prove that it is the PPD which has led to the change. The main use of indicators is for monitoring purposes, to provide a sense of whether members of the target population have been reached and have improved performance on various dimensions.

To be able to attribute observed changes specifically to the PPD requires more rigorous studies that involve comparison with a counterfactual – the hypothetical situation that would have occurred in the absence of the program.

Good impact assessments use random assignment, constructed controls, and/or other approaches to
isolate the impact of the program from other extraneous factors affecting outcomes.

D.6.8. Multiple measures are needed to calculate changes over time

The outcome and impact indicators included in this handbook are expressed as a measurement taken at a specific time. The intention is for programs to establish a baseline and be able to assess changes in the level of particular indicators over time. Changes can be calculated on an absolute or percentage basis.

The budget for data collection and analysis should be commensurate with the scale and importance of the project. As a general rule of thumb, M&E budgets should be in the order of three to five percent of the total program budget.

D.6.9. Indicators for comparative use and evaluative benchmarking of PPDs

This table suggests a limited number of indicators for comparative use and benchmarking of PPDs at the level of outcomes and impacts, which is the focus in the evaluation exercise.

Evaluation of Regulatory and Economic Outcomes and Impact of PPDS