Regulatory and economic outputs, outcomes and impacts of PPDs can be evaluated based on the logical framework approach. As objectives, anticipated results and activities are likely to be different in each PPD, the approach presented here focuses upon providing the general logical framework model. The specific sets of indicators will need to be customized to each PPD.
This framework focuses on “hard,” i.e. objectively verifiable outputs, outcomes, and impacts of PPD-processes. As such it complements the process-oriented framework and the framework for measuring the (often indirect) impact on reform processes, set out in the above two sections.
D.6.1. The structure of the logical framework
The logical framework identifies five basic levels in the evaluation exercise, which are related in a chain. This figure presents the five levels (in the first box, two levels are combined) and gives some examples of generic indicators related to each link in the chain.
Logical framework for evaluation
As has been mentioned, M&E is not a stand-alone activity that can be designed retrospectively. Rather, it is closely related to the program cycle. The logical framework is a powerful visual tool to illustrate the relation of M&E with planning and it shows that indicators need to be developed in the planning stage of the PPD process.
D.6.2. The different stages of the program cycle
Indicators are used at several stages in the project or program cycle (see figure). Since characteristics of these stages differ, the functions of indicators may also differ.
D.6.2.1. The identification stage
During the identification stage, the plan for the project or program is being formulated in general terms and, normally, planning indicators are not yet formulated at this stage. Nevertheless, indicators may already play a crucial role in getting more precise information on the context of the intervention and on the problems to be tackled. During this identification stage, task managers should try to reinforce the quality of the situational analysis of the PPD by using specific indicators.
D.6.2.2. The formulation stage
During the formulation stage, the initial plan needs to be worked out in more detail. Generally, several indicators will now be formulated. At this stage, the indicators are called either planning indicators or Objectively Verifiable Indicators (OVI’s) as different persons will come up with the same information.
If the plan is worked out in a logical framework matrix, the planning indicators or OVIs are presented in the second column of this matrix.
Indicators are generally formulated for the overall objective(s), the project purpose and the results. This provides sufficient level of detail for the formulation stage.
Together with the indicators, it is also necessary to identify the sources where the information referred to in the indicator can be found. These are called Sources of Verification (SoV) and included in the logical framework matrix in the third column.
Identification of these SoVs at this stage is highly recommended, since discussions on where to find the information or how to collect it often lead to reformulation of the
indicator. If the information referred to in the indicator cannot be obtained, the indicator becomes useless and a new one should be formulated.
The development of indicators during the formulation stage is crucial. First of all, when objectives are still vague, planning indicators are necessary to obtain information about target groups, timeframe and baseline and target values. Without this information, resource allocation and budgeting as well as operational planning remain almost impossible.
Example of good planning indicators: the Millennium Development Goals
Efforts towards achievement of the UN Millennium Development Goals by 2015 are being
monitored. The framework for reporting includes eight goals, based on the UN Millennium
Declaration. For each goal there is one or more specific targets, along with specific social,
economic and environmental indicators used to track progress towards the goals. Over 40
indicators are identified to monitor progress.
Goal 1: Eradicate extreme poverty and hunger.
Target 1 for 2015:
Halve the proportion of people living on less than a dollar a day and those who suffer from
1. Proportion of population below $1 per day (PPP-values)
2. Poverty gap ratio [incidence x depth of poverty]
3. Share of poorest quintile in national consumption
See also www.un.org/milleniumgoals
For example, an objective such as “increased worker productivity” means little without specifying the exact nature of productivity, and with how much it is supposed to increase, for how many workers, and within what timeframe.
Second, it is also necessary already at the formulation stage to define how to track progress towards the objectives during implementation.
See, for example, the indicators formulated to track progress towards the achievement of the MDGs. With the formulation of clear targets, the MDGs clearly indicate what is to be achieved in terms of development by 2015. The identified indicators are important signals (direct as well as indirect ones; see also below) indicating changes in relation to MDG 1.
As explained above, a complete plan will include information on target group, timeframe, baseline and target values. This information can be included already in the objectives or is further specified in the indicators. The completed plan, including the indicators, will now provide a solid basis for the management during implementation and thus also for monitoring and evaluation.
D.6.2.3. The implementation stage
During the implementation stage, monitoring and evaluation activities provide managers with
particular information, for which – again – indicators are used. Obviously, the planning indicators formulated at the level of results, purpose and overall goal will be the starting point for monitoring and evaluation.
However, during implementation, managers will also be interested in other aspects of the intervention, such as depletion of budget or specific bottlenecks caused by external factors.
In this context, some organizations distinguish input indicators, output indicators, and so on, referring to the type of monitoring or evaluation they are used for (see figure):
D.6.3. Different indicators and focus of monitoring
Moreover, a manager might want to monitor management aspects that are not even mentioned in the
logical framework (such as human resource management issues, reporting frequency, etc.). Based on the specific information needs of managers, a variety of monitoring indicators can be identified.
For evaluation – and, as a consequence, also for indicators for evaluation – the same principle is applicable. Depending on the evaluation criteria – such as relevance, effectiveness, efficiency, sustainability, and impact – the evaluator’s information needs and focus of the evaluation indicators will have to be further defined.
Again, planning and monitoring indicators will be a good basis for evaluation activities. For example, indicators of achievement of goals will facilitate the appreciation of impact, while indicators of results will provide information to evaluate effectiveness.
Monitoring indicators on budget depletion and on use of resources will facilitate the evaluation of efficiency and indicators related to context may well support appreciation of sustainability.
D.6.4. Type of indicators
We can distinguish two types of indicators:
1) Direct indicators, which refer directly to the subject they have been developed for.
2) Indirect indicators, which only refer in an indirect way to the subject.
D.6.4.1. Direct indicators
These indicators directly address the subject of interest, and are often suited to operational and technical subjects. Nonetheless, these might not be easy to measure. A good example of a direct indicator is “the proportion of the population below $1 per day”.
D.6.4.2. Indirect indicators
Indirect indicators, also referred to as proxy indicators, refer in an indirect way to the subject of interest. There can be several reasons to formulate indirect indicators:
- The subject of interest cannot be measured directly. This is particularly the case for more
qualitative subjects, like behavioral change, living conditions, good governance, etc.
- The subject of analysis can be measured directly, but it is too sensitive to do so – for example, level of income or, in the context of an HIV/AIDS intervention, “safe sex.”
- The use of an indirect indicator can be more cost-effective than the use of a direct one. As such, indirect indicators are very typical management tools.
Generally, managers are not looking for scientifically reliable data but for management information. An indirect indicator may well represent the right balance between reliability of information and the efforts needed to obtain the data.
Indicators are defined to provide unambiguous information about interventions and their achievements, but are never an end in themselves. In addition to the reasons outlined above for
defining indicators, the use of meaningful indicators in the evaluation exercise can lead to management information on the PPD.
D.6.5. Standardized indicators, e.g. Doing Business
In addition to customized indicators (see 6.6 below), a limited number of standardized indicators can help to enable comparison and benchmarking of PPDs in different contexts. We advise evaluators to refer to the list of indicators in the Doing Business reports issued every year by the World Bank (www.doingbusiness.org). The reports present a number of objective measurements of the costs of regulatory frameworks and procedures for entrepreneurs.
|Starting a business||Protecting investors|
Cost (% of income per capita)
Minimum capital (% of income per capita)
|Extent of disclosure index (0–10)
Extent of director liability index (0–10)
Ease of shareholder suits index (0–10)
Strength of investor protection index (0–10)
|Dealing with licenses||Paying taxes|
Cost (% of income per capita)
Time (hours per year)
Total tax payable (% of gross profit)
|Employing workers||Trading across borders|
|Difficulty of hiring index (0–100)
Rigidity of hours index (0–100)
Difficulty of firing index (0–100)
Rigidity of employment index (0–100)
Hiring cost (% of salary)
Firing cost (weeks of salary)
|Documents for export (number)
Signatures for export (number)
Time for export (days)
Documents for import (number)
Signatures for import (number)
Time for import (days)
|Registering property||Enforcing contracts|
Cost (% of property value)
Cost (% of debt)
|Getting credit||Closing a business|
|Strength of legal rights index (0–10)
Depth of credit information index (0–6)
Public registry coverage (% of adults)
Private bureau coverage (% of adults)
|Time (years) no practice
Cost (% of estate) no practice
Recovery rate (cents on the dollar)
From the table with Doing Business Indicators above, the evaluators should pick those aspects on which the PPD to be evaluated claimed to have had an impact. The related indicators under the relevant aspects should be added to other customized indicators that have been identified in the design of the logical framework of the PPD process.
The list of Doing Business indicators is also relevant as a reference for the set-up of new PPD processes. During the planning stage, Doing Business indicators can already be included in the project design and the development of the logical framework.
D.6.6. Developing customized indicators
Considerable efforts to formulate standardized indicators are being undertaken, such as the Doing Business Indicators. Nonetheless it will usually be necessary to develop some customized indicators. This section offers some suggestions.
D.6.6.1. Formulation of indicators
Because indicators for planning, monitoring or evaluation serve slightly different purposes, clarity on the status of the project or program in the life cycle is useful before starting to formulate the indicators. Formulation of good indicators cannot be hurried. It is good practice to go step-by-step and worry about the precise formulation of the indicator at a later stage. The following steps may be of help:
Brainstorm on the variables – what are the objectives, in terms of numbers of standards.
2. How much
Define the magnitude of the desired change.
Clarify the target group. Often specific information
on who belongs to the target group is necessary, for
example in cases when gender specificity is required.
Specify information on the intervention area, if this
does not yet become clear from step 3.
Define the timeframe.
The brainstorm on variables (step 1) may lead to a number of different options. Especially when indirect variables are identified, consider the likely validity, accuracy, sensitivity and cost-effectiveness of identifying these variables when deciding which will best serve the information needs of the involved managers.
Cost-effectiveness, of course, also needs to be considered for direct indicators and may be a reason to prefer indirect indicators.
In defining the indicators, emphasis should be placed on trying to ensure that indicators:
- are relevant to the types of technical assistance and facilitation most frequently undertaken in PPDs;
- are valid with respect to reflecting the underlying concept that is meant to be measured,
- are reliable in terms of minimizing potential measurement error, and
- are practical for facilities and regional offices to use given the effort required to collect necessary data.
D.6.6.2. From inputs to impacts
When customizing indicators, clarity is needed about different terms:
- Inputs refer to the resources used in program activities
- Activities are the actions taken or work performed in particular projects using specified inputs. These include assessments, advisory services, training, and public awareness campaigns.
- These activities are intended to result in outputs such as reports, advice, training events, and media coverage.
- In turn, these outputs are expected to yield certain outcomes in terms of changes in knowledge, behavior and performance among beneficiaries in the target population.
- Finally, it is anticipated that programs will generate development impacts including higher productivity, greater income, and economic growth.
D.6.6.3. Choose only indicators relevant to the specific PPD
The design of the PPD process will dictate which indicators are relevant and which are not. From the relevant indicators, select those for which data are available at a reasonable cost.
Examples of PPD-specific indicators
A working group concerned with improving infrastructure conditions to boost the export
market might consider as an impact indicator the time it takes for a truck with standard container to cross the country to the nearest port.
A working group concerned with improving ICT skills among the workforce might consider an
indicator such as the average scoring of ICT students on final examinations of higher
A PPD focused on cutting red tape will need impact indicators such as the % increase of
the number of newly registered companies.
A PPD that is working on improvement of the investment climate for international enterprises might want to measure the percent increase of foreign investments of international companies in a country.
D.6.6.4. Consider carefully the target groups for indicators
Outcome and impact indicators are used to measure different characteristics of PPDs and the results they anticipate for relevant stakeholders. These could be, for example, individuals, financial institutions, BMOs, or private enterprises, and they can be either direct or indirect beneficiaries of the PPD.
Examples of defining target groups for indicators
A working group aiming to improve conditions for micro, small and medium enterprises
(MSMEs) might want an indicator such as a percent increase in sales of MSME’s with a
maximum total turn-over of 10,000 USD per year. It will not be useful to know what is the percent increase in sales of the private sector as a whole.
A PPD working on improving the access of agricultural producers to foreign inputs might want
to know the percent increase of importation of fertilizers. It will not be interested in the percent increase of the value of total imports in the country.
The indicator of increased business registration would be too general for a PPD that is working
on promoting the enabling environment for tourism companies. It will need to know the
percent increase in registration of tourism companies.
D.6.7. The limitations of indicators
It is necessary to bear in mind that changes in indicators do not necessarily prove that it is the PPD which has led to the change. The main use of indicators is for monitoring purposes, to provide a sense of whether members of the target population have been reached and have improved performance on various dimensions.
To be able to attribute observed changes specifically to the PPD requires more rigorous studies that involve comparison with a counterfactual – the hypothetical situation that would have occurred in the absence of the program.
Good impact assessments use random assignment, constructed controls, and/or other approaches to
isolate the impact of the program from other extraneous factors affecting outcomes.
D.6.8. Multiple measures are needed to calculate changes over time
The outcome and impact indicators included in this handbook are expressed as a measurement taken at a specific time. The intention is for programs to establish a baseline and be able to assess changes in the level of particular indicators over time. Changes can be calculated on an absolute or percentage basis.
The budget for data collection and analysis should be commensurate with the scale and importance of the project. As a general rule of thumb, M&E budgets should be in the order of three to five percent of the total program budget.
D.6.9. Indicators for comparative use and evaluative benchmarking of PPDs
This table suggests a limited number of indicators for comparative use and benchmarking of PPDs at the level of outcomes and impacts, which is the focus in the evaluation exercise.
|Anticipated Result||Outcome indicator (examples)|
|Changes in laws and regulations related
to business enabling environment
|Number of laws and regulations promulgated (new or
amended) during the period under review.
|Changes in relevant administrative
|Number of administrative procedures during the
period of review.
|Transaction costs for administrative
|Official cost of starting a business.
Official cost of registering property.
Official cost of enforcing contracts.
|Days required to complete
|Average time to complete each procedure in
|Increase the rate of business
|Number of businesses registered during the year.|
|Increase private sector productivity
|Percentage of enterprises reporting that the business
enabling environment is conducive to their
productivity and growth.
Number of new jobs created.
Amount of FDI registered during the year.