Recurring errors in public policies and major projects: contributions and solutions from evaluation

@FAOEvaluation

Recurring errors in public policies and major projects: contributions and solutions from evaluation

Hello dear Community,

I hope that you are all doing well.

The topic I am proposing today concerns the main errors made in the design of public policies, especially those with a big social impact, or concerning major rural development projects (roads, electricity supply, telecommunications, dams, etc.):

  • What are the most common mistakes made in your country?
  • To what extent do you think these mistakes could have been avoided with better use of evaluation, or that evaluation could contribute to the success of policies and major development projects?
  • Have the results of evaluations allowed to amend the failures of public policies and development projects?
  • Have public policies and projects developed subsequent to the conducted evaluations taken into account previous errors and corrections?

Thank you all for the quality of the exchanges and the important responsiveness of the community!

Hynda Habchi Krachni
Ministère de Finances
Algérie

 

This discussion is now closed. Please contact info@evalforward.org for any further information.

Recurring Errors in Public Policy and Major Development Projects: Contributions and Solutions from Evaluation (Commentary)

Public Policies are only as good as the available finances to implement and enforce them. This then requires a viable tax base to support the civil services. Unfortunately, too often the host country does not have the necessary tax base. This is what I refer to as a financially suppressed economy. That is an economy serving a mostly impoverished society in which up to 80% of all income or production must be spent on a meager diet to feed the population. This leaves so little discretionary funds that can be taxed.  No taxes, no services!! This leaves most host governments financially stalled, barely able to meet their salary and fringe benefit obligations for the civil officers with virtually no operational funds to implement programs or support policies. Thus, most public policies are paper polices, show a governments’ good intentions, and perhaps appease donors interested in providing financial assistance and funding development projects.

The result is:

  1. that the data going into policy decisions might be of limited quality, in proportion to the budget allowance to collect it. This was elaborated on in a previous submission to this forum.
  2. The service proclaimed as provided maybe highly compromised by relying on the honor/gratuity/baksheesh system. As appears to be the case for certified seed in Kano, Nigeria.

One must be very careful in building a technical capacity that can not be operationally supported with adequately supported financially. This can be a major disservice to the intended beneficiaries and possible the general public. While not meeting certified seed quality standards may not be particularly deferential, claiming a food safety program based mainly on the honor/gratuity/baksheesh system can easily result in contaminated food getting into the supply chain, resulting in multiple sickness and even death. Better to have no system than a misleading system to prevent people becoming over confident in the food safety and avoiding taking personal caution in preparing food.

There may not be much you can do about this other than be aware that it can and mostly likely is occurring. Also, it might be better to minimize the involvement of host governments in projects as the marginally paid civil officers are usually looking for some informal (politely avoiding the horrible “C” word) opportunities from within the project and can be more a hinderance than an assistance to project implementation. Unfortunately, too often the informal income is a financial necessity for civil officers to obtain any reasonable professional live style.

Please review the following webpages from the https://smallholderagriculture.agsci.colostate.edu/ :

https://smallholderagriculture.agsci.colostate.edu/financially-suppressed-economy-2/

https://smallholderagriculture.agsci.colostate.edu/financially-stalled-governments/ 

https://smallholderagriculture.agsci.colostate.edu/impact-of-financially-stalled-government-limited-variety-improvement-seed-certification/ 

https://smallholderagriculture.agsci.colostate.edu/informal-income-opportunities-seed-fertilizer-voucher-program-of-afghanistan/

Thank you,

Dick Tinsley

Colorado State University

Chère communauté.

Je suis désolée de ne pas être revenue plus tôt vers vous. Cela est dû à un grand chantier professionnel, avant le départ en congé.

Merci pour vos contributions et le partage de vos opinions et analyses, concernant ma question sur les erreurs lors de la conception des politiques publiques et des projets.

Vos réponses sont toutes pertinentes et se complètent parfaitement.

Un merci particulier à Mustapha MALKI, à Lal-Manavado, à Isha Miranda,Emile N.Houngbo et à Gninnan Oumar Sako, pour votre réactivité et pour la qualité de vos contributions.

Hynda Habchi Krachni 

Greetings!

I think your questions can be given generic answers, whose importance may vary according to the prevailing local conditions. I would like to underline that no 'development effort' will be free of some or all of them irrespective of the political maturity and economic status of a country. After all, even though great deal of lip service is done to deal with 'thinking in silos', precious little is being done to guard against 'working in silos' throughout the world. Let us always keep this distinction in mind if we wish to be realistic.

Now to the questions:

What are the most common mistakes made in your country?

The greatest mistakes are the failure to ascertain the following before a project/plan is designed and executed:

1. Are the beneficiaries willing and able to derive sufficient real benefits from it with reference to their abilities and expectations and not according to those of some distant planners?

2. Are there sufficient local resources both physical and human, needed to make the best use of what has been planned? For instance, there is no need for a multi-million Dollar bridge to transport a couple of tonnes of vegetables to a nearby city.

3. Does the area/country has sufficient technical skills and financial resources to maintain the end-result on its completion?

4. Are there other better alternatives to the current proposal? For instance, in an area where high unemployment rates obtain, it would be more appropriate to select a labour-intensive alternative than a capital-intensive high tech one. After all, the purpose of every development effort should be to enable as many people as possible to secure a decent livelihood.

This list is not exhaustive, but I think its general drift is quite clear.

• To what extent do you think these mistakes could have been avoided with better use of evaluation, or that evaluation could contribute to the success of policies and major development projects?

This is indeed a tricky question. If we speak of evaluation in a very narrow technical sense as it is often done, it can not be of much use here. However, if we are willing to work out of silos as it were, and opt for a holistic notion of evaluation, it could make a significant contribution. That 'if' is logical.

Let me explain; if we are planning to evaluation not the mere hardware of en effort say a hospital or a bridge, and extend our activities to its intended purpose, i.e., benefiting a group of real live people, then it would be invaluable. This might be called pre-effort evaluation of possiblebenefits made with reference to the 4 points above.

• Have the results of evaluations allowed to amend the failures of public policies and development projects?

If evaluation is only concerned with 'the hardware', it could influence policies and implementation strategies only insofar as they are concerned with the end-result, but never with its benefit yield to real people. For example, better bridge building strategies can result in better bridges, but that does not address the question of their utility. So, it would be reasonable to suggest that only holistic pre-evaluation could be of use in better policy design and implementation strategies.

• Have public policies and projects developed subsequent to the conducted evaluations taken into account previous errors and corrections?

Another contributor has already made many perspicacious remarks on this.

Best wishes!

Lal.

Dear All, 

I am sharing the Sri Lankan experience as a contribution to this debate. I am currently participating in the development of the Policy framework of M&E and one of the guiding principles we are using is the SDG concept of "Leaving no one behind".

What are the most common mistakes that you face in your country?

  1. Common mistakes are linked to the fact that politicians make election promises that are not achievable and deliver election winning economic analysis.
  2. The biggest issue is that the country holds to outdated policies and laws, which remain unattended. For instance, it is common that countries that have been under a colonial era still have policies drafted in that era being used in government economic and social governance as well as laws in the country, which are ineffective and irrelevant.
  3. Too many governing structures: national, provincial and local authorities govern in parallel but they have underneath different political agendas which can be seen mishandling some policies, misinterpreting national priorities, mismanaging national interventions and resources etc.
  4. Politicization of governance structure and bureaucracy: everybody wants to remain in power and to remain in the position, which paralyzes the governance.
  5. Corruption at all levels, and hidden corruption even more dangerous than the visible one.

Here are my suggestions to improve the use of evaluation in policies and programmes:

  1. Key to the any country economics planning is setting up a separate independent unit for monitoring and evaluation under the Act or Policy of government constitution/legislation to safeguard taxpayers money, accountability and transparency of the government programmes and projects/ interventions. This unit can be established under either the Planning ministry or the Ministry of finance but needs to maintain independence.
  2. The National Auditor general should focus performance audit function, operation or the management systems and procedures of a government entities to assess whether the entity is achieving economy, efficiency and effectiveness in the welfare and employment of available resources. This is a qualitative method.
  3. The Ministry of policy planning or another responsible entity should undertake in-depth analysis of all policies and trade agreements, government circulars and amendments with policy activities, to identify the gaps and lesson learned
  4. Develop a National M&E policy and Policy framework (the Sri Lankan government is in the progress of developing the framework). The M&E Policy can be mandatory. Set up expert and technical committee of National M&E / reviews and assessments committee. This will be the backbone of the planning and finance entities. The committee responds to the ministry of finance and Ministry of policy planning and economic reform.
  5. Promote joint evaluation with other funding and donor agencies and public participation in order to strengthen ownership.

Dear all,

I will begin by answering your last question: "Have public policies and projects developed subsequent to the conducted evaluations taken into account previous errors and corrections?"

To this question, I answer NO. If previous mistakes and corrections were regularly taken into account in Africa, and especially in French-speaking African countries, these countries would have developed a long time ago. Serious problems jeopardize the development and management of public policies in these countries. First, the development of these public policies and programs are often not assigned to true professionals in the field. But, this is not the most serious issue. Most importantly, monitoring and evaluation of public policies is not taken seriously, just as managers of these programs are often in disagreement with monitoring and evaluation officers. I really know something about this as I have been responsible for monitoring and evaluation for several development projects. However, the lack of emphasis on monitoring and evaluation prevents us from really following the established indicators and, in the end, it prevents a good database, information and relevant statistics to guide the improvement of the quality of future programs. The issue of monitoring and evaluation is very serious. In other cases, it is thought that it is a waste to spend money on monitoring and evaluation. Developing countries, however, are the ones who value monitoring and evaluation. Because, it allows them not to repeat the same mistakes and go faster in the implementation of future programs. Really, I do not know if one day we will begin in our countries to better consider the monitoring-evaluation to the point of following the recommendations it allows to identify. If that happened, it would be a big step towards development. It is my wish.

========== 
Dr Ir. Emile N. HOUNGBO 
Agroéconomiste, Enseignant-Chercheur 
Directeur, Ecole d'Agrobusiness et de Politiques Agricoles
Université Nationale d'Agriculture, Bénin 

Hello everyone,

First of all, a big thank you to Hynda who puts on the table a well relevant topic of debate that we must all bring answers to without taboos in order to put at the center evaluation compared to many related considerations about development work in every possible sense of the term “development”.  

In her message, Hynda highlights the term "mistake" as it is perceived by many of us in everyday life. However, we should place this term in the context of public policy planning in order to distinguish the intentional and the unintentional error in planning in our countries and in public policies more specifically. Then, Oumar tries to bring some snatches of answer but ends up very quickly pouring into the normative instead of staying in the real - what is done and why.

For my part, given my modest experience in administration and my modest research to understand how what I will call the "development theater" works, I caricature the sphere of development by the existence of different roles played by different actors and therefore the presence of different rationalities.

I would like to say at the outset that there is no pure naïve person who believes that development is an apolitical work that obeys exclusively to technical considerations. So when we talk about error in this context, we must talk about these unintentional errors that we could identify in our evaluation of public policies and it is also necessary that evaluation appreciates how and in what knowledge context these policies were formulated. This is where evaluation can become an interesting instrument to show us that the mistakes that we can identify through our evaluations are far from being unintentional. Indeed, such errors are strongly related to the balance of power that exists in the "theater of development" when planning a public policy. Here I agree with Oumar who admits that we have not done a lot of evaluations in our countries and that even when sometimes they are done, they are rather done as part of a "ballet folklore "- very often "imposed" by foreign donors.

And since these evaluations are done in the context of development programs and projects, the repercussions in the sphere of public policy planning remain limited, if not none, and the results of these evaluations are never seen as tools to help decision-making. It is important to recognize the separation in some of our countries between programs and development projects financed by foreign donors and public policies funded through public budgets and therefore most often fall within the domain of national sovereignty.

This is not to say that there are no other mistakes in public policy planning, such as lack of scientific and technical knowledge to develop a coherent public policy with relevant objectives and realistic outcomes. Mistakes that can come from a real lack of knowledge (either proven skills, reliable statistics, etc.) can come from other causes related to the famous "balance of power" mentioned above and this brings us back to the need to distinguish the intentional from the unintentional in our planning mistakes.

There are errors related to the existence of a "one and only" document that allows any reader to understand the public policy that some senior sector official talks about. I have personally experienced many examples of senior sectoral officials who spoke of a sectoral policy that existed only in their "head".

There are also errors related to the setting of objectives and results that are relevant and clear, evaluable, etc. As a Results-Based Management Specialist, I know something about the resistance some high level officials have and their need to avoid this kind of debate in the public policy planning document, when it exists and is made public. This makes things more difficult when talking about "accountability" of decision-makers in terms of achieving the objectives.

Other types of errors can be identified with respect to the allocation of resources for public policy and the logical links of allocated resources to the objectives and results assigned to the public policy.

Finally, there is another type of mistake that relates to "changeability" of the public policy. In a number of sectors, public policy is launched on the basis of ideas that are still insufficiently identified or apprehended and that sector policy officers are eager to implement in the field; then as the feedback (inconsistency protest, etc.) comes back from the field, public officers improve the "content" of the policy and this is done recurrently throughout the life of the policy, which makes it difficult to evaluate.

All this to say that the problem is not in public policy or its evaluation.

It is only fair that under such circumstances, the evaluation of a public policy elaborated in an administrative "straitjacket" devoid of logic and without knowledge is not possible and will have no conclusive result on the improvement of the development work ... It can only be used to tell the sectoral politician what he likes to hear, and this is not the role of evaluation and what is done in the advanced world.

Yours truly

Mustapha

 

Mustapha Malki, PhD

535 avenue Ampere #5

Laval, QC, Canada

Dear All,

This is a very interesting and timely topic of discussion.

In my view, the fact is that very few evaluations are carried out in our African countries because of the idea or opinion that often people have about evaluation. In fact, evaluation is not always perceived as a tool for decision-making. Efforts should focus first and foremost on improving this view of evaluation by presenting it as a tool for decision-making and as an instrument for improving public policy formulation.

Also, the presentation and the communication around the conclusions of evaluations will have to be improved to facilitate the appropriation of these conclusions by the policy makers who could use them to guide the formulation of the public policies.

In sum, improving the formulation of public policies on the basis of consideration of evaluation results requires first good communication about the importance of evaluations in order to further promote the practice and also the refinement of methods or techniques for presenting these results to policy makers in order to capture their attention.

SAKO Gninnakan Oumar

Expert en Planification stratégique, suivi-évaluation

Côte d'Ivoire