RE: Neutrality-impartiality-independence. At which stage of the evaluation is each concept important?   | Eval Forward

Again, we have another interesting, critical, and challenging topic to comment on. As I have previously stated, the most important contribution of any project evaluation is the guidance it provides to future project so they can better serve the beneficiaries. Unfortunately, too often evaluations become propaganda tools necessary to appease donors and assure future projects, while doing little if anything for the intended beneficiaries.

Thus, while the commentary shows a good consensus on the importance of Neutrality-impartiality-independence, the commentary also shows that obtaining such can be very challenging as rarely do projects provide funds for external reviews by independent evaluators, but if when funds are available, vested interest in potential future assignments, will often bias the objectively needed to be sufficiently critical of results to guide future projects.  

One way to minimize the bias might be to have some clear well-defined targets that will separate project success from failure. Has anyone ever seen a set of evaluation criteria that would do this? I have not! This set of criteria needs to be established at the beginning of a project. They also need to be close to what interested underwriting taxpayers are expecting and reflected in the project reporting. They may also need to be expressed in percentages of potential, instead of aggregate number. As when your emphasis aggregate numbers you can generate some very impressive but meaningless value that only reflect the massive size of projects and investments, while still having a trivial total impact. This is basically what the USAID MEL evaluation reflects. Instead, emphasis on percentage of the potential will give a better evaluation of project effectiveness. Also, there is a need to remember that most project are defined by the community they serve, and thus the evaluation needs to reflect community impact more than individual impact.

Allow me to illustrate with my favorite development concern. That is the nearly 40 years of over reliance on producer organizations to funnel assistance to smallholder farmers. While I agree producer organizations may be socially desirable, but they are also administratively cumbersome which quickly converts to overhead costs, that can more than consume the overall financial benefits so relying on the producer organizations will force smallholder farmers deeper into poverty. This saying nothing about the super inconvenience of consignment marketing in a highly cash-oriented society. Thus, the famers widely and wisely avoid them, so that they require continuous external facilitation and subsidies to exist, then collapse once external support end. Yet the smallholder agriculture development effort is totally committed to imposing them on the communities, and claiming they are the essential salvation for smallholders, as shown by the rhetoric accompanying any presentation that would lead one to believe anyone was foolish not to participate.

My approach is prior to my international agriculture class discussion on producer organizations, I always asked the students, many of whom were interested in international development with plans to join the Peace Corps, to list what they thought would be the minimum values for various business parameters associated with Producer Organizations and compared them to the best estimate I could obtain scrutinizing various reports with some simple computation. The results are in the webpage: https://agsci.colostate.edu/smallholderagriculture/request-for-information-basic-business-parameters/.

The expectation is for producer organizations to enroll well over 50% of the potential beneficiaries and they market through the organization 70% of their produce while side selling less that 10%. The actual result is only about 10 - 15% of the farmers participated, and even those will side-sell the bulk of their produce when possible, so the total impact on the community is a trivial >5%. Not what you would really want as a successful project.  Please take a few minutes and review the webpage and some of the linked pages and provide comment back to this forum. Are the:

  • Criteria I suggest correct for evaluating a project providing a business service to smallholders?
  • Criteria ever included in an evaluation of a business service for smallholders that you know of? I have never seen them used.
  • Students’ expectations for minimum values realistic from an underwriting perspective?
  • Actual values accurate from your experience?
  • If these target results were promoted at the beginning, would they allow an independent or even in-house evaluation to be more forthright in criticizing the results without jeopardizing future opportunities?
  • Would it have guided future projects to seek more effective means of assisting smallholder farmers out of poverty?

I feel the persistence in relying on producer organization represents a lack of or compromised independent evaluations. The limited acceptance by smallholders should have been identified decades ago, and the development effort moved on to identify and implement more effective options. This then represents donors with no sincere interest in assisting the plight of smallholder producers, but fully committed to imposing this horrible business model on them, that fortunately they are not gullible enough to fall for. The best you can say for the practice is it shows an easily publicized good intention without accomplishing anything. My bottom line on producer organizations is a real multi-decade, multi-billion-dollar scandal, with some possible serious liabilities for the cover-up.  Am I wrong? If there had been clear target as what constituted success vs. failure could this have been avoided?

Some additional webpages that go into more detail are:

https://agsci.colostate.edu/smallholderagriculture/perpetuating-cooperatives-deceptivedishonest-spin-reporting/

https://agsci.colostate.edu/smallholderagriculture/loss-of-competitive-advantage-areas-of-concern/

https://agsci.colostate.edu/smallholderagriculture/vulnerability-for-class-action-litigation-a-whistleblowers-brief/

Thank you.