Ram [user:field_middlename] Khanal

Ram Khanal

Advisor
Community of Evaluator (COE) Nepal
Nepal

I have been working as an expert in programme evaluation and management. I have carried out many projects and programme evaluation related to livelihoods enhancement, economic development, natural resources management, climate change, disaster risk reduction and sustainable development in Nepal and other South Asian countries. Currently, I am serving as a president of community of evaluators in Nepal and am also engaged in evaluation field building while working with the government of Nepal.

My contributions

  • Artificial intelligence in the context of evaluation

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      I have not used in evaluation per se but used for literature review and other purposes. I am aware this will be a useful tool but not sure whether the information available or used by chat gpt is adequately represent the context and information for the developing countries like Nepal. I do hope - colleagues who have used the tool will be in position to share their experience.

  • How are we progressing in SDG evaluation?

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Dear Emilia,

      Thanks for raising this important issue.

      1.      I have had some opportunities to be involved in the evaluation of some aspects of SDGs in Nepal. I find evaluating specific objectives or targets within the SDGs is a normal process and we can learn from the existing evaluation approaches. Evaluating sustainability at a higher order considering a space-time dimension of certain interventions during or at the end of the intervention may however be a challenging task. With my limited experience, I have encountered some challenges while evaluating SDG-related interventions. I feel it was good to view the scope of interventions at a higher level and consider the whole ecosystem of the selected interventions (as SDGs are intersectoral, collaborative and many more utopian terms), this theoretically appealing concept however posed some noted difficulties in evaluation. Some of them are narrow scope or project-based interventions (short term, thinly spread, too focused on activity delivery, isolated, weak MEAL systems and many more); traditional approaches and mindset of commissioning and managing evaluation (such as lack of system thinking approach & anticipatory views that intervention may lead in future); and operationalizing sustainability evaluation (such as weak intervention design, critical data gap, attribution gap and the limited time allocated to assess the condition for sustainability). In addition, there are substantive evaluation policy gaps, institutional failure and critical capacity gaps to assess sustainability at national and sub-national levels. In addition, I do not see much interest/enthusiasm to carry out ex-post evaluation of programmes even by large donors and development partners.

      2.      Carrying out SDG evaluation is still an emerging thinking for many organizations including UN agencies. As SDGs are still new at the sub-national level, demand from the sub-national level is so far low. 

      With best regards, 

      Ram Chandra Khanal 

  • Proving the value of agroecology for farmers and food systems: what methods and evidence do we have?

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Dear Ravi ji,

      Thanks for raising this important point. I have my two cents' input based on my limited experience in this area. 

      1. Agro-ecology is a multi-dimensional concept at different levels or scales so capturing the multifunctionality of agro-ecology for assessing the performance by developing a common framework may be challenging. Given its externalities/known–unknown/unknown-unknown interactions and functions, performance assessment by using quantitative techniques at the household or farm level will not be adequate. Participatory tools such as community scorecards, by using people’s observations/satisfaction can be useful. 
      2. Environmental, economic and social/institutional aspects are important to see the performance but also its likelihood of continuation of the results. The performance indicators may vary depending on the local context but I used some indicators such as the presence of pollinators, level of pest attack, dilatory diversity & food security, use of traditional/indigenous crops, diversity of plants/crops used for food & medicinal purpose, contribution in farm income, level of stress (such as climate risks) tolerance and so on.   

      Best regards,

      Ram Chandra Khanal 

  • Can visual tools help evaluators communicate and engage better?

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Dear all,

      A lot of colleagues have mentioned the usefulness of visual tools for better communication.

      I echo the usefulness of the approach. I have noted that the visual tools – the language with universal clarity - help to communicate (during evaluation and after evaluation) easily as the tools address language barriers with multiple cultural and literacy levels among the groups. This is more useful while engaging development beneficiaries. In my experience, people become more cooperative, understand the context and objective easily and help to create quick awareness, better response and cross-learning.

      But, these evaluations, in many cases, become a mechanical process with a long-written report (with so-called advanced English which is not generally understood by the many stakeholders) that serve the purpose of accountability but not for learning. In participatory evaluation, I have some experience using visual tools such as social–resource maps, Venn diagrams, mobility /historical maps and community score card for different purposes. The tools can be developed based on the needs/context but optimum use of the tools may provide better results.  

      With best regards,

      Ram

  • How to evaluate science, technology and innovation in a R4D context? New guidelines offer some solutions

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Thank you, Svetlana, for sharing the document and asking for our input. I have limited knowledge of CGIAR systems but I am sharing my observations based on my previous work on agriculture research and development programmes in the Asian context.

      Evaluation of Research and Development of agriculture and associated natural resources management interventions is not straightforward. Hence, I commend the team’s work in bringing important themes in a concise and actionable form. There are however some observations which might be useful for thinking/re-thinking to make it more inclusive and a decision-making tool for the stakeholders.

      I am just trying to limit myself to one question: i.e. ‘Do you think the Guidelines respond to the challenges of evaluating the quality of science and research in process and performance evaluations?’

      The guideline may focus more on the system perspectives and emergence: 

      The document has highlighted the changing context for evaluation in CGIAR. It has raised important issues related to the food security future with a mission to deliver science and innovation to transform food, land, and water systems in a climate crisis and also mentions transformative changes. There is still room to integrate these important components in the actual administration of the evaluation.

      The guideline may need to go beyond the technical driven to inclusive or beneficiary focus evaluation.

      The guideline mentioned some audiences and users (such as funder and implementing agencies) but there is little emphasis on communities who might also be an important stakeholder of research and innovation. There are many successful research and development activities (such as participatory plant breeding, participatory selection, participatory technology prioritization and selection) in which communities/ farmers are important stakeholders. It seems their role is a little missing in this guideline.

      The impact pathways of Research and development intervention are long and unpredictable so evaluation criteria or questions should embrace these aspects.

      Once the research outputs are generated (there will also be cases that the research may not generate the expected outputs), the technology diffusion process may take longer time due to substantial development lag and adoption process and this process may affect the realization of impact/benefits of the technology within the intervention period. This may also influence the sustainability aspects. Integration of these aspects could be a challenge in the research evaluation process.  

  • A lack of learning in the monitoring and evaluation of agriculture projects

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Dear Daniel and other Evalforward members,

      evaluation has been primarily developed and used mechanically and served mainly the tick mark purpose (donor accountability) rather than for learning and improvement. Now, we know the indicators and so-called ‘log frame’ become more or less redundant in the complex situations in which most agricultural projects are run.

      Please allow me to share my recent experience. I am part of a team to assess the contribution of budget support with a small TA (3 years intervention) by a donor to the government to implement a national agriculture development strategy in one of the South Asian countries. As an evaluator, I have noted the following issues during the evaluation process:

      a)       The ‘Budget Support’ is provided to the government treasury and not earmarked for the agriculture sector. In this case, there is a high possibility of fungibility. We do not know whether the sector received the fund they have the opportunity to have incremental work. And how to evaluate the contribution.

      b)      The funding contract contained ambitious and not-relevant targets: The programme has six targets with annual milestones to be fulfilled for getting the fund. These targets are not only ambitious for the 3 years intervention but they are also outside the scope of the agriculture ministry. For example, decreasing the stunting percentage and increase in the percentage of land ownership by women at the national level. This is not a direct intervention from the ministry of agriculture and there are many other main contributing responsibly to attain that in a long period. There were also inadequate coordination and collaboration mechanisms among the ministries and government agencies to get information/progress. In addition, there are no M & E systems to collect data from the sub-national level.

      c)       The governance structure has also been changed from a unitary to a federal structure. The three tiers of the government are functioning on their own without having proper coordination and reporting mechanisms. Institutions and policies are in the process of development where as there exists a serious capacity gap. In this case, it has been difficult for the ministry to collect data and compile reporting.

      In this context, the log frame is still there without revision and evaluators are asked to assess the contribution of the fund on those indicators/targets. Both implementing agencies and the donors are still trying to attribute the impact of the fund which is like ‘squeezing water from a stone’. Perhaps, a push is still a far way to make the M & E approach more contextual and useful.  

      Agree: ‘here we go again’ and ‘repeat’ unfortunately.  

      Ram Chandra Khanal, PhD
      Independent evaluator, Kathmandu, Nepal

       

  • The pervasive power of western evaluation culture: how and in what ways do you wrestle with ensuring evaluation is culturally appropriate and beneficial to those who legitimise development aid?

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Dear all,

      Thanks for sharing the illuminating experience. I have a minor input based on my experience.

      The value of culture in evaluation is less discussed and practiced discourse among the development researchers, professionals, academics and funders. Due to various reasons, cultural issues are less represented in the evaluation design and subsequent phases. While designing an undertaking evaluation, most of the inquiry and observation methods/tools do not consider the context -space and time and are mainly focused on the results and their associated indicators. This is more important when dealing with socio-developmental issues. The recent approach of using the Theory of Change, in principle, covers the wider spectrum of the context but an understanding of people and their practices (=culture) have not been an important part of the analysis. For example – in a group of people (let’s take the example of the ‘hill Bramin’ in Nepal) women do not generally say their husband’s name, do not shake hands (they have a different way of greetings when they meet people) and may not speak with a man frankly from outside. In addition, family roles of men and women are also determined by the social systems/culture on which they are used for generations which may be strange for people from the West. Some communities worship their god before they initiate project-related activities or complete the project tasks. There are cases where development interventions are designed without considering the cultural aspects (such as demolishing temples or sacred or religious places of certain communities to construct a road that directly affects their culture). These are some examples. In this case, an evaluator without understanding the local context and culture – s/he might understand different ways and overall evaluation findings might be different. I feel having a local expert (who knows the culture) and a respectful conversation with the communities are two strategies I have used in my evaluations.

      Have a lovely weekend.

      Best regards, 

      Ram

  • Neutrality-impartiality-independence. At which stage of the evaluation is each concept important?  

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Dear Malika,

      I could not agree more with the points you have raised. Independence and enough coverage of data/information are key for a credible evaluation. But the other side of the coin is that evaluators are generally not given adequate time/ days and full information (despite requests) to reach out to the stakeholders, beneficiaries, comparable groups and potentially negatively affected people (this is seriously lacking in most of the evaluations) during the evaluation. This makes evaluation processes are not accountable to the people for whom the interventions are planned.  

      Best regards,

      -----------------------------------------

      Ram Chandra Khanal

       

  • Making data collection meaningful and useful to farmers: what is your experience?

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Dear David,

      Thanks for bringing these important issues to this community of Practice. Based on my experience working in the developing countries, I have the following input for your reference.

      With best regards, 

      1. Striking a balance between depth and length of assessment: monitoring and assessment exercises based on interviews and farm surveys can put significant burden on respondents, for example diverting time that would be otherwise allocated to other activities. Respondent fatigue due to lengthy interviews/surveys can also result in lower quality of data collected, and therefore in lower reliability of results. On the other hand, a shorter assessment may result in a level of depth that is insufficient to design effective interventions.

      How can the burden on smallholder farmers be reduced during M&E assessments?

      I prefer to do/ am doing: 

      • Making objective oriented short questionnaires
      • Mostly close ended but also provision of sharing their views and perspectives
      • Interview in their own setting and preferred time
      • Making them feel they are also benefitted from this exercise
      • Create strong rapport (interpersonal skills) – (not mechanical but also speak on their personal issues)
      • Provide some present (this can be to their children)

      What are the best ways to incentivize farmers to take part in the survey (e.g. non-monetary incentives, participation in survey tailoring, in presentation of results)?

      I prefer to do / am doing: 

      • When I was programme/ project manager I used to provide farmers some financial compensation (I strongly feel we need to pay the information provider as we information collectors are making a good sum of money for similar kind of functions)
      • Compensate their time with good snacks / refreshment
      • Provide them a present as a ‘token of love’
      • Acknowledging their support

      2. Making findings from M&E assessments useful to farmers: considering the burden on farmers resulting from M&E exercises, it is key to ensure results are meaningful and accessible to them. This is in fact an explicit objective of the M&E tool we are developing. The assessment seeks to provide an indication of sustainability strengths and weaknesses that can be used by e.g. extension agents to help farmers identify targeted practices that can increase overall sustainability of production.

      Based on your experience, what could be the most effective ways to communicate results from the sustainability assessment to farmers (e.g. field visits and peer learning, technical information workshop)? What kind of communication materials (e.g. briefs, leaflets, others) are most appropriate to support knowledge sharing events?

      • Clarify the objectives – how sustainability assessment are important to farmers and their groups/
      • Organize sharing meeting and get their feedback on the result
      • Take full use of local resources persons/ local groups or trusted partners while sharing the results
      • Use illustration/visual aids/local language  
      • Less of use of technical words and complex terms

      Do you have experience in comparing results among farmers in a participatory way? What method have you used to do this? Was it effective?

      When possible:

      • Making sharing group according to the interest groups (such as women farmers, youth farmers, farmers groups based on their production or participation in different value chain)
      • Use illustration / examples / visual aids/ simple demonstration (such as big maize cub vs small cub) relevant to local context  (for example if you say 50%, in many cases – they do not understand, if you give example such as 100 unit and 150 unit (50% additional)- may be will be in position to understand

      How can the results be used for non-formal education of farmers (e.g. to raise awareness and/or build capacity on ways to increase farm sustainability)?

      • Develop participatory based farmer centred training module considering the need of the farmers
      • Provide sharing opportunity by farmers (farmer to farmer approach)
      • Use of Use illustration / examples / visual aids/ simple demonstration

      -----------------------------------------

      Ram Chandra Khanal (PhD)

      Evaluator and programme manager: Climate change/NRM/Agriculture

  • The farmer as a key participant of M&E: lessons and experiences from Participatory M&E systems

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Dear Emma,

      Thanks for posing this important question. Most times, M & E is considered as an accountability function of the donors and implementing agencies also follow the same approach. In this case, participation of farmers is not considered as prerequisite but sometime viewed that farmers are not ‘knowledge’ on these technical matters and only technical persons can provide this service on behalf of beneficiaries (as the technical persons also understand the local/farmers context and needs).

      Participatory M & E has emerged to rectify this challenge where farmers would be involved in all stages of M & E – from planning to final evaluation. There is however mix experience in real world situations. I have noticed three types of P M&E process while working for various development organizations. In first group, project P M & E process duly respect its basic premise of participatory principle and involved farmers in all or most of the project cycle (deep engagement). In the second project use more opportunistic approach. Farmers are invited just before or after the initiation of project (most times during the project inception phase) and share the M & E strategy/plan and claimed farmers’ involvement in M & E process. In my experience this is the most common approach applied in agriculture project in managing M & E (medium level engagement). The third category, where technocrats prepared the M & E and share with farmers to provide their feedback on their already prepared M & E strategy / plan (low engagement).  

      There are many participatory P M & E tools/methods that are dependent on the context and the technical matters. For example, for planning the project, ‘social and resource mapping’ would be very useful whereas for ‘agriculture market’ project ‘Venn diagram’ might provide some good understanding. Similarly, for food security project analysis, ‘seasonal calendar’ would provide very useful information. I have also used community score card to assess the performance (efficiency and effectiveness) of the project. There are many tools available, but it is vital to understand the basic principle of the participatory process and one should have a strong rapport with farmers and patients to use the tools.

      Best regards,

      Ram Chandra Khanal

  • Youth in agriculture: what lessons can we draw from evaluations?

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Hi Dorothy,

      I do not have an opportunity to have a systematic study on the issue your raised but based on my work experience, I have the following points to make.

      Subsistence farming with low productivity (both labour and land) is common in agriculture in most of the developing countries. Hence, most of the young population are leaving agriculture farming and are joining international labour force in Nepal. It is considered that agriculture farming is the profession of uneducated people who cannot make livelihoods from other sources. Agriculture universities are preparing ‘good graduates’ for international universities abroad and most of them remain do not come back. The policy-practice gap is huge and support provided by the government in the name of agriculture are also captured by elites and political influencers. This has created a perverse incentive to the young who want to start their own agri enterprises. But there are also some silver lines as well. Some young people who went abroad for employment come back with skills to use new technologies and management ability to run agri enterprises, especially in the urban and peri-urban areas. It may require a big shift in thinking and support from the government side and involve young entrepreneurs who can bring new innovation that helps to transform from subsistence farming to a profitable enterprise that can ensure their livelihoods. 

      Best regards,

      ---------------------------------------

      Ram Chandra Khanal, PhD

      Ex-President - Community of Evaluators - Nepal, BoDs - Nepal Krishi (agri) and board member SEF

      Kathmandu, Nepal

       

  • What can we do to improve the quality of development projects?

    Discussion

    • Ram Khanal

      Advisor Community of Evaluator (COE) Nepal
      Nepal

      Hi all,

      Thanks for initiating a dialogue.

      From my experience, one important aspect is to know whether the project is really relevant to beneficiaries both in short and longer term and how the intervention is going to contribute the specific development objectives. Readiness of the beneficiaries to collaborate is another critical factor.
      It is also important to have a comprehensive theory change as the intervention is influenced by other internal/external factors that are responsible for creating enabling or disabling environment. Sometime – we generate our logic and reasons (we basically create an island in most of the time) but that may not held true and our interventions do not yield results. So, in many cases, relevancy and weak design have been the main reason for a success and failure of an intervention. I am sure there also other issues while implementing the project.

      Best regards,
      ---------------------------------------

      Ram Chandra Khanal, *PhD*

      *President - Community of Evaluators - Nepal and BoDs - Nepal Krishi (agri)*

      Kathmandu, Nepal