Do you have examples on reviewing Theories of Change?

Do you have examples on reviewing Theories of Change?
17 contributions

Do you have examples on reviewing Theories of Change?

image
©Anna Maria Augustyn

Dear EvalForward Community,

As an evaluator, several times I got a task to perform a review of the Theory of Change / Intervention logic (or Logframe in the case of EU-funded projects). As I was browsing through the web in search of some interesting examples of how it was done in other evaluations, I did not find much in terms of guidance or best practice examples.

My approach to this task was to develop a set of guiding questions and focus on the ToC / Logframe content revision. I involved the main people dealing with the project implementation into a reflection process, as far as possible in a participatory manner.

I was wondering if you have some experiences to share in this respect.

  1. Do you know any good examples of guidelines to help in this task?
  2. Can you share some relevant reports or chapters which present findings of similar work? (if publicly permitted)
  3. What was the follow up of the ToC review: Were any significant changes implemented? Were there any barriers to introduce changes in the project?

I will be grateful for your insights.

Best wishes,

Anna Maria Augustyn

 

This discussion is now closed. Please contact info@evalforward.org for any further information.
  • Dear Anna Maria

    Here is a link to a 3 minute super simple guide we made to creating a theory of change (for UNEP training unit).

    I'd love to see what else you find as I'm sure this is something the FAO evaluation team I'm now working with would be interested in.

    All the best

    Harriet Matsaert

    Consultant FAO OED communications team

  • Hello Anna, thank you for sharing with me the documents and experiences that will be shared with you by colleagues. I will be happy to receive them.

    Sincerely

  • Thank you, Anna Maria, for facilitating this conversation on ToC.  I have worked with organizations that have been working for 20+ years and have periodically revised their vision, mission, and strategic goals; some have an unwritten understanding of the philosophical underpinnings with respect to their desired outcomes. Yet many of these mature organizations have never worked with an evaluator before, and many have not given enough consideration to their ToC.  I believe there are many cases where the evaluation of a social impact organization needs to begin by working with the client to define, organize, and visualize their ToC by integrating all their implicit and explicit ways concepts and frames -- the process needs to be collaborative and participatory, and it is a bit like a Foucauldian archaeology/anthropology of knowledge. 

  • Dear all,

    Theory of change (ToC) as well as log frame analysis are models to carry out framework assessment. While log frame is linear, ToC is a non linear human-centered model used for complex situations like Disasters and climate change as well.

    ToC can be use for simulation and prediction.

    Regards.

    Norbert

  • Dear Anna Maria,

    Hi and I think you're on the right lines - develop some questions with which to frame conversations with people who have been in involved in implementation. I would also add those who were involved in developing the ToC (who may be different folk).

    My own experience in reviewing ToC follows three broad themes each with an overarching question:

    • Inclusiveness of the approach/method - who was involved and how well and whose theory is it? For example, was it the donor advisors consulting with beneficiaries/clients or just the donor staff doing their own  research and toing and froing around different technical areas and then signed off by an internal QA unit. or .......? 
    • Robustness of the evidence - on what basis were the assumptions developed and the results - outputs, outcome and impact - arrived at - the pathways to change and the changes themselves ?; and 
    • Coherence and plausibility of the product - is the diagram/visual accompanied with a clear narrative explaining HOW the Action Theory (ie, activities and outputs) will stimulate WHAT change among WHOM and WHY ie, the Change Theory). 

    The look of the product will also vary and, in this regard, Silva makes a good point, though i wouldn't profess to have the software skills to produce the second diagram, nor the intellectual capacity to understand it!!! The action and change theories rarely follow a linear trajectory, but there is no right or wrong. A key difference is in how the product makes clear the consequences for the monitoring and learning process. If it's building a bridge, then you simply engineer a set process to hell from beginning to end and monitor this accordingly. However, if its to do with programme outputs striving to stimulate changes in the behaviours and relationships among people - or outcomes - then this has obvious implications on monitoring: the assumptions made about how and why they will respond to outputs - matter as much as, if not more than, the outcome indicators.  

    Depending on who funds the work you are doing, each donor has a slightly different take on/guidelines for for  ToC (and LFs). I developed some for reviewing the content of log frames and have attached them. Hope they are of some help.  

    On log frames.......The methodology used for developing what people call a ToC is not so different to how some, like GTZ (now GIZ), develop Logframes. See here. I think this is the best method i have seen, thus strongly recommend it as reference in assessing the quality of the process and, in many ways, the product. Its essence is well captured by Harriet Maria's neat video. Claims as to how ToC take better account than LFs of complexity and with its emphasis on assumptions and better explaining the why's and how's of change ring somewhat hollow.   

    As with many methods and tools, there is nothing i believe to be intrinsic to LFs that encouraged many donor agencies to either ignore or mis-use the method and arrive at a product that is too simplistic and  deemed as not fit for purpose. Given they did, it didn't surprise me that they moved to ToC......!

    I hope some of this helps and good luck. Please do get back to me if you want to talk anything through.

    Best wishes,

    Daniel  

     

     

  • Emiel Buffel

    Emiel Buffel

    Evaluation Analyst FAO

    Dear EvalForward Members,

    Hivos ToC Guidelines demonstrate how the ToC should incorporate all aspects of change interventions, including social processes where individual actions continuously interact with those of others.

    Hivos developed eight steps to think critically while designing a ToC. Of particular interest might be Step 8, which lists some core suggestions on how to embed the ToC in the project process. To learn effectively from the ToC, the authors recommend regular reflection sessions with staff and relevant stakeholders to discuss activities, experiences and results.


    These participatory sessions, where pathways, assumptions and strategies have to be revisited, combined with appropriate data collection, will align regular management processes with the ToC.
    In addition, these ToC design sessions should acknowledge power relations and involve appropriate leadership roles. The ToC should also be transparent and visible in the daily life of the team and project participants.

    Please find here the useful report of Hivos, shared on EvalForward’s resources section: https://www.evalforward.org/resources/theory-change-thinking-practice

    Best regards, Emiel

  • Dear Anna Maria,

    The review of the theory of change (ToC) in the implementation of a development project is based on a participatory mid-term evaluation of the project. This evaluation should involve all project stakeholders. It is done on the basis of quality criteria according to which it is decided to review the intervention strategy. For example, the relevance of the project, its effectiveness and the participation of stakeholders should be assessed. The ToC review will be a readjustment of problematic elements of the logical framework, including the intervention logic and important assumptions. The major blockages to project success often come from these two categories of logframe elements.

    Thank you. 

    Dr Emile N. HOUNGBO

    Agroeconomist, Teacher-Researcher

    Expert in Elaboration and monitoring-evaluation of development projects

  • Dear EvalForward Community,

    Many thanks for the contributors to discussion so far: Tom Archibald, Brian Belcher, Harriet Maria Matsaer, Hayat Askar, Moussa Coulibaly, Silva Ferretti, Seda Kojoyan, Nelson Godfried Agyemang, Nasser Samih Quadous and Alan Ferguson.

    I’m happy to see many interesting experiences and insights into the topic of ToC review. Thank you also for the useful document resources, video and links.

    The following main points emerged from the discussion:

    1.    There is a shared understanding that the ToC review is a beneficial exercise for the projects as it helps to better capture the underlying assumptions and identify the rationale behind success or failure of the specific interventions. Organizations use those reviews as a learning tool that can help to improve the project implementation or designing the follow up projects. 

    2.    As the projects and programmes operate within the complex systems, similar is with the ToC. There is a visible challenge in adapting the linear approach for measuring the progress with the systems thinking that better captures this complexity. Against this background, the evaluation policies of the donors usually tend to prefer the more focused and fragmented picture of the project implementation.

    3.    The ToC reviews are often performed within the projects / programmes evaluation, however there are some practical issues with implementing the suggested changes. As the Logframe or other frameworks supporting the project implementation are rather fixed at the project’s outset, it’s difficult to introduce changes in its course. 

    Concerning my last point, I’ve another issue to consider: 

    Are you familiar with any evaluation cases where the ToC review resulted in changing the expected outcome or impact indicators of a project? How difficult was to introduce those changes?

    I will welcome your comments and any further links or documents.

    Best wishes,
    Anna

  • Anna - It's not always easy to convince project managers that they need to develop a TOC mid-course since the original logframe or TOC is typically forgotten once a project commences. There is a case for being strategic and selective in your approach. The attitude, as one manager stated, is "if we do a good job implementing the outputs, the results will take care of themselves". Major design 'repair' is viewed as disruptive and therefore resisted.  The process may be more critical than the methodology. Re-orienting a project around a clear TOC can be difficult unless you focus on particular elements where the participants recognize a design problem. For example, in  a recent evaluation we recommended a tailored review of certain structural issues focusing on clarifying the key drivers, agents, linkages and pathways to end results:

    "It is very common for projects to have a mid-course internal review and update of their strategy, with a special focus on the pathways to measurable outcomes. There are several potential approaches. For example, this could involve having the project implementation partners identify (a) change drivers and agents for the immediate outcomes associated with the main project components, (b) the linkages between components that facilitate these drivers and agents, and (c) the subsequent joint pathways to well-defined, measurable results at the regional, country and local levels. ………While the logframe effectively outlines the results hierarchy (‘the what’), more emphasis on the main theory of change pathways (‘the how’) would be useful in the forthcoming second phase."

  • Dear EvalForward team,

    Yes, reconstruction of the theory of change is the first step we follow in designing an evaluation. I did it in all evaluations I participated in the last 5 years. The review and reconstruction is participatory with the program/ project team. Usually many projects have issues in their design of intervention logic. The preparation of evaluation tools and agreeing on questions needs agreeing with the implementation team and later minimizing any misunderstanding or lack of information. The new Theory of change is presented in the inception report as well as in the final report.

    Regards,

    Naser Qadous
    Palestinian Evaluation Association 
    Ramallah

  • In reality TOC is 'Theory of Flux' TOF. This is what we have found in Pan-African Consulting Consortium and Institute (PACCI https://pacciafrica.wixsite.com/pacci-net). All life is in a Flux with uncertainty, complexity and unrealized predictability, making change unpredictable needing an attitude of utmost flexibility, in order to address in an iteratively continuous manner multidimensional and multidirectional emerging issues, literally thinking and acting on your feet.

    Nelson Godfried Agyemang 

    President, Pan-African Consulting Consortium and Institute (PACCI)

    Akyeremade Ashanti Ghana

  • I wanted to express agreement with Silva Ferretti below, and also thank others for sharing their guidance notes and documents!

  • My two cents.

    When a "theory of change" (as illustrated in the manual) looks like this - with arrows going in the same direction,  and with a linear outline (IF... THEN),  it is just a logframe in disguise.

    logframe

    A proper "theory of change", fit for complex setups, will have arrows going in many diverse directions, interlinking ideas in ways that are hard to disentangle.

    It is messier, harder to navigate, but... ehy! This is what reality most often looks like.  (this is the "obesity map", in case you wonder)

    TOC

    There is a WORLD OF DIFFERENCE amongst the two.

    You cannot really approach them in the same way and with the same thinking.

    This should be the starting point of any analysis!

    Once you understand if you are dealing with a linear or complex theory of change, then you need to remember

    • In some cases linear thinking has a reason to be.
    • when addressing social change, most often not.

     

    I feel that it is quite unfortunate that the "theory of change" idea - born to appreciate complexity, ended up just being a different way to picture logframe thinking.

    At least we should be able to distinguish what is a logframe on steroids and what is appreciating complexity, and move on from there.

  • Dear Mrs. Hayat,

    Thank you so much for sharing this document. It will be useful for someone (including myself).

    Best,

    Moussa

  • Hi Anna-Maria,

    Attached is a useful TOC guide we used at school. 

    Also, The USAID Monitoring and Evaluation and Learning Activity in Jordan created a full training package on facilitating a TOC exercise:

    https://jordankmportal.com/collections/toc-training-materials

    Western Michigan University has some useful checklists on specific areas on evaluation. You may refer to these if you are looking into guidance on a specific component / detail of the TOC and even in the evaluation process that follows. Here is the link:

    https://wmich.edu/evaluation/checklists

    Regards,

    Hayat

  • Dear Anna Maria,

    My colleagues and I have done quite a lot of work developing and using ToC in research evaluation. While research evaluation has some unique characteristics, many of the principles are more generally applicable. We have some resources (Resources - Sustainability Research Effectiveness) on the program website that may be useful to you. The “additional resources” (Additional Resources - Sustainability Research Effectiveness) may be most relevant for your purposes.

    All the best,

    Brian Belcher

     

  • Hi Anna Maria,

    This is a great topic and set of questions! I, too, find ToC review to be a common (and highly useful) task, yet one which would benefit from more explicit guidance and attention. In my work, I have connected it to the broader notion of evaluative thinking--collaborative ToC creation and review happens in a mutually-beneficial and reinforcing feedback loop with evaluative thinking.

    In terms of resources, one paper on this process, with some specific steps/guidelines suggested, is attached and also available via the USAID LearningLab here

    It describes work supported by Catholic Relief Services to do exactly what you are referring to. I hope this is helpful!

    Best,

    Tom Archibald

    https://sites.google.com/vt.edu/tgarch