Management matters: exploring the link between management models and the use of evaluations

Dear Colleagues/Members,

My EvalForward Contribution.

Beginning by initiating the important of the role of technical expertise and management of evaluation are both important and vital to the existence and delivery of quality evaluation and implementation programmes. However, both are essential to research institution of evaluation, to enhance work quality and data management. Expressly speaking, there is need for technical expertise and need for a continuous management in the vicinity of evaluation which are fundamental principles of evaluation. Both work hand in hand to strengthen the beauty and quality of evaluation in the concept of research, programme management and policy implementation. In other words, whatever that needs to be researched or might have been researched out there is method or methodology to that outcome and success. Both function to synergy and collaborate to foster the development of evaluation and data communities for enhancement and strengthening.

In furtherance to the concepts, does a supervisor(s) differs from manager? In a nutshell, this connotes the use of the word, often seen or perceived as external body or technical adviser that validates the team work or research work. It gives a different view to the fundamental principles on quality on technical ground or management. Majorly, as an advisory body that serves and initiates a design work, procedure, process or guide to work with or follow objectively. Invariably, the perception of this depends on the concept internally and externally that tracks the performance and input technically to evaluate the process and success of evaluation. The manager is more of effective team value that enhances the principle and objective of evaluation and programme. Again, manager serves as a team player in the workplace, the demand is in line with management goal as a support team for management decision and enforcement. Elucidating this concepts from both ends, externally and internally day to day activities of evaluators, it is more of managerial concept team work which is broader in nature than the latter that is technically outline. Diplomatically speaking, both leverage on the differences to succeed as one interchangeably entity.

Meanwhile, in my view, the roles of both might differ a bit but have one central focus point in evaluation management mechanism which is accurate data management, research and programme management.

A manager may be more of team work or player than a supervisor in this context but obviously they are same in delivery. Ensure that stakeholders’ concepts are well delivered and served because of team oriented.

Internally a manager participates in day to day activities of evaluation than supervisor who prioritize its validation of work done and create guideline.

A supervisor is more of a technical adviser or input than a manager who is more of welfare, team concern and continuity of programme management.  It all depends on the usage of the word and perspective.

A manager enhances the principle of accountability in workplace and continuity of better data and programme management. It is more of management incline than supervisor.

Manager might be broad in terms of knowledge ability in management community while that of supervisor might be streamline or restrain to a specific area or subject.

However, the view of both in evaluation management techniques and skills might differ a bit but the mind is one goal and result for the team, research and the programme.

It is a certain principle and mechanisms that organization, programme/project must be carried out and function this respect to decision making and implementation. There is always a synergy that builds collaboration on the ground of technicality and management objectively in programme. The main purpose is for both to be aware for the reason of the outcome that is basically decision based making and implementation in collaboration modalities and implementation. However, there is always room for both to calibrate in knowledge integration and management.

Dear Ibtissem,

Thank you for initiating such an enriching discussion. The insights shared resonate deeply with my current doctoral research, where I am exploring how evaluation results are utilized within agricultural R&D contexts. My focus lies in understanding how evidence from evaluations can guide agricultural R&D institutions towards greater impact.

I would like to contribute to this discussion by emphasizing the role of the evaluator as a key component in the process of utilizing evaluation results. The concept of the evaluator as a Knowledge Broker is particularly relevant, linking directly to the importance of starting evaluations with the utilization premise in mind. However, this alone is not sufficient; institutional engagement and strategic thinking about evaluations, integrated within the organizational context, are also crucial.

I have been working on some publications on this topic and am currently writing about a framework I've developed called AgroRadarEval. This framework incorporates eight dimensions and various variables derived from literature and qualitative data collection, aimed at maximizing the impact of agricultural R&D actions regarding the use of evaluation's results. For those interested in knowing more about AgroRadarEval, please visit: https://bit.ly/agroradareval_en. I would greatly appreciate any feedback or interaction on this framework.

This topic holds great interest for me, and I am keen to discuss it further in other venues and formats. Please feel free to contact me anytime as I am eager to contribute to and learn from ongoing dialogues in this field.

Best regards,

Daniela 

Dear Colleagues,

Thank you for sharing your thoughts. Your insights have highlighted key points about this topic. Our Evaluation Function is committed to continuing this discussion. Currently, we are conducting a study to identify the conditions under which different evaluation management approaches are likely to succeed and to explore collaboration strategies between independent offices to overcome management challenges. We will soon launch an online survey to map evaluation practices across independent evaluation offices and plan to bring this discussion topic to the upcoming European Evaluation Conference.

I enjoyed reading your contributions and I have summarized your key points here:

Lal Manavado, you emphasized the importance of effective evaluation management in facilitating evaluators' work, ensuring relevant data collection, and fostering collaboration. You also highlighted how managers can provide holistic guidance to enhance the evaluation process, especially when evaluators need extensive background information.

Gebril Mahjoub Osman, you underscored the necessity of preserving the independence of evaluation teams. Your argument against the active participation of evaluation managers, due to potential biases and conflicts of interest, suggests a role in facilitation and support rather than direct involvement.

Vicente Plata, you stressed the value of effective communication, such as initial and final meetings between evaluation managers and teams, to provide contextual insights and refine conclusions. Your point that evaluations must balance data with an understanding of the project's broader impact on the actors involved is well-taken.

Cristian Maneiro, you recommended that evaluation managers should be supported by evaluation analysts to manage workloads effectively. Your note on certain evaluation approaches, such as Developmental Evaluation, emphasizes a more formative focus. In these cases, the evaluation manager's involvement as an integral part of the program being evaluated is essential. This approach fosters greater ownership and promotes internal learning within the organization.

Hadera Gebru, you supported the idea that the roles and responsibilities of evaluation managers should be clearly defined and communicated. Your advocacy for strategic involvement to ensure high-quality and credible evaluation results, while preserving the independence of evaluators, is valuable in my point of view.

Adéléké Oguniyi, I agree with you on the importance of the inception phase as a foundation for successful evaluations. Your highlight on the need for clear communication, setting expectations, and collaborative planning between evaluation managers and external evaluators is key to the process.

Anne Clémence Owen, you discussed the dual role of evaluation managers in supporting the evaluation process and promoting learning. Your point that managers' involvement should be clearly defined from the design stage to ensure alignment with evaluation goals and organizational requirements is important.

Musa K. Sanoe, you noted the significance of proper orientation and clear role definitions for evaluation managers. Your emphasis on the need for strategic involvement of managers at critical steps to maintain the evaluation’s credibility and independence is very insightful.

In conclusion, these collective insights underscore the value of balancing the involvement and independence of evaluation managers. Thank you all for your valuable contributions to this important discussion. Please let me know if you would like to be involved further in this project. Don’t hesitate to write to me at: i.jouini@cgiar.org.

Best regards, 

Ibtissem

This is interesting. To make the roles of the evaluation manager meaningful in the evaluation process, a proper orientation, and clear roles and responsibilities matter. I see many of the discussants emphasizing the relevance of the evaluation manager throughout the process. At what points (intervals) the evaluation team needs to bring in the evaluation manager? This is critical. Again, the independence of the evaluation needs to be protected to get credible results. It would be interesting to bring in the evaluation manager at the beginning and end of every critical step. This can take the form of debriefing, to allow the evaluation manager to contribute. I have led and participated in an evaluation where the evaluation manager tried to get involved with dictating which participants to be sampled. Similarly, the evaluation manager also attempted to model the minds of the participants. These actions undermine the independence and the credibility of the evaluation.

Dear colleagues, thank you for the opportunity to reflect a bit further on this: indeed, management approaches influence the credibility and quality of evaluations. From where I stand, an evaluation manager that is available to their team and that knows the organization/donor well is an asset. 

Purpose of an evaluation manager: support the evaluation until publication; promote learning.

In addition to managing evaluations (recruit teams, set workplans, meet deadlines) the manager’s purpose is to guide and facilitate the evaluation process and to provide support continuously, from inception through ensuring organization or donor specific requirements are included (conditions for these to be met), to data collection by facilitating more delicate meetings or accessing documentation, to Quality Assurance. All these are crucial to the evaluation’s goal: to be useful and shared i.e. published. 

When support is unavailable, the onus is on the externally hired evaluation team to either meet the expectations or to fail to have their work accepted and published: this is not to hide or thwart the evaluation team’s work, but simply because the evaluation does not meet minimum standards. Hence for the evaluation manager to ensure an evaluation proceeds smoothly, establishing initial expectations is foundational (particularly at onboarding and inception phases).

Furthermore, evaluation teams require a mix of both evaluation and technical expertise. The evaluation manager serves as additional evaluative expertise. This is an opportunity for independent consultants and technical experts as Independent Evaluation Offices must value accountability (i.e. publishing the evaluation) vis-à-vis learning (which includes, among others, fostering national capacities particularly where these are weak): the presence of an evaluation manager helps align competencies and experiences in the evaluation team and helps guide younger/emergent evaluators.

Provided evaluation independence is maintained (as posited), in more complex evaluations, a more active involvement of the evaluation manager may be warranted. 

The key is to include the evaluation manager’s involvement in the design stage of the evaluation, so that, as mentioned above, expectations are clear on all sides. There are different logics for this. 

In addition to evaluation, the evaluation manager may also have specific technical expertise useful to complement the evaluation team: his/her active participation is therefore an asset, particularly when there may be budgetary or other limiting factors: disagreements between external consultants and internal teams, the manager's active participation may be considered a mitigating strategy. 

Or, for example in the case of a corporate evaluation commissioned to the independent office, the evaluation manager can contribute or even lead the evaluation. Having a team with a representative of the office of evaluations may help the intended audience (such as other units within the organization, other organizations, certain country offices) hear/ adopt evaluation findings better: active participation of the evaluation manager is a strategic choice. 

The degree of involvement is evaluation-specific, and a function of evaluation design and resources (financial, time, and HR).

Dear Ibtissem, I echo your thoughts: the quality and utility of an evaluation is greatly influenced by the professionalism, expertise, and practical experience of the designated evaluation management team. 

The role and responsibilities of the evaluation management team, as well as the extent of their involvement, are contingent upon the various stages or phases of the evaluation process. 

I would like to offer my contribution from the perspective of the inception phase. In the complex landscape of project evaluation, the inception phase serves as a compass, guiding the journey toward success. It marks the initiation of a collaborative, participatory and learning journey between an organization and independent consultants engaged in project evaluation. It is a crucial activity where clarity, expectations, and mutual understanding are established.The inception phase is a critical component of project evaluation. It’s not only a formality but a strategic investment in the success of collaboration. Organizing a rigorous inception phase helps maximize external consultants' contributions from day one and establish a strong foundation for a successful partnership. This is where success is mapped, and the course is charted. Starting off on the right foot leads to a more efficient evaluation and increases the likelihood of meeting evaluation objectives. This is where the evaluation management team and external evaluators agree on dos and don'ts by:

• reviewing the project's background and context
• clarifying evaluation objectives and scope
• setting expectations for stakeholders' engagement
• discussing approaches and methodologies
• discussing deliverables formats and timelines:
• clarifying reporting and communication protocols
• providing access to project documentation
• discussing ethical considerations and confidentiality
• establishing a feedback mechanism
• gauging and address individual needs
• ...

As an external evaluators, the collaboration with evaluation managers significantly improves the relevance and utility of evidence for decision-making processes. Their expertise, stakeholder engagement initiatives, adaptability, and quality assurance efforts guarantee that evaluations are carried out efficiently and provide actionable insights that guide decision-making. It requires an opened-mindset of the "evaluation demand" and the "evaluation supply". 

Thank you Ibtissem for raising this interesting discussion topic.

As external evaluator, my contribution is the following: I agree with you that “the quality of an evaluation isn’t solely shaped by the technical expertise of the independent evaluation team; and an effective evaluation management is also critical to its success”.  With the objective to ensure the validity and reliability (to mention few) of evaluation results, Independent Evaluation Offices within development agencies quite often establish evaluation frameworks that provide a structured approach to manage evaluations and assure consistency across different projects. 

To conduct effective evaluations IEOs commission external evaluators with cleat Terms of Reference (TOR) and also assign an evaluation manager from their office with a clear mandate. Now the core issue is: how should the independent evaluation team and the evaluation manager effectively shoulder their respective roles and responsibilities during the evaluation period? This is to guarantee the evaluation results are of high standard, credible, and dependable for the way forward and decision making. 

1.Involvementof evaluation manages: The involvement of evaluation managers across the evaluation phases, should be strategically calibrated as per their mandate/ roles and responsibilities provided by the IEOs/their office/. As such, the independence of evaluations can be preserved.  What is required from them is to effectively employ appropriate approaches to manage the evaluation process, which are indispensable for quality/high standard/credible evaluation results. In fact, it is very important that the roles and the responsibilities of the evaluation managers need to be clearly communicated with the external evaluation team from the outset, to avoid confusion (although, this communication is quite often overlooked.  In conclusion, I strongly believe that if the evaluation team and evaluation managers perform a good job as per their mandate, the evaluation results are usually of high quality. 

2. Role of evaluation managers: As noted in the above section, I believe that the role of an evaluation manager should not encompass active participation as a team member, to any degree regardless of their relevant technical expertise. Actually, to my understanding the role and responsibility of evaluation manager is not just supervising. There are diverse approaches developed by the IEOs that the evaluation manager should effectively employ, without which the findings of the external evaluation team cannot be of high standard (reference can be made to the diverse approaches developed by IEOs to manage evaluation).  I do not see significant advantage from encompassing active participation as a team member in the role of evaluation managers. The disadvantages include: (i) there could be conflict of interest/biases, as they come from the same organization; (ii) independent evaluation may not be preserved; and (iii) their active participation can lead to lack of focus on their very mandate, which can affect the quality of the evaluation results. 

3. Collaboration with evaluation managers: As an external evaluator my insight and experience in this regard is that there is often collaboration with evaluation managers and functions. The diverse approaches employed by the evaluation managers have been very instrumental to perform quality evaluation work. For sure, their participation has usually enhanced the relevance and utility of evidence for decision-making processes (the diverse approaches developed by IEOs to manage evaluations can be referred, which are documented in the IEOs. I strongly believed that collaboration with evaluation managers provided the managers effectively shoulder their roles and responsibilities is a must, and not a choice, for quality/credible evaluation results.
 

With best regards,

Hadera Gebru
Addis Ababa, Ethiopia

Hello Colleagues,

Greetings from Uruguay!

Thank you, Ibtissem, for bringing up this intriguing topic. Having experienced both sides of the evaluation process (commissioning evaluations for WFP and conducting as independent consultant for UNICEF and UNFPA), I completely agree that the Evaluation Manager plays a pivotal role and bears significant responsibility for ensuring the quality of the evaluation results, which ultimately determines their usefulness.

Building on what other colleagues have already mentioned, I'd like to offer a couple of additional points that haven't been raised yet on the EM support and its role on Evaluation:

Ideally, the EM shouldn't shoulder the entire burden alone. It's advantageous for them to be supported by at least one Evaluation Analyst. This team composition mimics the structure of an external evaluation team and facilitates smoother communication and coordination. Evaluation Analysts can handle bilateral meetings with data analysts or other external evaluation team members, allowing the EM to focus on overseeing the calendar, meeting deadlines, and making high-level decisions in consultation with the team leader.

Furthermore, it's important to acknowledge that when discussing evaluation independence, we often assume we're referring to external evaluations. However, certain evaluation approaches, (e.g  Developmental Evaluation), emphasize a more formative focus. In these cases, the Evaluation Manager's involvement as an integral part of the program being evaluated is essential. This approach fosters greater ownership and promotes internal learning within the organization.

Thanks and best regards,

Cristian

Dear Ibtisssem and colleagues,

I absolutely agree with Dr. Osman. I would like to add some comments.

My experience is as an evaluation manager responsible for project evaluations at the FAO Office in Uruguay.

I have found that it is important for the person responsible for the evaluation, after the evaluation team has been completed, to hold an initial meeting with the team to provide background information about the country (and the area or sector targeted by the project, if applicable)
during the execution of the project and at the time of carrying out the evaluation. Likewise, I have found that it is important to hold a meeting with the team at the end of the evaluation to receive preliminary conclusions and add "contextual" considerations that the evaluation manager
might understand were not considered or known.

We cannot forget that projects are not "laboratory" initiatives but rather interventions in living communities that are always in absolute relationship to the context that surrounds them. Therefore, project evaluations are not "laboratory evaluations" either, but rather they must be evaluations with a large proportion of common sense, a learning opportunity for all actors participating in the project, and an initiative that allows learning to be achieved for the project. time after the project.

I found that in projects and project evaluations there are always trade-offs on how or what to report between "absolutely objective and traceable charts and tables" of goals, outputs, activities, beneficiaries, and other concepts, and the more subtle understanding of what was the "impact achieved by the project implementation in actors behaviour, environment, and institutional strengthening and development opportunities".

Thanks for raising such an interesting topic!

Vicente Plata
Uruguay

 

Dear Ibtissem and Colleagues,
Here are my contribution to this discussion:

1. Involvement of evaluation managers: While preserving the independence of evaluations, how should the involvement of an evaluation manager be strategically calibrated across the evaluation phases?

Involving evaluation managers in the evaluation stages strategically is a matter that varies from one organization to another. It must be clear that the evaluation team is completely independent. The evaluation managers have no choice but to facilitate the work of the evaluation team, facilitate their access to the required documents, and conduct the necessary interviews, whether with the stakeholders concerned or the project's beneficiaries. Evaluation managers have an important role in reviewing the evaluation report with the evaluation team in terms of the standards set for the form and content of the report.

2. Role of evaluation managers: To what degree should the role of an evaluation manager encompass active participation as a team member, as opposed to just supervising the evaluation process? What are the advantages and disadvantages of each level of involvement?

In addition to the answers mentioned in the previous question, the evaluation manager should not be an active member of the evaluation team because he ultimately belongs to the organization, which is a disadvantage to the independence of the evaluation. Another disadvantage of his participation is that he may influence the evaluation team in one way or another and take the evaluation team in directions that are not neutral. The direct involvement of the evaluation manager may lead to unfair highlighting of the results/outcomes of the implementation and effectiveness of a certain activity/component of the project against any other activities/component of the project, simply because there is some negative/positive relationship with the project implementation team.

3. Collaboration with evaluation managers: As an external evaluator, can you share your insights and experiences regarding collaboration with evaluation managers and functions? Has their participation enhanced the relevance and utility of evidence for decision-making processes?

Yes, it is possible to share the ideas and experiences of the evaluation team with the evaluation manager and his functions during the discussion stage of the inception report which includes the evaluation methodology and detailed evaluation plan, as well as in the discussion stage of the evaluation report to clarify the evaluation team’s justifications for their recommendations and findings from the evaluation.

Thanks and  Best Regards,

Gebril
Gebril Mahjoub Osman, PhD

Cairo, Egypt

Dear Ibtissem,

Using the results of evaluation to guide policy formulation and strategy design, requires such results to be as sound as possible. As you point out, conduct of an evaluation and its successful management call for two different types of skill sets. Entrusting both tasks to a single individual has several disadvantages like the level of required competence in one of those fields, personal bias, etc.

As for your first question, competent management of evaluation is important in all types of evaluations. It is particularly invaluable in policy and strategy evaluation and in those cases where the evaluation can influence strategic decisions on implementing/changing or stopping projects or activities such as pre-project and "mid-term" evaluations.   

Effective management of evaluations entails facilitating the work of evaluators, ensuring the collection of relevant facts, and fostering dialogue to refine templates and methodologies. The manager's role, therefore, can be viewed as that of a facilitator, complementing the evaluation process.

On the other hand, a competent evaluator may need a big volume of relevant background information to ascertain the environmental consequences of a policy, strategy, operation or a field endeavour. A holistic view from a capable manager can guide and assist evaluators in their tasks, enriching the evaluation process.

Depending on their respective competence, interaction between a manager and an evaluator could broaden the horizon of each other resulting in better evaluation.

Cheers!

Lal.