Management matters: exploring the link between management models and the use of evaluations

This is interesting. To make the roles of the evaluation manager meaningful in the evaluation process, a proper orientation, and clear roles and responsibilities matter. I see many of the discussants emphasizing the relevance of the evaluation manager throughout the process. At what points (intervals) the evaluation team needs to bring in the evaluation manager? This is critical. Again, the independence of the evaluation needs to be protected to get credible results. It would be interesting to bring in the evaluation manager at the beginning and end of every critical step. This can take the form of debriefing, to allow the evaluation manager to contribute. I have led and participated in an evaluation where the evaluation manager tried to get involved with dictating which participants to be sampled. Similarly, the evaluation manager also attempted to model the minds of the participants. These actions undermine the independence and the credibility of the evaluation.

Dear colleagues, thank you for the opportunity to reflect a bit further on this: indeed, management approaches influence the credibility and quality of evaluations. From where I stand, an evaluation manager that is available to their team and that knows the organization/donor well is an asset. 

Purpose of an evaluation manager: support the evaluation until publication; promote learning.

In addition to managing evaluations (recruit teams, set workplans, meet deadlines) the manager’s purpose is to guide and facilitate the evaluation process and to provide support continuously, from inception through ensuring organization or donor specific requirements are included (conditions for these to be met), to data collection by facilitating more delicate meetings or accessing documentation, to Quality Assurance. All these are crucial to the evaluation’s goal: to be useful and shared i.e. published. 

When support is unavailable, the onus is on the externally hired evaluation team to either meet the expectations or to fail to have their work accepted and published: this is not to hide or thwart the evaluation team’s work, but simply because the evaluation does not meet minimum standards. Hence for the evaluation manager to ensure an evaluation proceeds smoothly, establishing initial expectations is foundational (particularly at onboarding and inception phases).

Furthermore, evaluation teams require a mix of both evaluation and technical expertise. The evaluation manager serves as additional evaluative expertise. This is an opportunity for independent consultants and technical experts as Independent Evaluation Offices must value accountability (i.e. publishing the evaluation) vis-à-vis learning (which includes, among others, fostering national capacities particularly where these are weak): the presence of an evaluation manager helps align competencies and experiences in the evaluation team and helps guide younger/emergent evaluators.

Provided evaluation independence is maintained (as posited), in more complex evaluations, a more active involvement of the evaluation manager may be warranted. 

The key is to include the evaluation manager’s involvement in the design stage of the evaluation, so that, as mentioned above, expectations are clear on all sides. There are different logics for this. 

In addition to evaluation, the evaluation manager may also have specific technical expertise useful to complement the evaluation team: his/her active participation is therefore an asset, particularly when there may be budgetary or other limiting factors: disagreements between external consultants and internal teams, the manager's active participation may be considered a mitigating strategy. 

Or, for example in the case of a corporate evaluation commissioned to the independent office, the evaluation manager can contribute or even lead the evaluation. Having a team with a representative of the office of evaluations may help the intended audience (such as other units within the organization, other organizations, certain country offices) hear/ adopt evaluation findings better: active participation of the evaluation manager is a strategic choice. 

The degree of involvement is evaluation-specific, and a function of evaluation design and resources (financial, time, and HR).

Dear Ibtissem, I echo your thoughts: the quality and utility of an evaluation is greatly influenced by the professionalism, expertise, and practical experience of the designated evaluation management team. 

The role and responsibilities of the evaluation management team, as well as the extent of their involvement, are contingent upon the various stages or phases of the evaluation process. 

I would like to offer my contribution from the perspective of the inception phase. In the complex landscape of project evaluation, the inception phase serves as a compass, guiding the journey toward success. It marks the initiation of a collaborative, participatory and learning journey between an organization and independent consultants engaged in project evaluation. It is a crucial activity where clarity, expectations, and mutual understanding are established.The inception phase is a critical component of project evaluation. It’s not only a formality but a strategic investment in the success of collaboration. Organizing a rigorous inception phase helps maximize external consultants' contributions from day one and establish a strong foundation for a successful partnership. This is where success is mapped, and the course is charted. Starting off on the right foot leads to a more efficient evaluation and increases the likelihood of meeting evaluation objectives. This is where the evaluation management team and external evaluators agree on dos and don'ts by:

• reviewing the project's background and context
• clarifying evaluation objectives and scope
• setting expectations for stakeholders' engagement
• discussing approaches and methodologies
• discussing deliverables formats and timelines:
• clarifying reporting and communication protocols
• providing access to project documentation
• discussing ethical considerations and confidentiality
• establishing a feedback mechanism
• gauging and address individual needs
• ...

As an external evaluators, the collaboration with evaluation managers significantly improves the relevance and utility of evidence for decision-making processes. Their expertise, stakeholder engagement initiatives, adaptability, and quality assurance efforts guarantee that evaluations are carried out efficiently and provide actionable insights that guide decision-making. It requires an opened-mindset of the "evaluation demand" and the "evaluation supply". 

Thank you Ibtissem for raising this interesting discussion topic.

As external evaluator, my contribution is the following: I agree with you that “the quality of an evaluation isn’t solely shaped by the technical expertise of the independent evaluation team; and an effective evaluation management is also critical to its success”.  With the objective to ensure the validity and reliability (to mention few) of evaluation results, Independent Evaluation Offices within development agencies quite often establish evaluation frameworks that provide a structured approach to manage evaluations and assure consistency across different projects. 

To conduct effective evaluations IEOs commission external evaluators with cleat Terms of Reference (TOR) and also assign an evaluation manager from their office with a clear mandate. Now the core issue is: how should the independent evaluation team and the evaluation manager effectively shoulder their respective roles and responsibilities during the evaluation period? This is to guarantee the evaluation results are of high standard, credible, and dependable for the way forward and decision making. 

1.Involvementof evaluation manages: The involvement of evaluation managers across the evaluation phases, should be strategically calibrated as per their mandate/ roles and responsibilities provided by the IEOs/their office/. As such, the independence of evaluations can be preserved.  What is required from them is to effectively employ appropriate approaches to manage the evaluation process, which are indispensable for quality/high standard/credible evaluation results. In fact, it is very important that the roles and the responsibilities of the evaluation managers need to be clearly communicated with the external evaluation team from the outset, to avoid confusion (although, this communication is quite often overlooked.  In conclusion, I strongly believe that if the evaluation team and evaluation managers perform a good job as per their mandate, the evaluation results are usually of high quality. 

2. Role of evaluation managers: As noted in the above section, I believe that the role of an evaluation manager should not encompass active participation as a team member, to any degree regardless of their relevant technical expertise. Actually, to my understanding the role and responsibility of evaluation manager is not just supervising. There are diverse approaches developed by the IEOs that the evaluation manager should effectively employ, without which the findings of the external evaluation team cannot be of high standard (reference can be made to the diverse approaches developed by IEOs to manage evaluation).  I do not see significant advantage from encompassing active participation as a team member in the role of evaluation managers. The disadvantages include: (i) there could be conflict of interest/biases, as they come from the same organization; (ii) independent evaluation may not be preserved; and (iii) their active participation can lead to lack of focus on their very mandate, which can affect the quality of the evaluation results. 

3. Collaboration with evaluation managers: As an external evaluator my insight and experience in this regard is that there is often collaboration with evaluation managers and functions. The diverse approaches employed by the evaluation managers have been very instrumental to perform quality evaluation work. For sure, their participation has usually enhanced the relevance and utility of evidence for decision-making processes (the diverse approaches developed by IEOs to manage evaluations can be referred, which are documented in the IEOs. I strongly believed that collaboration with evaluation managers provided the managers effectively shoulder their roles and responsibilities is a must, and not a choice, for quality/credible evaluation results.
 

With best regards,

Hadera Gebru
Addis Ababa, Ethiopia

Hello Colleagues,

Greetings from Uruguay!

Thank you, Ibtissem, for bringing up this intriguing topic. Having experienced both sides of the evaluation process (commissioning evaluations for WFP and conducting as independent consultant for UNICEF and UNFPA), I completely agree that the Evaluation Manager plays a pivotal role and bears significant responsibility for ensuring the quality of the evaluation results, which ultimately determines their usefulness.

Building on what other colleagues have already mentioned, I'd like to offer a couple of additional points that haven't been raised yet on the EM support and its role on Evaluation:

Ideally, the EM shouldn't shoulder the entire burden alone. It's advantageous for them to be supported by at least one Evaluation Analyst. This team composition mimics the structure of an external evaluation team and facilitates smoother communication and coordination. Evaluation Analysts can handle bilateral meetings with data analysts or other external evaluation team members, allowing the EM to focus on overseeing the calendar, meeting deadlines, and making high-level decisions in consultation with the team leader.

Furthermore, it's important to acknowledge that when discussing evaluation independence, we often assume we're referring to external evaluations. However, certain evaluation approaches, (e.g  Developmental Evaluation), emphasize a more formative focus. In these cases, the Evaluation Manager's involvement as an integral part of the program being evaluated is essential. This approach fosters greater ownership and promotes internal learning within the organization.

Thanks and best regards,

Cristian

Dear Ibtisssem and colleagues,

I absolutely agree with Dr. Osman. I would like to add some comments.

My experience is as an evaluation manager responsible for project evaluations at the FAO Office in Uruguay.

I have found that it is important for the person responsible for the evaluation, after the evaluation team has been completed, to hold an initial meeting with the team to provide background information about the country (and the area or sector targeted by the project, if applicable)
during the execution of the project and at the time of carrying out the evaluation. Likewise, I have found that it is important to hold a meeting with the team at the end of the evaluation to receive preliminary conclusions and add "contextual" considerations that the evaluation manager
might understand were not considered or known.

We cannot forget that projects are not "laboratory" initiatives but rather interventions in living communities that are always in absolute relationship to the context that surrounds them. Therefore, project evaluations are not "laboratory evaluations" either, but rather they must be evaluations with a large proportion of common sense, a learning opportunity for all actors participating in the project, and an initiative that allows learning to be achieved for the project. time after the project.

I found that in projects and project evaluations there are always trade-offs on how or what to report between "absolutely objective and traceable charts and tables" of goals, outputs, activities, beneficiaries, and other concepts, and the more subtle understanding of what was the "impact achieved by the project implementation in actors behaviour, environment, and institutional strengthening and development opportunities".

Thanks for raising such an interesting topic!

Vicente Plata
Uruguay

 

Dear Ibtissem and Colleagues,
Here are my contribution to this discussion:

1. Involvement of evaluation managers: While preserving the independence of evaluations, how should the involvement of an evaluation manager be strategically calibrated across the evaluation phases?

Involving evaluation managers in the evaluation stages strategically is a matter that varies from one organization to another. It must be clear that the evaluation team is completely independent. The evaluation managers have no choice but to facilitate the work of the evaluation team, facilitate their access to the required documents, and conduct the necessary interviews, whether with the stakeholders concerned or the project's beneficiaries. Evaluation managers have an important role in reviewing the evaluation report with the evaluation team in terms of the standards set for the form and content of the report.

2. Role of evaluation managers: To what degree should the role of an evaluation manager encompass active participation as a team member, as opposed to just supervising the evaluation process? What are the advantages and disadvantages of each level of involvement?

In addition to the answers mentioned in the previous question, the evaluation manager should not be an active member of the evaluation team because he ultimately belongs to the organization, which is a disadvantage to the independence of the evaluation. Another disadvantage of his participation is that he may influence the evaluation team in one way or another and take the evaluation team in directions that are not neutral. The direct involvement of the evaluation manager may lead to unfair highlighting of the results/outcomes of the implementation and effectiveness of a certain activity/component of the project against any other activities/component of the project, simply because there is some negative/positive relationship with the project implementation team.

3. Collaboration with evaluation managers: As an external evaluator, can you share your insights and experiences regarding collaboration with evaluation managers and functions? Has their participation enhanced the relevance and utility of evidence for decision-making processes?

Yes, it is possible to share the ideas and experiences of the evaluation team with the evaluation manager and his functions during the discussion stage of the inception report which includes the evaluation methodology and detailed evaluation plan, as well as in the discussion stage of the evaluation report to clarify the evaluation team’s justifications for their recommendations and findings from the evaluation.

Thanks and  Best Regards,

Gebril
Gebril Mahjoub Osman, PhD

Cairo, Egypt

Dear Ibtissem,

Using the results of evaluation to guide policy formulation and strategy design, requires such results to be as sound as possible. As you point out, conduct of an evaluation and its successful management call for two different types of skill sets. Entrusting both tasks to a single individual has several disadvantages like the level of required competence in one of those fields, personal bias, etc.

As for your first question, competent management of evaluation is important in all types of evaluations. It is particularly invaluable in policy and strategy evaluation and in those cases where the evaluation can influence strategic decisions on implementing/changing or stopping projects or activities such as pre-project and "mid-term" evaluations.   

Effective management of evaluations entails facilitating the work of evaluators, ensuring the collection of relevant facts, and fostering dialogue to refine templates and methodologies. The manager's role, therefore, can be viewed as that of a facilitator, complementing the evaluation process.

On the other hand, a competent evaluator may need a big volume of relevant background information to ascertain the environmental consequences of a policy, strategy, operation or a field endeavour. A holistic view from a capable manager can guide and assist evaluators in their tasks, enriching the evaluation process.

Depending on their respective competence, interaction between a manager and an evaluator could broaden the horizon of each other resulting in better evaluation.

Cheers!

Lal.