Monitoring and Evaluation
Collective Insights design Monitoring and Evaluation frameworks which support learning for improvement, decision making and accountability. We take a practical approach which ensures frameworks can be feasibly implemented in light of staff and partner time, resources and capacity.
We have designed tailored Monitoring and Evaluation frameworks and tools for large and small organisations in collaboration with program users, and are experienced in designing measures of success and participatory data collection and analysis processes that enable stakeholders to assess and report on the effectiveness of their programs. We also undertake Monitoring and Evaluation reviews and assist clients to strengthen their Monitoring and Evaluation practices.
Collective Insights has designed Monitoring and Evaluation frameworks for the Australian government, the UN and NGOs. Some examples of our work in Monitoring and Evaluation include:
- Design of a Monitoring and Evaluation framework for the Business Partnerships Platform for the Australia Government (contracted by Strategic Development Associates).
- Design of a Monitoring evaluation for the Australian Civilian Corps – a whole-of-government initiative (contracted by Strategic Development Associates)
- Design of a Monitoring and Evaluation framework for the Australian Humanitarian Partnership for the Australian Government (contracted by Strategic Development Associates).
- Design of a Monitoring and Evaluation framework for Anglican Oversees Aid’s international programs.
- Review of the monitoring, evaluation and performance systems of Amnesty International UK’s Community Organising Unit, Human Rights Education Unit and Events and Artist Liaison Unit.
Value for Money
Collective Insights has led exploratory Value for Money work within Australia’s aid sector. We take a practical and integral approach and works with teams to integrate Value for Money considerations into program designs, Monitoring and Evaluation frameworks and assessment processes.
Collective Insights has assessed Value for Money as part of wider evaluations and as standalone assessment processes. We have conducted Value for Money assessments for Oxfam Australia, Amnesty International and the Australian Red Cross.
Collective Insights’ director Michelle Besley developed a new Value for Money Assessment Tool for Oxfam Australia. The tool applied Value for Money theory to enable a practical assessment that seeks to enhance program quality. In developing the tool, Michelle worked with three program teams to facilitate reflection on the Value for Money of their programs, and conducted three independent Value for Money assessments. Michelle was a co-recipient of the 2016 Australasian Evaluation Society Indigenous Evaluation Award for her Value for Money Assessment of the Oxfam Australia Aboriginal and Torres Strait Islander Peoples’ Program which used this new tool.
More About the Tool
The Value for Money assessment tool assesses the core components of Value for Money – Economy, Efficiency, Effectiveness and Equity. It was developed iteratively as three assessments were undertaken by testing its application in different ways and in different contexts (in South Africa, Australia and Sri Lanka). Assessment criteria were developed which were common across all programs.
The three Value for Money assessments have been published and can be downloaded here. The publications outline the assessment methodology and theoretical underpinnings of the tool. Each publication features an Executive Summary and full Value for Money assessment which outlines key learnings and recommendations for improving the program’s Value for Money, and for strengthening Monitoring and Evaluation.
One version of the tool features a rating system which was used for programs nearing completion with donor imperatives to formally assess Value for Money. The other version of the tool does not feature ratings, and supports teams to get a sense of how a program is tracking in relation to Value for Money and how it can be strengthened. Both versions of the tool have been designed to draw out learnings and develop recommendations for improving Value for Money.
The tool aims to strengthen the assessment of costs and investment, which is often lacking in program review, and help teams to better consider cost and investment options and make evidence-based Value for Money decisions. The tool recognises that assessment of Value for Money can only be made if appropriate data across the core components of Value for Money is available. It has been designed to help teams to strengthen their Monitoring, Evaluation and Learning frameworks to collect the necessary information to inform on Value for Money, and better manage programs to achieve Value for Money.
Theory of Change
Collective Insights designs and facilitates Theory of Change / Program Logic processes with program teams to help them articulate how a program expects to achieve change and draw out the assumptions, risks and preconditions for success underpinning a program’s theory.
Collective Insights has developed Theories of Change as part of design processes and in the development of Monitoring and Evaluation frameworks. We commonly design theories of change retrospectively when conducting evaluations when a program logic has not previously been developed to test the validity of the program’s theory in the evaluation process.
Collective Insights has designed Theories of Change / Program Logics for the Australian government, the UN and NGOs. Some examples include:
- The Australian Humanitarian Partnership investment design for the Australian Government (contracted by Strategic Development Associates)
- The ‘Free Prior and Informed Consent’ Program managed by Oxfam Australia’s Mining Team
- UNRWA’s Child and Family Protection Project for Palestine refugees in Marka camp for in Jordan
- World Vision’s Social Accountability Program in Madang Province in Papua New Guinea
- The Australian Government’s Leadership for Development Program (contracted by Pamodzi Consulting)
Collective Insights undertakes design of rights-based inclusive programs. We have a strong understanding of disability inclusion, gender and child protection. We can provide specialist technical support as part of design teams, or develop concepts, proposals and strategies.
Prior to becoming a consultant, Collective Insights’ director Michelle Besley worked for UNDP in Afghanistan to support the design and monitoring and evaluation of disability specific and mainstreaming programs in the areas of health, education, livelihoods and advocacy. She subsequently worked for Oxfam Australia’s Program Quality Unit where she led the design and implementation of Oxfam’s Disability Inclusion and Child Protection policies and implementation strategies. Michelle commonly integrates issues of equity and protection into the design of Theories of Change and Monitoring and Evaluation frameworks. She regularly assesses disability inclusion, gender and child protection as part of evaluation.
Collective Insights has provided technical support in program design to government, NGOs and contractors. Some examples of our work in rights-based inclusive design include:
- The design of a Disability Inclusion Strategy for the Lao Australia Development Learning Facility managed by Adam Smith International
- Facilitation of Disability Inclusion training to assist Oxfam Great Britain staff based in Jordan working in Syrian refugee camps to make their programs disability inclusive
- Member of the scoping team to develop a concept note for the Disability Inclusive Development through Sports Program in the Pacific managed by the Australian Sports Commission and DFAT (contracted by Pamodzi Consulting)
- Provision of technical support to an Afghan NGO to develop a full project proposal to the EC to deliver physiotherapy training across Afghanistan including in remote areas
Collective Insights conducts independent evaluations for NGOs, UN and government. We have experience leading evaluation teams, designing methodologies, using participatory methods and approaches, and carrying out both process-focused and impact evaluations.
Collective Insights has designed tailored methodologies to evaluate a wide range of programs including in the areas of: governance; health; education; livelihoods; crisis; protection; women’s empowerment; leadership; disability inclusion; capacity building; and active citizenship and advocacy. We have experience using different approaches including: Participatory Learning and Action; Most Significant Change; Outcome Mapping; Contribution Analysis; Process Tracing; and Value for Money.
We have led evaluation teams and carried out evaluations in Australia, Afghanistan, Jordan, Egypt, India, Sri Lanka, Cambodia, Lao PDR, Vietnam, Papua New Guinea and South Africa. Some examples of evaluations we have led include:
- Project evaluations of the NT PHN After Hours Program in the Northern Territory including projects in Darwin, Tennent Creek, Elcho Island and Alice Springs
- Amnesty International’s Crisis and Transition Project in Egypt
- CBM’s Disability Mainstreaming Project in India
- Oxfam’s Mekong Regional Water Governance Program and Inclusion Project
- The Vietnam Australia Vision Support Program, implemented by Vision 2020 consortia members
- Oxfam’s Australian Community Rehabilitation Program (ACRP Phase3) in Sri Lanka
Collective Insights facilitates program workshops to assist organisations to reflect on and improve aspects of their development practice. They have run strategy, planning and reflection workshops which have supported program staff and senior management to discuss their programs and make decisions.
As a development practitioner, Collective Insights director Michelle Besley draws on her experience of working in-country on the ground on long term programs, and of working in Head Offices with teams managing programs and donor relationships to facilitate program effectiveness processes. She has developed a strong understanding of program quality, organisational development and development effective principles which she brings to facilitation.
Some examples of our work in the area of development effectiveness include:
- DFAT accreditation reviews; Collective Insights director Michelle Besley is a development specialist and participates in DFAT accreditation reviews which assess an organisation’s program management systems, development philosophies and approaches to partnership and development collaboration
- Facilitation of a program visioning and strategy workshop for Anglican Overseas Aid (Anglicord) with program staff and board members
- Facilitation of a reflection and redesign workshop for the Oxfam Sri Lanka country program for all staff including field staff and senior management