Evaluating in complexity: What have we learned about designing and running evaluations

Judy Oakden presented this paper at the American Evaluation Association Conference in Atlanta, Georgia on 28 September 2016.

In 1998 Eoyang and Berkas’s wrote a seminal article, Evaluating Performance in Complex Adaptive Systems. They suggested several systems thinking and complexity approaches evaluators might use to evaluate performance in complex adaptive systems.

This workshop considerd the subsequent progress made in using complexity informed approaches in evaluation. Designed for evaluation practitioners, this workshop also provided ideas of ways to work in the future.  It suggested some useful approaches to navigate the characteristic behaviours of complex adaptive systems. Suggestions included:

  • evaluative criteria which act an ‘anchor point’ to hold the evaluation together
  • generic performance ratings to hold shifts in the system
  • pattern spotting approaches to allow for nuanced and shared judgments – considering the emerging context and competing values of stakeholders
  • considering different ways to communicate findings.
Download PDF

Collective Sensemaking – How to Plan Yours to Great Effect

With Irene Guijt, Judy Oakden presented this paper at the European Evaluation Society Conference in 2016.

Sensemaking is essential in evaluation design. It promotes deeper stakeholder engagement and can lead to better insights and more evaluation use. This paper discussed how to design a collective sensemaking process as part of M&E practice. It considered ways to help navigate values and needs between different stakeholders. We argued sensemaking can make evaluation more useful. The presentation addressed five questions:

  • What is ‘collective sensemaking’ in M&E?
  • What forms can sensemaking take?
  • What are the conditions for successful collective sensemaking?
  • What role can sensemaking play to responsibly navigate the values, needs and understandings of stakeholders?
  • Why is collective sensemaking not more prevalent? How can we strengthen this part of M&E practice?

Notable innovation with qualitative data collection methods and of analytical procedures in quantitative reasoning has occured in M&E. However, innovations in the analytical processes for M&E of mixed methods appear to be lagging behind.

Download PDF

What does good and bad procurement look like from a contracting and relationship management perspective?

Judy Oakden presented this paper at the International Year of Evaluation 2015 Commissioning Evaluation Seminar, hosted by the Ministry for Business, Innovation and Employment in Wellington.

Judy discussed the challenges of contracting and looked at whether current practices are “fit for function”.  This presentation targeted people with buying responsibilities. It discussed the need to set up contracts in ways that take account of complexity.

Download PDF