News

Back to Blog

Focussed evaluation skills development helps do more with less

Public sector agencies and NGOs have new challenges to show how effective their programmes are. These organizations must show who benefits from their programmes and in what context. Where an evaluation has yet to be done, resource constraints mean teams may need fast methods to collect data, analyse it, and draw evaluative conclusions.Judy Oakden, the Director of Pragmatica, has developed a workshop suitable for public sector agencies and NGOs to help programme teams draw findings from the data ...

Inspiring farm community led environmental action

Many New Zealand farmers are keen to understand better the environmental challenges they are experiencing on their farms and be able to respond to them. Supported by Thriving Southland through the Change and Innovation Project, many Southland farmers are highly engaged in accessing and interpreting new, relevant science and making the necessary changes to support thriving farming businesses.Pragmatica recently prepared a further case study on the work of Thriving Southland highlighting the ongoi...

Growing evaluation capability through mentoring

Pragmatica Director Judy Oakden has recently started leading a 2023-2024 mentoring group focussing on complexity in evaluation as part of the mentoring programme for Australian Evaluation Society (AES) members.  The AES Group Mentoring Program is an innovative online mentoring programme in which seasoned evaluators guide small groups of emerging and mid-career evaluators through their professional journey. The mentoring program is offered to members of the Australian Evaluation Society...

New perspective on public sector contracting

Many managers in the public sector are being asked to commission and contract in ways that are more flexible. In doing so, they can find themselves on the margins. Pragmatica recently facilitated training for public servants and evaluators. We explored why some of the contracting systems and processes that we have work in the way they do.By putting on a different pair of glasses “complexity lenses” we can saw aspects of contracting differently.Using a complexity lens we generated altern...

Extending opportunities to improve children's reading achievement in Pacific communities

Pragmatica was pleased to be asked to prepare a case study to show how an innovative reading programme has extended its reach in partnership with churches within the Samoan community.The reading programme, Reading Together® Te Pānui Ngātahi, is a research-based and cost-effective workshop programme for parents, whānau and aiga to support their children's reading at home more effectively.Originally, the Reading Together® programme design focused on delivery into school settings. In 2018...

The difference that makes a difference

Farming faces a time of major change in the external expectations about how land managers run their businesses. In response, farmers want to understand better and respond to the environmental challenges they are experiencing on their farms. Despite regulatory uncertainty, many are keen to access and interpret new, relevant science.Thriving Southland is a community-led, cross-sectoral group of farmers established in December 2020 with a vision to create a prosperous Southland, healthy people, hea...

Maintaining learning momentum within Pacific communities during COVID-19 lockdown

Te Tāhuhu o te Mātauranga | Ministry of Education recently published a series of case studies prepared by Pragmatica showing the different and innovative ways education and learning was maintained within the Pacific community during the COVID-19 lockdown.The initiatives were based around Talanoa Ako, an education programme for Pacific parents, families, and communities. The programme supports them to build knowledge and confidence, enabling them to champion their children's education...

Supporting parents and whānau to effectively raise their children’s reading achievement

Children from Aotearoa New Zealand are falling behind other countries in literacy measures. There is an ongoing lack of equity in student outcomes, particularly for Māori and Pacific children. Doing nothing is not an option, schools need effective ways to address the decline in reading.The Reading Together® programme, Te Pānui Ngātahi, (Reading Together®) is a research-based, four-session workshop programme that supports parents and whānau to effectively raise their children’s reading ac...

Exploring what COVID-19 means for evaluation

While COVID-19 has significantly disrupted our lives, a closer inspection shows we are not all impacted in the same ways. For example, some groups have experienced enormous financial and social impacts, while others have not.  Recently, practitioners in economic development and evaluation made sense of the fast-shifting contexts they find themselves in. The two events I attended were: the Economic Development New Zealand (EDNZ) online conference and the Australian Evaluation Society’s onl...

A complexity view of contracting public health services

Public health and social services are often hard to specify, complex to deliver and challenging to measure. This research uses a complexity theory-informed lens to explore the challenges and opportunities of contracting out for public health and social services in Aotearoa New Zealand. This qualitative study considers the implications of complexity concepts with ten public sector managers experienced in contracting out for public health and social services. Our newly published findings show that...

Videoconferencing: the new normal for getting work done

Working online and working at home are not new to us, and probably aren’t to you either. But suddenly we find ourselves doing it more, and under different circumstances. For example, meetings, workshops and interviews that a couple of weeks ago we’d have planned to hold face-to-face, are now moving to videoconference. In these new situations, we’re finding that we need to work a bit differently. We’re learning that with the right technology, skilled facilitation and some adaptation,...

Envirolink Evaluation completed

The Ministry of Business, Innovation and Employment (MBIE) commissioned Pragmatica to review the Envirolink Fund. The review aimed to inform MBIE of how well the Envirolink Fund operates and whether it achieved its intended outcomes and provided value for money.Pragmatica found that:“Envirolink makes a worthwhile and valuable contribution in the way it supports select regional councils to engage with and use environmental science research and technology. According to respondents, Envirolink in...

Rubrics — a tool for unboxing evaluative reasoning

Judy Oakden presented a session about rubrics with Kinnect Group colleagues at the 2019 Australian Evaluation Conference. Rubrics are an intuitive way of implementing the evaluation-specific methodology. Rubrics can help to unbox, demystify and democratise evaluative reasoning. In addition, rubrics help facilitate a clear, shared understanding of how quality, value and effectiveness are defined. Kinnect Group members Judy Oakden, Julian King, Nan Wehipeihana, Kate McKegg, Adrian Field, and colle...

Evaluation Building Blocks: A guide

With colleagues from Kinnect Group we published an e-book on the use of rubrics in evaluation. Download your copy here The Kinnect Group developed the ideas in this book over ten years of collaboration. In the introduction they observed: “There are many other guides to evaluation and different ways to do it. This approach emphasises the place of evaluative reasoning in evaluation. It draws on the work of many evaluation theorists, as well as our practice-based body of knowledge. We find that ...

Evaluation chapter in Social Science Research in NZ

Julian King and I recently published a chapter about evaluation in Martin Tolich and Carl Davidson’s introductory social research textbook, Social science research in New Zealand: An introduction. For more information please click here...

“Making sense of the mess” Adaptive Action Lab

Change is quickening for individuals, communities, and institutions around the world. Often change brings excitement and opportunity, but sometimes it challenges us. How do we adapt to that change and cope with the messiness that results? Judy Oakden recently collaborated with Glenda Eoyang, founder of the Human Systems Dynamics Institute to run this virtual Adaptive Action Lab. The Adaptive Action Lab was a three-session online training programme. Most of the participants were from New Zealand...

Applying complexity theory to contracting out

As part of the early stages of her Master of Philosophy, Judy Oakden was encouraged by her supervisors to attend the Operations Research Society Conference, in Lancaster UK, in September 2018.  As this was the 60th Conference of the Operations Research Society, the organising committee worked hard to attract a wide range of systems thinkers from all over the world. The conference was well supported, and many of the leading systems thinkers, including Gerald Midgely, Mike Jackson, and Peter ...

Dealing with complex change in evaluation

Glenda Eoyang, the Human Systems Dynamics (HSD) Institute founder and Judy Oakden, Director of Pragmatica, recently ran a half-day workshop at the ANZEA conference. What set the workshop apart was that Glenda led the session via Zoom from Minneapolis, US, while Judy ran the session in the room in Auckland. The HSD Institute’s depth of experience delivering training globally, both in person and remotely, made this session possible. In addition, the technology was surprisingly easy to set up, an...

Regional Growth Programme Evaluation published

Pragmatica completed an evaluation for MBIE and MPI of the Regional Growth Programme (RGP). First, the review assessed whether the RGP worked as intended. Then, it considered some aspects of the RGP implementation, systems and processes. Finally, it also evaluated the value of the results so far.This evaluation focused on ways government agencies worked with each other. It also considered ways central agencies liaised with the regional stakeholders and with Māori in the regions. It was not an e...

New e-book on evaluative rubrics

Judy Oakden wrote this e-book for practitioners and those wanting to use evaluative rubrics. In her mentoring work, she observed that sometimes certain types of rubrics are not well suited for some kinds of evaluation. In these instances, evaluators could get ‘stuck’. Does this sound like you? “We spent weeks writing up the descriptions for ratings categories. Soon the rubrics became bigger than Ben Hur, taking up pages and pages of tables filled with minuscule writing…For us, the ...

Workshop on systems approaches in evaluation practice

Judy Oakden co-facilitated a workshop session with Marah Moore and Jan Noga at the American Evaluation Association Conference in Washington. It was a great chance to connect with others to think about applying systems thinking in our evaluation practice. We used an appreciative inquiry approach to help participants reflect on when systems thinking works well in their evaluation practice. First, we reflected on the questions that matter in developing evaluative criteria. We used four lenses: moti...

New paper published on Adaptive Evaluation

Glenda Eoyang and Judy Oakden recently had a paper called Adaptive Evaluation: A synergy between complexity theory and evaluation practice published in the journal Emergence: Complexity and Organisation.To download the artlcle, click here...

Putting the theory of evaluative rubrics into practice

Joint presentation with Jane Davidson and other Kinnect Group colleagues at CES Vancouver. Along with Jane Davidson, Kinnect Group members, Judy Oakden, Julian King, Nan Wehipeihana and Kate McKegg presented Evaluative Rubrics – delivering well reasoned answers to real evaluative questions recently at the Canadian Evaluation Society in Vancouver, May 2017.You may know what evaluative rubrics are, but how do you do them in practice?How do you use them to: engage stakeholders...

Using evaluative rubrics: Lessons from the field

Judy Oakden presented a paper on evaluation rubrics, and some lessons learned over the last five years’ in their use at the American Evaluation Association conference in Chicago. Drawing on some ideas from Human Systems Dynamics this paper was developed in collaboration with Glenda Eoyang....

Reaching across boundaries in high stakes collaboration

Judy Oakden, along with Kate McKegg and Viv Twyford presented a paper about our learnings about evaluating high stakes collaboration in local government settings at the Australasian Evaluation Society conference.Download PDF here....