A systems approach to evaluating impact: teaching at Marlboro College

A systems approach to evaluating impact: teaching at Marlboro College

Marta Ceroni is the Director of Programs at the Academy for Systems Change. Here she shares her experiences of teaching systems thinking to students at Marlboro College, Vermont.

Not many people know where Marlboro College is – in the hills north of Brattleboro, Vermont – and how stunningly beautiful and silent it is up there on an end-of-winter day. Over the four months of the winter term, I get up there one Saturday a month. I drive an hour and a half early in the morning, cruising south along Connecticut River to meet my students. Most of them are nonprofit managers or executives in their late twenties to early fifties. I get to experience their exciting visions and dilemmas through the lens of a course I co-teach called Outcomes and Evaluations for the Learning Organization. Hillary Orsini, my co-teacher, is a bright young change-maker herself with a blunt sense of humor and a huge reservoir of compassion.

When we all get in the same room at 8AM, next door to a deserted library, we acknowledge what we give up on a weekend day to be together before stepping into the three plus hours of the time we have.

Hillary is a master at Result Based Accountability (RBA). Despite the uninspiring name, the approach developed by Mark Friedman helps organizations and individuals take sequential steps to build accountability and move the needle toward desired outcomes. Hillary’s mastery is not only in her knowledge of the content, it is also in her art to chunk it up and deliver it in logical, engaging and entertaining ways – she is also a stand-up comedian in the Burlington scene. My role is to help support and expand that approach with systems thinking.

Too often nonprofits end up defining a problem as not having enough of their favorite solution. Hillary and I have found that RBA is an excellent scaffolding for thinking systemically about how to approach a complex issue.

It starts with reflecting on the qualities of a learning organization. Students name qualities such as deep dialogue and willingness to take risks. People and organizations in genuine learning mode know how to navigate the inevitable tension between current reality and vision, whether looking at financial reality, impact, resources, or alignment with purpose and mission. So, the first thing I do in class is a guided visioning activity and a reflection on the gap that might exist between vision and current reality in their lives and organizations. Feeling the gap can be uncomfortable and the discomfort may take many forms in their organizations, including denial, blame, fear. RBA is our tool of choice, Hillary and I say, to bridge that gap in an honest and courageous way, by asking not only “how much did we do?” and “how well did we do it?” but also “is anybody better off?”

Students apply RBA to making their programs or their whole organization more effective, for example by becoming more inclusive and diverse or by focusing on higher leverage initiatives. Or they might use it to achieve a more livable balance between work and family. But before diving into the technicalities of setting goals and performance measures at the organizational or personal level, Hillary and I spend a great deal of time building awareness of the larger field of the work they do, what Friedman calls the population level. In one activity called “Turn the Curve” we role play a group of stakeholders trying to understand what’s causing – for example – diabetes to continue to rise in Vermont and what works or has worked to make things better. In the few weeks preceding this fun moment, where people get to play for example medical personnel, the youth or lobbyists for food companies, we have built a sense of the feedback loops that might hinder progress by getting people stuck in an unsatisfying status quo or by accelerating undesirable trends. These patterns are known as “system traps” or “system archetypes”. The trap “Shift the burden” seems pervasive in the nonprofit sector: a quick fix reduces people’s willingness and capacity to implement a more fundamental solution. In health care, for example, investing mostly in treating illness might undermine the potential to dedicate resources to prevention.

In this exploration of systems we are accompanied and inspired by Donella (Dana) Meadows’ book “Thinking in Systems” Dana’s words are so deeply resounding, one student told me, that it is hard to process them in class as part of a group discussion. For our last class together I had assigned the last chapter, my absolute favorite, which is also known as “Dancing with Systems”. After a whole book spent orienting the reader on how to understand systems, Dana concludes by saying:

“We can’t control systems or figure them out. But we can dance with them!”

And as part of that dance, we are reminded for example to “listen to the wisdom of the system” or to “pay attention to what is important, not just what is quantifiable”.

The latter, in a course on evaluation and metrics, is of course particularly relevant. For this last part of the course, I continue to search for a deeper form of conversation, one that builds collectively on each other’s points, one that takes us beyond our “individual megaphones”, as the critical student said.

I leave the room with a longing for that missed connection of the hearts, wondering about the type of experience, the creative form that most aptly could bring it to life.  And then I join “Circle”, a social occasion at each of the residencies where the Marlboro community comes together to cultivate connections, gratitude, and to celebrate the students who have completed the whole Master program. Four women who are about to graduate are the focus of the gathering. They receive the appreciation of classmates, teachers, and family members in such an authentic way that I let go of my inner qualms; the connection I was striving for is palpable here and it has the whole community at its center. It is always beautifully humbling and disarming to discover, one notch beyond one’s circle of influence, the stronger power of the whole!

Share this post