Blog

New Year's Resolutions: Are your outcomes intact?


Happy New Year! Well, happy one month into the New Year. If you're like most people, your New Year's resolutions have already faded. The resolutions I described to my organization's funders, however, will at some point become vivid regardless of how faded they look right now.

 

To set myself up for success, I develop outcomes and an evaluative method by using three primary guiding questions: What are we evaluating? How will we gather data? How do we know our evaluation method is valid?

My family adopted a puppy in December. Let's look at how well we are doing in meeting our resolution to keep “Chewy” from chewing everything in the house.

What are we evaluating?
This question is asked once you know the goal of the program. If you are building a playground, you could be increasing interactions between parents and children, increasing children's health, or preserving local flora and fauna. The playground's completion is only one data point. The other questions will help you focus on the impact.

Our new dog, Chewy, has lived up to her name thus far. Data points include how many things she has destroyed and how many things she has attempted to chew. However, our evaluation should ultimately tell us whether we are creating a positive experience for the family and puppy.

How will we gather data?
It's nice to say that the park will increase interactions between families, but how will you capture that information? Will you seek qualitative (stories) or quantitative (numbers) data? How many staff hours can you dedicate to data collection? Will watching peoples' interaction at the park change their behavior (known as “The Observer Effect”)? Can you adjust your data collection method to account for The Observer Effect?

I didn't want to tell my son that his Lego box was shredded because the second shelf wasn't high enough to keep the box out of the dog's reach. However, he needed that data to change his behavior. Though we are gathering qualitative data via complaints and groans, we could log the time, type of item, and amount of destruction. This would allow pattern identification and insight into the interventions' effectiveness. A more robust data-collection method would help determine if the shelf was reached because of a stepstool next to it, or whether she is able to reach the shelf regardless of the stepstool.

How do we know that our evaluation method is valid?
Is the experiment/intervention repeatable? Would the results and method apply equally to other cultures? Is it based on valid assumptions? Are you accounting for the differing opinions and perspectives of evaluators? If your evaluator expresses his affection primarily through high fives and pats on the back, would he be as likely to identify park-goers' words of affirmation?

Though it is interesting to know what and how much Chewy has destroyed, those data points aren't helpful on their own. We need context. Are dogs more likely to destroy things in their first few months at a new home? How long does the “puppy phase” last for this breed? Does my son have the same definition of “destroyed” as I do?

Many organizations support quality evaluation practices. For example, The American Evaluation Association has published guidelines for evaluation. As you evaluate your own processes, consider how your evaluation methods stack up with those guidelines.

Where will the grant reviewers be able to chew holes in your evaluation method?

Jen Hurst, M.A. has developed programs and strategies for nonprofits since middle school, when she and a few friends started their philanthropic work by raising $150 for the local animal shelter.

GPC Competencies:
03: Knowledge of strategies for effective program and project design and development.
04: Knowledge of how to craft, construct, and submit an effective grant application.


 

Comments

There have been no comments made on this article. Why not be the first and add your own comment using the form below.

Leave a comment

Please complete the form below to submit a comment on this article. A valid email address is required to submit a comment though it will not be displayed on the site.

HTML has been disabled but if you wish to add any hyperlinks or text formatting you can use any of the following codes: [B]bold text[/B], [I]italic text[/I], [U]underlined text[/U], [S]strike through text[/S], [URL]http://www.yourlink.com[/URL], [URL=http//www.yourlink.com]your text[/URL]