Coghlan and colleagues (2003) suggest that the appreciative inquiry method can be a constructive approach to evaluating the function of an organization. Appreciative inquiry is used more often in the private sector, but is being seen more and more as an evaluation approach with applications in the public sector as well. In a nutshell, appreciative inquiry involves working with members of an organization to determine the aspects of their work that are going well, why they are going well, and what they would like to see more of.

Notice I didn’t say “see less of.” Some suggest that this approach denies that problems exist in an organization; however, proponents of the approach would suggest that problems are reframed in a more positive, constructive manner. The focus of appreciative inquiry is admittedly on a program’s strengths, but asking individuals in the organization what they want to see and having them describe their vision addresses current limitations in the organization that need to be improved.

The 4-D model, based on an appreciative inquiry framework, shows the connections between the following:

1. Discovery:  “what is the best of what is?” (appreciating what is best in the organization at this moment)

2. Dream:  “what might be?” (envisioning results for the organization)

3. Design:  “what would be the ideal?” (constructing the future of ideals for the organization)

4. Destiny:  “how to empower, learn and adjust/improvise?” (sustaining the change)

This model can also be adapted to evaluating programs within an organization, not just the organization itself. It isn’t difficult to imagine a situation where you discuss with a client what is working in their organization and end up discussing what is working in a specific program and what else is needed in that program.

In the case of evaluating a one specific program instead of an entire organization, the model can be used as we’ve described it, or it can be tailored to the program’s specific needs, which is also a great approach and a more likely one, given that the method is newer to evaluation of individual programs, and, as such, specific guidelines are not yet available.

Are you aware of an organization or program that seems like it could benefit from an appreciative inquiry evaluation? It is a flexible approach that may even fit in tight budgets.

Resource: Coghlan, A. T., Preskill, H., & Catsambas, T.T. (2003). An overview of appreciative inquiry in evaluation. New Directions for Evaluation, 100,5-22).

Pawson and colleagues highlight the ongoing theme of their article in the following Donald Rumsfeld quote:

“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don’t know. But there are also unknown unknowns. These are things we do not know we don’t know.”

Many times the “known knowns” in research are somewhat conditional and can be interpreted differently depending on context and/or circumstances. In their example, the authors reviewed literature that measured the air quality in the back seat of a car when the driver was smoking.Despite the fact that this measurement provides a clear, numerical output, which is argued by policymakers to be evidence for the smoking ban policy, the authors highlighted the various conditions that could have impacted the measurement, such as open windows or blowing air conditioning.

Research is embedded in a complex system and as a result, findings are not always as black and white as we would like to think.The article’s authors provide an extensive argument for the complexity of conducting research in the real world, where there are no true constants and all evidence is more adequately described as a conditional explanation, rather than empirical proof, of the findings.They describe the role of the researcher in converting “unknowns” to “knowns”, and in weighing the significance of the information they reviewed, quite powerfully:

“In short, the evidence does not deliver the legislative decision.It does, however, provide the grounds on which the policy maker can make a more informed decision.”

This explanation helps to highlight that data does not stand on its own, but rather always requires interpretation.I found this to be a particularly good reminder for evaluation research.It’s important to feed the data back to the programs that we’re evaluating and gather the program staff’s interpretation to incorporate into the findings. At CRC, we have begun to create “learning circles” with program staff in certain domains to discuss data and collect their interpretation of it.We present quantitative and qualitative data we’ve collected and then ask the staff to reflect and help us explain why we’re seeing certain patterns.We recognize that the staff members are the experts in their field and this process has allowed us to include their insight.

What processes do you go through to interpret data?

Reference: Pawson, R., Wong, G., & Owen, L. (2011).Known Knowns, Known Unknowns, Unknown Unknowns: The Predicament of Evidence-Based Policy. American Journal of Evaluation, 32(4), 518-546.

The approach is very purposeful, in that you don’t select a random sample of participants; you go to participants at both ends of the learner spectrum to gather information. It might seem odd to not focus on the average learner, but not so. Focusing on the extremes can offer you much more specific information that is likely to help the average learner as the goal is to seek out successes and failures and rigorously describe the story of how they got there. Design firms agree; Tim Brown, author of Change by Design (2009) believes that learning efforts directed at these two groups may provide the most useful information. For example if you want to develop user friendly kitchen gadgets you may want to focus on the elderly or children to see how they manipulate current kitchen gadgets.

The Success Case Method involves eight steps:
1. Determine the program’s learning objectives
2. Survey a representative sample of the learners
3. Identify highly successful and unsuccessful learners
4. Conduct interviews with the identified learners
5. Attempt to quantify the impact of the teaching
6. Identify the learning outcomes and contributions
7. Describe the learning process
8. Describe any factors that contributed to the learning process (including the learning context, like the room)

The most challenging aspects of this approach include developing survey questions that cover the learning objectives and interviewing the learners to appropriately and completely elicit their learning process. This is because you must adequately define success and failure as well as create a clear structure for the interview, while still remaining open to discovering new information. It is also time sensitive. Waiting too long to assess the objectives and follow-up with the learners can dramatically limit the quality of information you elicit from them. They simply may have forgotten the details of the program and how it impacted them.

Even with these few challenges, the Success Case Method is highly successful and relevant. We believe that as people continue to focus on obtaining maximum usable information in an efficient way, the popularity of this approach will likely grow.

Resource: Brinkerhoff, R. (2001). The Success Case Method. Berrett-Koehler Publishers, Inc. San Francisco, CA

A common approach is to first have a discussion with the evaluator about the various evaluation designs that may be appropriate for this particular program. Part of that discussion will likely involve the strengths and weaknesses of each methodology and the costs associated with each. Oftentimes, when designing a budget for an evaluation, we propose “phases of evaluation”, such as logic model development (Phase 1), evaluation design (Phase 2), and evaluation implementation (Phase 3). For example, in Phase 1 we would lay out the theory of change and the outcomes that would be expected from the program. In Phase 2 we would discuss the data that needs to be collected, how it will be collected, and what method would be used for collection. Phase 3 would be when the data would be collected, analyzed, and reported. All three phases would be proposed and outlined in the original budget and the program director could choose how many of the phases they choose to fund. In some cases, a program may utilize an external evaluator solely for Phase 1. This approach allows flexibility in the evaluation funding. You can request this budgeting format from an evaluator.

Depending on the level of control you have over your program’s expenses, an initial exercise may be to conduct a budget review, to ensure, before starting the evaluation, that your program’s finances are being spent as efficiently as possible.

Even if you don’t control the budget, you can still determine exactly how much can be dedicated to the evaluation. To do this, review the initial evaluation plan (e.g., the one in the grant that ultimately funded your program) with your evaluator.

Before you meet with the evaluator though, consider the outcomes of the evaluation that are most important, as they may drive (up or down) the costs of the evaluation. Do you need to focus on the program’s implementation to obtain the additional funding that will allow you to focus on several future outcomes? Or is your program far enough along that you need only be concerned with one or two key outcomes? Have aspects of the program changed or will they change as a result of the limited budget?

Another consideration may be a cost-benefit (costs of the evaluation relative to benefits derived from the outcomes, to which you assign a monetary value) or cost-effectiveness (costs of this program relative to other programs) analysis to determine if your program is worthy of the budget it is receiving.

There are multiple ways to conduct a budget-conscious evaluation; the ideas listed here are just a few. What are some other approaches that you would recommend?

Often, leaders in nonprofit organizations are promoted based on their experience or content knowledge, without necessarily having the business skills needed to run the organization.Omidyar and Lewis see the lack of management training within nonprofits as a critical issue to address. They have gone so far as to make their funding contingent on the organization receiving managerial training.

Strom’s article quotes several recipients of their managerial training, and the organizational impact is clear. “I could easily make the case that the nonfinancial resources we got from Omidyar were as equally beneficial as the money they gave us,” said one individual.

This unique approach to the funding of nonprofit organizations could have a dramatic impact on the accountability of programs in the nonprofit sector.By using standards set in the private sector, nonprofits can develop a managerial structure that focuses on growth and goal setting, which would allow their executive directors to focus on a program’s big picture, rather than micromanaging staff issues.

This shift could, in turn, positively affect the evaluation of programs receiving this type of training.It’s no secret that many nonprofit executives are stretched thin.Implementing managerial training could help create leaders who are more skilled and more effective at managing their resources, which would free up their time to focus on evaluation, a task often put on the back burner to give way to fighting fires.

Despite the pushback from nonprofit leaders when they hear their organization referred to as a business, it’s these private sector business skills that could allow their program and their vision to flourish.

What are your thoughts on this new approach to nonprofit management?

The authors recommended that evaluators collaborate with the program staff (i.e. the stakeholders) in the evaluation design, and that the stakeholders are provided real-time, actionable feedback that allows the program to improve its effectiveness. They believe that evaluators need to help non-profits better understand evaluative data and more effectively use the data to improve their programs.

In a recent daily alert email, the American Evaluation Association (AEA) discussed a potential solution for this disconnect, which included the involvement of the stakeholders in the analysis and interpretation of data. In a subsequent white paper, the same authors (Veena Pankaj and Myia Welsh from Innovation Network) recommended three questions for determining if this type of participatory analysis is a good fit for an evaluation:

1.How would including the stakeholders in the analysis improve the quality of the findings?

2.How would including the stakeholders in the analysis improve their engagement with the evaluation?

3.Will including the stakeholders in the analysis fit with a project’s timeline and resources?

Once an evaluation is completed using participatory analysis, the evaluator leads the stakeholders through a constructive discussion of the data and the results.This may include:

1.presenting the data to stakeholders for their feedback, or

2.providing drafts of reports to stakeholders and including their comments in the final report.

This novel approach helps keep the client engaged and interested in the evaluation process and also ensures that their voice is included in the discussion and evaluation.Additionally, it provides more relevant results that are more likely to be put to use.In line with the Public/Private Ventures report, this process would help the organization better understand the data, which would ultimately lead to them using the results.

These reports prompted me to reflect on the evaluations I’ve conducted with non-profit organizations.I was struck by the impact that my use of collaboration has had on my experience conducting evaluations.When I’m fortunate enough to have a close partnership with the program staff of the program that I’m evaluating, the results I provide are much richer and much more comprehensive.

These partnerships give me an insider’s perspective to the program implementation. By taking the time to build these close relationships, I can provide real-time feedback on the data trends I notice, which may in turn help to improve program implementation and program effectiveness. The program gets a more thorough and insightful evaluation, since partnering means I have a better understanding of what’s really going on. In return program staff often ask me for data or insights when they are trying to make decisions about their own program implementations.

What is your experience with evaluators as strategic partners?

In their 2009 study, Cross and colleagues¹ evaluated interagency collaboration using a mixed-method design, which is not an easy task. They approached it from a variety of perspectives, and incorporated qualitative and quantitative data that included network analysis.To completely evaluate this mixed-use approach, they:

• Held focus groups (qualitative) to determine agency classifications and linkages
• Collected ratings of linkage (quantitative): networking, alliance, partnership, coalition, collaboration, and no contact.
• Combined the information using network analysis to create diagrams of the connections with varying thicknesses to represent the linkage ratings

I found this to be an excellent approach.Not only did they design an evaluation that would answer their questions, but they found a way to present that data in an easily understandable way.Their final product was a useful set of visual depictions of interagency networks that allowed a range of stakeholders to quickly identify areas of strong and weak collaboration.These figures also gives the evaluator a clear way to provide recommendations for improving partnerships when necessary.

When people think of mixed-method approaches, most assume there will be a section for quantitative results and one for qualitative results.Or they think that the qualitative data will be used to drive the type of quantitative data gathered (e.g., if you’re developing a questionnaire and you first hold focus groups to decide what questions to ask), instead of one that brings the two types of information together.

In the Cross et al.study, each type of data contributed equally to the finished product and the result was greater than the individual parts.I believe that maintaining this perspective in mixed-method designs will improve the continued effort to advocate the benefits of both qualitative and quantitative data.

Resources: Cross, J.E., Dickmann, E., Newman-Gonchar, R., & Fagan, J.M. (2009).Using mixed-method design and network analysis to measure development of interagency collaboration.American Journal of Evaluation, 30, 310-329.

The implementation of social impact bonds could open doors for the role of evaluation in governmental programs.  Typically, the government has been void of rigorous evaluations for social programs, leaving policy makers to guess as to which programs are having an impact.  Or, when programs are evaluated and found to be ineffective, funding continues.  In fact, 9 out of 10 major social programs for youth have been shown to have insignificant results.  This suggests that a new approach is needed.

While the business of evaluation research will likely experience a boom if social impact bonds are approved, it will be an area of work where we will need to proceed with caution. The political and financial agendas involved with such a project are likely to be high stakes. Evaluators in this line of work must be ready to answer tough questions, with a lot riding on the answers.

Resource: http://www.nytimes.com/2011/02/09/business/economy/09leonhardt.html?_r=2&adxnnl=1&adxnnlx=1298055904-oR9nMnvMD8BWqv/coi6Bwg

Ahh..the focus group! This is perhaps one of the most well-known evaluation methods.What makes focus groups so popular?First, a focus group is typically a small group of people (<10) who are guided through a structured conversation by a facilitator (likely the evaluator in this case). The evaluator will work with stakeholders to identify who should be a part of the focus group, the purpose of the focus group, and what questions should be asked.A strength of focus groups is that they are often a low cost approach that allows group members to provide information about at topic in a way that the resulting information will likely be richer than if only a single person was interviewed.However, focus groups can’t be used for pre/post comparisons or when confidentiality must be guaranteed.A key component of the focus group is a strong facilitator that keeps the conversation on topic but still allows the group members to address relevant issues that were not in the initial focus group plan.The facilitator must also be comfortable encouraging participation from all group members without offending more opinionated group members.Being aware of the time and allowing for breaks is also important.

Focus groups are popular given their interactive nature.It also gives those relevant to the program a chance to share their thoughts in a way that is not confrontational (this also needs a strong facilitator). Those thoughts are likely to be richer than those that could be captured through paper and pencil or even a one-on-one interview. Returning to the low cost (compared to other approaches) of focus groups is also key.Though time will be spent developing the questions, gathering the participants, paying the facilitator, these costs are still likely to be less than if separate interviews were conducted.Additionally, despite the challenges of bringing together groups of people, the time demands on the evaluator are more limited.Overall, the benefits exponentially exceed the costs, particularly in terms of the quality of information that can be obtained.

Resource: Campbell, P.B. (2001).Conducting focus groups.Prompting Effective Program Evaluation: Reading 4-6.The American Physiological Society.

So to get things rolling, we have a question to pose to you…

What constitutes a good evaluation?

Think about that for a minute.

Good questions? Good design? Good data? Good analysis? Good report writing? Good goals?

Should I even give the shallow answer “it depends” or “it is a combination of them all”? Why do these answers satisfy us (or at least get us to move onto the next question)? Probably because they are reasonable and they make some sense; however, a response to a question is different than an answer. Giving an answer to the overall question of what makes a good evaluation will take some thought. An article by Pullin and Knight (2009) in the New Directions for Evaluation journal discussed data credibility. Their opening paragraph offers that collecting and analyzing data is necessary to answer what works and what doesn’t, with the caveat that the data need to be credible. That sounds straightforward enough. It is like Snapple- better stuff means a better drink. In an ideal world, an evaluation would start with good goals and questions. Then a design appropriate to those questions would be developed. The data would be immaculately collected and maintained, with concurrent analyses and reports generated. But in reality, we are more likely to start asking the questions once we all ready have the data than we are to start with our goals and questions. The method will be modified to fit budgetary and timeline constraints, and as such will likely no longer address the original questions exactly. As the program begins and the data are collected it will become clear that a few changes are necessary. The changes will be made with the short-term goal of the program moving forward, but not the questions and outcomes in mind. The report will be generated with caveats and suggestions for next time. Is this likely reality still a good evaluation?

Pullin and Knight (2009) suggest implementing high standards for the methods and determining the necessary “strength of evidence”. I’d have to agree that determining your methods capabilities initially may create more realistic expectations and attainable outcomes. A good evaluation above all has to meet the stakeholder’s needs in a useful way. If you have strong methods you will be starting from a credible base.

Resources: Pullin, A.S., & Knight, T.M. (2009). Data credibility: A perspective from systematic reviews in environmental management. In M. Birnbaum & p. Mickwitz (Eds.), Environmental program and policy evaluation: Addressing methodological challenges. New Directions for Evaluation, 122, 65-74