Carson Research Consulting: Realize

If you want to know if your program’s participants mastered the objectives of the program, the Success Case Method might be for you. (See this report for a summary of this method: http://tinyurl.com/successcasemethod). This approach involves focusing on those individuals who were either particularly successful or particularly unsuccessful at learning your program’s objectives.

The approach is very purposeful, in that you don’t select a random sample of participants; you go to participants at both ends of the learner spectrum to gather information. It might seem odd to not focus on the average learner, but not so. Focusing on the extremes can offer you much more specific information that is likely to help the average learner as the goal is to seek out successes and failures and rigorously describe the story of how they got there. Design firms agree; Tim Brown, author of Change by Design (2009) believes that learning efforts directed at these two groups may provide the most useful information. For example if you want to develop user friendly kitchen gadgets you may want to focus on the elderly or children to see how they manipulate current kitchen gadgets.

The Success Case Method involves eight steps:
1. Determine the program’s learning objectives
2. Survey a representative sample of the learners
3. Identify highly successful and unsuccessful learners
4. Conduct interviews with the identified learners
5. Attempt to quantify the impact of the teaching
6. Identify the learning outcomes and contributions
7. Describe the learning process
8. Describe any factors that contributed to the learning process (including the learning context, like the room)

The most challenging aspects of this approach include developing survey questions that cover the learning objectives and interviewing the learners to appropriately and completely elicit their learning process. This is because you must adequately define success and failure as well as create a clear structure for the interview, while still remaining open to discovering new information. It is also time sensitive. Waiting too long to assess the objectives and follow-up with the learners can dramatically limit the quality of information you elicit from them. They simply may have forgotten the details of the program and how it impacted them.

Even with these few challenges, the Success Case Method is highly successful and relevant. We believe that as people continue to focus on obtaining maximum usable information in an efficient way, the popularity of this approach will likely grow.

Resource: Brinkerhoff, R. (2001). The Success Case Method. Berrett-Koehler Publishers, Inc. San Francisco, CA

Having more demands on a program than a budget will allow was common well before the market went into decline. Today, programs have even fewer resources than several years ago, but they are still expected to maintain all documented aspects of their programs, including evaluation, which makes the need for efficiency highly critical. How do you carry out an evaluation on a tightly constrained budget?

A common approach is to first have a discussion with the evaluator about the various evaluation designs that may be appropriate for this particular program. Part of that discussion will likely involve the strengths and weaknesses of each methodology and the costs associated with each. Oftentimes, when designing a budget for an evaluation, we propose “phases of evaluation,” such as logic model development (Phase 1), evaluation design (Phase 2), and evaluation implementation (Phase 3). For example, in Phase 1 we would lay out the theory of change and the outcomes that would be expected from the program. In Phase 2 we would discuss the data that needs to be collected, how it will be collected, and what method would be used for collection. Phase 3 would be when the data would be collected, analyzed, and reported. All three phases would be proposed and outlined in the original budget and the program director could choose how many of the phases they choose to fund. In some cases, a program may utilize an external evaluator solely for Phase 1. This approach allows flexibility in the evaluation funding. You can request this budgeting format from an evaluator.

Depending on the level of control you have over your program’s expenses, an initial exercise may be to conduct a budget review, to ensure, before starting the evaluation, that your program’s finances are being spent as efficiently as possible.

Even if you don’t control the budget, you can still determine exactly how much can be dedicated to the evaluation. To do this, review the initial evaluation plan (e.g., the one in the grant that ultimately funded your program) with your evaluator.

Before you meet with the evaluator though, consider the outcomes of the evaluation that are most important, as they may drive (up or down) the costs of the evaluation. Do you need to focus on the program’s implementation to obtain the additional funding that will allow you to focus on several future outcomes? Or is your program far enough along that you need only be concerned with one or two key outcomes? Have aspects of the program changed or will they change as a result of the limited budget?

Another consideration may be a cost-benefit (costs of the evaluation relative to benefits derived from the outcomes, to which you assign a monetary value) or cost-effectiveness (costs of this program relative to other programs) analysis to determine if your program is worthy of the budget it is receiving.

There are multiple ways to conduct a budget-conscious evaluation; the ideas listed here are just a few. What are some other approaches that you would recommend?

A recent New York Times article by Stephanie Strom describes how two philanthropists, Pierre Omidyar of eBay and Peter Lewis of Progressive Insurance, have each created a nonprofit organization to serve other nonprofits. The primary goal of The Omidyar Network and The Management Center is to provide business consulting services and human resource advice that will improve the managerial skills among directors of nonprofits.

Often, leaders in nonprofit organizations are promoted based on their experience or content knowledge, without necessarily having the business skills needed to run the organization. Omidyar and Lewis see the lack of management training within nonprofits as a critical issue to address. They have gone so far as to make their funding contingent on the organization receiving managerial training.

Strom’s article quotes several recipients of their managerial training, and the organizational impact is clear. “I could easily make the case that the nonfinancial resources we got from Omidyar were as equally beneficial as the money they gave us,” said one individual.

This unique approach to the funding of nonprofit organizations could have a dramatic impact on the accountability of programs in the nonprofit sector. By using standards set in the private sector, nonprofits can develop a managerial structure that focuses on growth and goal setting, which would allow their executive directors to focus on a program’s big picture, rather than micromanaging staff issues.

This shift could, in turn, positively affect the evaluation of programs receiving this type of training. It’s no secret that many nonprofit executives are stretched thin. Implementing managerial training could help create leaders who are more skilled and more effective at managing their resources, which would free up their time to focus on evaluation, a task often put on the back burner to give way to fighting fires.

Despite the pushback from nonprofit leaders when they hear their organization referred to as a business, it’s these private sector business skills that could allow their program and their vision to flourish.

What are your thoughts on this new approach to nonprofit management?

A report released by Public /Private Ventures in March 2011 titled “Priorities for a New Decade: Making (More) Social Programs Work (Better),” discussed a critical problem with the evaluation process for non-profit programs: Often times, evaluators do not collaborate with a program and therefore programs are passively evaluated. In addition, funders may not ask for the right evidence and an often impractical report is usually produced months later. This gives the non-profit no voice in the evaluation process and no time to make adjustments or improvements in their program.

The authors recommended that evaluators collaborate with the program staff (i.e. the stakeholders) in the evaluation design, and that the stakeholders are provided real-time, actionable feedback that allows the program to improve its effectiveness. They believe that evaluators need to help non-profits better understand evaluative data and more effectively use the data to improve their programs.

In a recent daily alert email, the American Evaluation Association (AEA) discussed a potential solution for this disconnect, which included the involvement of the stakeholders in the analysis and interpretation of data. In a subsequent white paper, the same authors (Veena Pankaj and Myia Welsh from Innovation Network) recommended three questions for determining if this type of participatory analysis is a good fit for an evaluation:

1. How would including the stakeholders in the analysis improve the quality of the findings?

2. How would including the stakeholders in the analysis improve their engagement with the evaluation?

3. Will including the stakeholders in the analysis fit with a project’s timeline and resources?

Once an evaluation is completed using participatory analysis, the evaluator leads the stakeholders through a constructive discussion of the data and the results. This may include:

1. presenting the data to stakeholders for their feedback, or

2. providing drafts of reports to stakeholders and including their comments in the final report.

This novel approach helps keep the client engaged and interested in the evaluation process and also ensures that their voice is included in the discussion and evaluation. Additionally, it provides more relevant results that are more likely to be put to use. In line with the Public/Private Ventures report, this process would help the organization better understand the data, which would ultimately lead to them using the results.

These reports prompted me to reflect on the evaluations I’ve conducted with non-profit organizations. I was struck by the impact that my use of collaboration has had on my experience conducting evaluations. When I’m fortunate enough to have a close partnership with the program staff of the program that I’m evaluating, the results I provide are much richer and much more comprehensive.

These partnerships give me an insider’s perspective to the program implementation. By taking the time to build these close relationships, I can provide real-time feedback on the data trends I notice, which may in turn help to improve program implementation and program effectiveness. The program gets a more thorough and insightful evaluation, since partnering means I have a better understanding of what’s really going on. In return program staff often ask me for data or insights when they are trying to make decisions about their own program implementations.

What is your experience with evaluators as strategic partners?

A number of our evaluation projects are community based, and at times grants are funded to unite community agencies, so they can work more closely together to achieve their goals. How do you determine how well organizations are collaborating? How do you improve their collaboration? As a result, we’re always looking for evaluation tools that are straightforward and provide complete, easily interpretable results.

In their 2009 study, Cross and colleagues¹ evaluated interagency collaboration using a mixed-method design, which is not an easy task. They approached it from a variety of perspectives, and incorporated qualitative and quantitative data that included network analysis.To completely evaluate this mixed-use approach, they:

• Held focus groups (qualitative) to determine agency classifications and linkages
• Collected ratings of linkage (quantitative): networking, alliance, partnership, coalition, collaboration, and no contact.
• Combined the information using network analysis to create diagrams of the connections with varying thicknesses to represent the linkage ratings

I found this to be an excellent approach. Not only did they design an evaluation that would answer their questions, but they found a way to present that data in an easily understandable way. Their final product was a useful set of visual depictions of interagency networks that allowed a range of stakeholders to quickly identify areas of strong and weak collaboration. These figures also gives the evaluator a clear way to provide recommendations for improving partnerships when necessary.

When people think of mixed-method approaches, most assume there will be a section for quantitative results and one for qualitative results. Or they think that the qualitative data will be used to drive the type of quantitative data gathered (e.g., if you’re developing a questionnaire and you first hold focus groups to decide what questions to ask), instead of one that brings the two types of information together.

In the Cross et al. study, each type of data contributed equally to the finished product and the result was greater than the individual parts. I believe that maintaining this perspective in mixed-method designs will improve the continued effort to advocate the benefits of both qualitative and quantitative data.

Resources: Cross, J.E., Dickmann, E., Newman-Gonchar, R., & Fagan, J.M. (2009). “Using mixed-method design and network analysis to measure development of interagency collaboration.” American Journal of Evaluation, 30, 310-329.

In February, President Obama proposed the use of social impact bonds for seven pilot programs to elicit better results from social services.  In an effort to reduce support for programs that are not effective, social impact bonds would create accountability for programs to succeed. Specifically, nonprofits, particularly foundations, would pay for the programs up front with the support of the government.  The nonprofit and government would agree on performance measures used to evaluate the program’s success.  If, several years later, those performance measures determine that the program is a success, the government would repay the nonprofit, and possibly provide interest with the repayment.  This would reduce the burden of taxpayers’ funds being utilized to back ineffective programs.

The implementation of social impact bonds could open doors for the role of evaluation in governmental programs.  Typically, the government has been void of rigorous evaluations for social programs, leaving policy makers to guess as to which programs are having an impact.  Or, when programs are evaluated and found to be ineffective, funding continues.  In fact, 9 out of 10 major social programs for youth have been shown to have insignificant results.  This suggests that a new approach is needed.

While the business of evaluation research will likely experience a boom if social impact bonds are approved, it will be an area of work where we will need to proceed with caution. The political and financial agendas involved with such a project are likely to be high stakes. Evaluators in this line of work must be ready to answer tough questions, with a lot riding on the answers.

Resource: http://www.nytimes.com/2011/02/09/business/economy/09leonhardt.html?_r=2&adxnnl=1&adxnnlx=1298055904-oR9nMnvMD8BWqv/coi6Bwg

Ahh..the focus group! This is perhaps one of the most well-known evaluation methods. What makes focus groups so popular? First, a focus group is typically a small group of people (<10) who are guided through a structured conversation by a facilitator (likely the evaluator in this case). The evaluator will work with stakeholders to identify who should be a part of the focus group, the purpose of the focus group, and what questions should be asked. A strength of focus groups is that they are often a low cost approach that allows group members to provide information about at topic in a way that the resulting information will likely be richer than if only a single person was interviewed. However, focus groups can’t be used for pre/post comparisons or when confidentiality must be guaranteed. A key component of the focus group is a strong facilitator that keeps the conversation on topic but still allows the group members to address relevant issues that were not in the initial focus group plan. The facilitator must also be comfortable encouraging participation from all group members without offending more opinionated group members. Being aware of the time and allowing for breaks is also important.

Focus groups are popular given their interactive nature. It also gives those relevant to the program a chance to share their thoughts in a way that is not confrontational (this also needs a strong facilitator). Those thoughts are likely to be richer than those that could be captured through paper and pencil or even a one-on-one interview. Returning to the low cost (compared to other approaches) of focus groups is also key. Though time will be spent developing the questions, gathering the participants, paying the facilitator, these costs are still likely to be less than if separate interviews were conducted. Additionally, despite the challenges of bringing together groups of people, the time demands on the evaluator are more limited. Overall, the benefits exponentially exceed the costs, particularly in terms of the quality of information that can be obtained.

Resource: Campbell, P.B. (2001). Conducting focus groups. Prompting Effective Program Evaluation: Reading 4-6. The American Physiological Society.

After a bit of a hiatus from the blog, we are happy to be back!  We plan to post a new blog entry on the first of each month and hope our entries incite some discussion among you in the comments section.

So to get things rolling, we have a question to pose to you…

What constitutes a good evaluation?

Think about that for a minute.

Good questions? Good design? Good data? Good analysis? Good report writing? Good goals?

Should I even give the shallow answer “it depends” or “it is a combination of them all”? Why do these answers satisfy us (or at least get us to move onto the next question)? Probably because they are reasonable and they make some sense; however, a response to a question is different than an answer. Giving an answer to the overall question of what makes a good evaluation will take some thought. An article by Pullin and Knight (2009) in the New Directions for Evaluation journal discussed data credibility. Their opening paragraph offers that collecting and analyzing data is necessary to answer what works and what doesn’t, with the caveat that the data need to be credible. That sounds straightforward enough. It is like Snapple- better stuff means a better drink. In an ideal world, an evaluation would start with good goals and questions. Then a design appropriate to those questions would be developed. The data would be immaculately collected and maintained, with concurrent analyses and reports generated. But in reality, we are more likely to start asking the questions once we all ready have the data than we are to start with our goals and questions. The method will be modified to fit budgetary and timeline constraints, and as such will likely no longer address the original questions exactly. As the program begins and the data are collected it will become clear that a few changes are necessary. The changes will be made with the short-term goal of the program moving forward, but not the questions and outcomes in mind. The report will be generated with caveats and suggestions for next time. Is this likely reality still a good evaluation?

Pullin and Knight (2009) suggest implementing high standards for the methods and determining the necessary “strength of evidence”. I’d have to agree that determining your methods capabilities initially may create more realistic expectations and attainable outcomes. A good evaluation above all has to meet the stakeholder’s needs in a useful way. If you have strong methods you will be starting from a credible base.

Resources: Pullin, A.S., & Knight, T.M. (2009). Data credibility: A perspective from systematic reviews in environmental management. In M. Birnbaum & p. Mickwitz (Eds.), Environmental program and policy evaluation: Addressing methodological challenges. New Directions for Evaluation, 122, 65-74

Predictions of when the full economic recovery will occur change frequently. For example about two months ago, I was reading ‘The Wall Street Journal’ (August 14, 2009) and a couple of their articles about the economy were striking. For the first time in recorded history the US is not the first country to lead the global economy back into the black. Countries like China and Europe seem to be showing signs of recovery before the US. Europe in particular is reporting a rebound in retail sales, unlike the US. Now that time has passed it seems like the recovery overseas will be slower than initially anticipated. However there is still the issue of what is driving the shifted recovery?

The US may have changed its spending perspective, while other countries are resuming spending. Another article on the popular youth clothing store Abercrombie and Fitch offers a perspective that suggests the US may spend differently for the foreseeable long-term. Abercrombie and Fitch is one of the only retail stores that has not drastically cut their prices in the current market. Why? Apparently, this store is marketed overseas in Europe as a luxury American brand. The retailer doesn’t want to cut costs here and risk losing its image (and profits) overseas. However, it was suggested that the retailer may lose its image on US soil because “cool” is now associated with value. I repeat, “cool” is now associated with value. I thought the latter was worth repeating because it suggests that the recovery in the US at least isn’t going to be similar to more recent economic downturns, but may more resemble the recovery following the Great Depression, which shifted the perspective of that generation forever (think about your parents or grandparents who save everything).

How have you adapted your business to fit with this current, potentially long-term, perspective that “value” is trendy? I’ll go even further to add, that savings and “back to basics” (think the new Allstate commercial) are also part of that value. Abercrombie and Fitch is hoping they won’t have to change and can simply ride out the recession (they have a solid cash reserve and still report growing sales in Europe), but is such a steadfast approach appropriate for small businesses in the US, or even large ones for that matter? I think it goes back to knowing your customer and having a plan in place that will alert you to possible “red flags” that you may need to make more significant changes to keep your business viable. Ignoring or avoiding the situation and signs won’t make them disappear. Often we ignore and avoid them anyway because we aren’t sure what to do about them. Having expert assistance to give you back a sense of control, as opposed to the current “wait and see” approach, may make all the difference in how your business comes out of this recession.

I recently attended a conference that discussed the pros and cons of technology to my work. A consistent topic of discussion was how virtual communications such as e-mail and social networking help or hinder face-to-face communication. People had varying opinions, but it was clear that people didn’t expect these virtual forms of communication to become less popular or cease to be used. What was clear was that people needed to remain conscious of their choices regarding the use of technology.

Then I saw an article on CNN about a guy who was laid off from AIG and used his social networking skills to start his own company to help people find jobs. Intrigued, I wasn’t sure what made this new job site better than a site like Monster or TheLadders. This site is designed around social networking. For example, he has a Facebook page inviting people to his Pink Slip Party where employers and employees can network together about everything job related. This seems like a point in favor of using virtual communications.

Then I saw another article on CNN where a recent University graduate used all of the communication resources the Internet has to offer to try and find a job (hiring new graduates is down about 20% from last year). He ended up with such a diffuse approach he got no return. He ended up focusing his search, tailoring his cover letters, and holding practice interviews to land a job. In this case using social networking does not seem to be as effective. Now what do we do?

If you own a business or manage projects, technology is likely one of the most cost-effective ways to market and evaluate your business, but you have to use the technology consciously. Just like email can lead to spam, social networking sites can lead to people posting everything from what they ate for breakfast to truly interesting stories and insights. Think about your social network as having different constituents in it. For example, Facebook is where I mostly connect with friends. They are more likely to care about what state I’m in, what I saw on my morning bike ride, or how I feel about the weather. Linked in and twitter are reserved for business contacts (and friends). I use these sites to communicate information that is more useful, such as links to new blogs like this one, information about interesting books or articles I’ve read, or updates on conferences I’m attending. So choose your social networking mediums carefully, and develop a strategy for how you use them. They can create great value for very little money, but the effort is still needed to keep you from becoming a social network spammer.