ExploreEngageExpand

Carrying Out an External Evaluation (Part 1 of 3)

By Patricia Steele
HigherEd Insight

Conducting an external evaluation can provide valuable information for program implementation, give an added level of credibility in demonstrating outcomes, and help programs raise additional funds.  But what is evaluation, exactly? How do you decide if external evaluation is right for your program? How much does external evaluation cost? How do you determine what you need from an evaluation?

This blog builds on an ACCN webinar by Patricia Steele of HigherEd Insight.  You can watch the webinar here, or view the slides here.

Hot Tips and Engaging Examples

The goal of evaluation is learning. External evaluation is conducted by researchers from outside your program or organization. While many people think external evaluation means judging a program’s success or failure, the purpose of a high-quality external evaluation is learning, not “gotcha.” It’s important to keep a focus on learning in all steps of evaluation by frequently asking, “What can my organization learn from this evaluation in order to improve what we do?”

Evaluation comes in two broad categories. Formative evaluations are conducted during program implementation, while activities are still in progress. It aims to determine whether or not the program is being implemented as anticipated and how program staff are dealing with unanticipated challenges, and is used to shape the program as its activities continue. A summative evaluation measures program outcomes at the end of a cycle of program activities. External evaluations are often, but not always, summative. A combination of the two is also sometimes useful.

External Evaluation Pros and Cons. Grants Northwest reports in “Using an Outside Evaluator” that external evaluation has many benefits:

  • External evaluation can provided needed capacity when your staff does not have time to conduct evaluation.
  • Evaluators are professionals in the field and bring that to bear on their work. 
  • An external evaluator provides an outside view of your program and stakeholders and funders often view these evaluation results as more credible than an internal evaluation.

There are also a few possible challenges to consider when determining if evaluation should be external:

  • Your organization or program will have less direct control over the evaluation process.
  • No one knows your program better than your own staff.
  • Conducting internal evaluation can build internal capacity.
  • External evaluation can be costly.

Weigh the pros and cons to determine whether internal or external evaluation is best for your program or organization.

Taking the first step in an evaluation. First, make an evaluation plan.  Identify and involve your key stakeholders in this process from the start.

Your plan should describe what it is your program or organization wants to understand about your work. Develop some big-picture research questions and list the types of data you want to gather and report (or are required to report). For example, you might want to know, “What part of our computer science program is most effective in helping to retain students until they earn an associate’s degree?” Data you might use could include student satisfaction, program graduation rates, employers’ views of what makes the program effective, and student demographics.

There are many online resources for planning evaluation. The University of Wisconsin’s Cooperative Extension offers a comprehensive, user-friendly guide that is a good place to start.

You might also consider hiring a consultant to help you determine your external evaluation needs. A day, or even a half-day, of a consultant’s time might prove to be invaluable in getting what you need in the end.

Determine your evaluation budget. Once you’ve done some initial planning, it’s helpful to think about a budget. Most funders expect that you will spend roughly between about five and 10 percent of your total program costs on evaluation. Budgets can vary a great deal and may exceed 10 percent, depending on the evaluation you want and need. For example, a program with multiple sites and complex goals, more than one program model, or varying implementation timelines can increase evaluation costs. You may need to be flexible with your budget, or adapt the evaluation design of the evaluation, or look  for additional funds to support your evaluation goals.  The Baltimore Grantmaker’s Association has a comprehensive guide to external evaluation that includes helpful hints on budgeting for evaluation.

In our next blog, we’ll discuss laying out your request for an evaluation proposal (RFP), identifying and interviewing potential evaluators, contracting, and working with evaluators to get what you need.

Tell Us More

Has your organization or project participated in external evaluation?  Why or why not?  If you have had one, what was the most helpful part of the experience?  The least helpful?

Note: HigherEd Insight serves as the independent evaluator for Lumina Foundation's adult college completion strategy.

Comments

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <div> <p> <br>
  • Lines and paragraphs break automatically.