FAQs > Learn More about Evaluation
About Home | Snapshot | FAQs | Careers > Learn More
Hiring an External Evaluator When selecting an external evaluator consider several factors. What is the evaluator's academic preparation and credentials? What is the evaluator's recent professional evaluation experience? How many projects has the evaluator completed? For whom? Is the evaluator's expertise and experience a fit for your project? Can the evaluator provide sound references? Is the evaluator's personality and work style a fit for your team? Beyond professional expertise and an outside perspective, a good external evaluator provides impartial and credible assessment of the project. As a trusted outsider the evaluator should be able to elicit sensitive information from users and team members that may not get shared with an internal evaluator. Good evaluators are often familiar with project management, funding-agency requirements, legal issues, and the project's potential liability. A good education evaluator will identify clear stakeholder groups. These might include the sponsor or funding agency that pays for an education project; the client and their employees who develop or deliver education programming; participants or learners for whom the programs are designed; and other audiences such as parents, the community, and the professional field who can learn from the evaluation. When deciding to hire an external evaluator it is important to consider the reasons for the evaluation. Does the funding agency require an evaluator? Is there a legal requirement for evaluation? Does the project team want an evaluator? Did administrators ask that the program be evaluated? Is the purpose of the evaluation clearly stated? Are there clear evaluation criteria? Will findings be supported by data and defendable? Is the program ready to be evaluated? What influence and side effects will the evaluation have? Will anyone use the results of the evaluation? This short, perhaps overly simple, article attempts to demystify evaluation. The goal is to offer a beginning evaluation vocabulary. There are many ways to classify evaluation. This article uses the terms "formative" and "summative," first offered in 1967 by the respected evaluator Michael Scriven, Ph.D. The article also talks about internal and external evaluation. Front-end research helps determine what learners want, know and need. Formative evaluation happens at the beginning of a project, to guide and improve program design. Formative-Summative (or "remedial") evaluation helps improve or modify an existing program. Summative evaluation occurs at the end of a project, to document success and failure. A team member carries out internal evaluation. An evaluation contractor leads external evaluation. Meta evaluation evaluates the evaluation. Tips on Hiring an External Evaluator "When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative." - Robert Stakes, Ph.D., a noted evaluator. Front-end research normally precedes formative evaluation. During front-end research evaluators talk to users before development begins. Front-end research is a little like education's market research. Evaluators use focus groups, surveys, and in-depth interviews to learn what users are interested in, what they expect, and what they already know about a given topic. Front-end research may explore learners' cognitive and developmental levels to ensure age-appropriate education and communication. The result of front-end research is information used to define clear goals, making them easier to realize. So if tasting soup is formative, then front-end research asks guests in advance, "What kind of soup do you want? How do you like your soup?" Front-end research tells you:
Formative Evaluation - finding and fixing problems early Formative evaluation usually happens as programs, projects, or products develop. Evaluators see how users react to prototype programs, curricula, or museum exhibits. Evaluators communicate their findings to the design team so that it can improve the program or product. Field-testing is a kind of formative evaluation. Formative evaluation:
Formative-Summative: Modifying existing programs Formative-summative (or remedial) evaluation assesses on-going projects to offer practical advice for improvement. Mid-point evaluation identifies what program features are working and specific points of weakness, leading to concrete recommendations for change. Mid-point evaluation is used to:
Summative Evaluation: Outcomes Summative evaluation is an after-the-fact assessment. It determines whether the project or program met its goals. Summative evaluation looks at initial goals to assess the overall effectiveness, impact, or cost-benefit of a project. Summative evaluators are also interested in user satisfaction and outcomes - intended and unintended. A book review or software review is a simple example of a summative evaluation. Summative evaluation asks:
Internal evaluation is led by a member of the client's project team. This person is usually knowledgeable about the project and understands its strengths and issues. Internal evaluation is often less expensive and quicker than external evaluation. The risks of doing an internal evaluation are potential bias and the evaluator being too close to the project to notice weaknesses that may be clear to an outsider. Many projects include an internal and external evaluator. This approach combines the low cost and local presence of an internal evaluator with the objectivity and experience of an external evaluator. External evaluators are independent third parties who offer a clear and unbiased point of view. External evaluators may bring new evaluation techniques and experience from several projects. An external evaluator's fresh view often sparks new ideas within the client's team. Although more expensive than internal reviews, external evaluators offer credibility, quality control, and new ideas that are important to customers and funders. Summative evaluations are almost always led by external evaluators. A meta evaluation is an evaluation of the evaluation. An external evaluator reviews the process used to evaluate the client's program and analyzes the results of that evaluation. In practice the external evaluators may serve as an objective third party to assess the feasibility and accuracy of an evaluation; the competence of evaluators, and the integrity of the evaluation findings and the evaluation's strengths and weaknesses. In other cases the external evaluator acts as a coach and critical friend to help plan, define, and sometimes guide the evaluation. Internal evaluators collect information, handle logistics, and do the day-to-day evaluation tasks. The external evaluator audits the evaluation process and reviews the data, the data analysis, and the findings, and verifies the accuracy of the final report. |
