ADDENDUM N
EVALUATION STRATEGIES
(as used in formal education)
|
Phase One: Formulation, Specification, and Prototyping
-
Selection of audience and analysis of learner characteristics.
-
Initial determination of learner outcomes (crude
objectives, not yet behavioral).
- literature
search to attempt to locate existing materials that will meet the intended
outcomes for the audience selected.
-
Tentative selection of experiential learning
methods (activities).
-
Preliminary contacts for development and field
test sites.
-
Outline of content developed (Intended Outcomes
+ Content + Methods).
-
Final specification of intended outcomes, in
measurable terms (Condition + Performance + Standards).
- Pretesting of criterion materials, designed to
measure achievement of outcomes.
-
Prototype developed for one unit, module, or
section.
-
Reading level,
layout, and fonts used tested for appropriateness to target audience.
|
|
Phase Two: Prototype
Development, Testing, and Draft Writing
-
Drafts produced, appropriate items for each
intended outcome drafted.
-
Developmental trial conducted on draft materials
and test items.
- Revision
and new drafts produced based on developmental trial results, until criterion
level of achievement is reached on each intended outcome.
-
Sets of specifications produced for audiovisual
and/or computer elements.
-
Development of AV/computer, related activities
underway.
-
Contacts for field trials finalized.
|
|
Phase Three: Testing and
Revision Cycles
-
Design team trains field test leader/facilitator
on use of curriculum.
-
Field testing conduced with observers or
extensive feedback from leader/facilitator. Pre/post testing and scoring done.
Data analyzed, interviews with target audience learners and leader/facilitator
completed.
-
Revisions made in training plan, text materials,
AV, activities, evaluation feedback system, etc.
-
Materials subjected to expert external review.
-
Revision as recommended by external review.
-
"Hands off" field trial conducted,
using local staff to conduct leader/facilitator training.
-
Data analyzed, interviews conducted, revisions
as necessary.
-
Final versions of all materials prepared, with
professional youth art, layout, typography, and editing. Printing, videotaping,
etc. completed.
|
Adapted from: Curriculum
Development for Issues Programming: A National Handbook for Extension Youth
Development Professionals. USDA/ES,
1992.
During the curriculum
development process, the curriculum design team should make plans for a final
evaluation from the very beginning: "Start with the end in mind." You
must apply evaluation standards to your work, to the curriculum designed, and
to the training presented.
Assess the performance of your
curriculum product to determine whether it is valid and reliable. Validity
means it does what you said it would do. Reliability means outcomes are
consistently attained given the same methods and conditions for the intended
audience.
SUMMATIVE
EVALUATION.
Summative evaluation is a
measurement of the intended (and unintended) outcomes resulting from the implementation of a program or
curriculum. It's a report of the results after the curriculum or program has
been fully implemented. It's an opportunity to show how well we have done, in
meeting the needs of the audience and in attaining the intended outcomes.
Everyone agrees it should be
much more than a head count of those who have participated. If you have
developed and implemented a needs-based curriculum, you should be able to demonstrate an impact on the needs you set
out to address. That's the important reason for focusing on clearly stating the
intended outcomes and matching activities to them.
How you go about your "summative
evaluation" efforts or accountability will vary depending on the intended outcomes and the desired
outcomes of the stakeholder/shareholders. Guidelines for reporting are included
in the text of this Guide. Decide on the kinds of reports you will need. Most
importantly, tell your successes, clearly and succinctly, so stakeholders and
decision makers at all levels can understand and appreciate the impact of your
program. Try to avoid jargon that won't be easily understood! The following
questions may serve as a helpful guide to help you prepare your report.
- Who are the stakeholders or shareholders to
whom you need to report? (Decision makers at the local level, administrators,
donors, policy makers, organizational leadership, youth, population education
facilitators, etc.)
- What kinds and levels of accomplishment will
be acceptable and/or required for each stakeholder?
- Do your plans include ways of gathering the
data you have identified as necessary?
- What resources will be available to help
carry out the evaluation (i.e., time, money, staff, supplies, equipment, etc.)
?
- What are the advantages, disadvantages, or
limitations of various methods of collecting needed data? How will you market/publicize your results?
- What type of final evaluation product is
desired? How do the target audiences
for your evaluation prefer to get their information? When is it needed? (Timing may be everything!)
