PHASE IV: EVALUATION AND REPORTING

This phase of the curriculum development process is divided into two steps, 1) evaluation strategies and 2) reporting and securing resources. At the beginning of the curriculum development process, a reminder was given to "start with the end in mind." Formative evaluations were conducted in two of the previous steps, Step 3: Needs Assessment and Step 8: Test and Revise. The evaluation strategy step (11) describes summative evaluations and their use. The twelfth and final step focuses on what to report and how to use the reports for securing or accounting for funding. These two steps are important to make judgements regarding the extent the intended outcomes are achieved, to determine accountability (i.e., was the outcome worth the investment of time and resources), and to provide evidence that further investments will have a greater impact on the problem. The summative evaluation process provides the reporting information to secure additional resources.

STEP 11: EVALUATION STRATEGIES

"If You Don't Know Where You Are Going,
 You'll End Up Somewhere Else."

Two types of evaluation are highlighted here. Formative evaluation uses data to make judgements about how well a portion of the process has achieved the intended outcome. The aim of formative evaluation is to provide information helpful to final product design. It also provides baseline data and lays the foundation for the summative evaluation (i.e., the sum of all parts). The aim of a summative evaluation is to step back, put all the parts together, and make judgements as to how well the overall intended outcomes have been achieved. It answers the question: Did we achieve what we intended?

The following strategies are adapted and synthesized from several resources (Boone, 1985; Boyle, 1981; Case, 1987; and Rossi & Freeman, 1985) listed in the evaluation reference section.

At six phases in programs or projects, evaluations should occur. The evaluations either improve activities to help learners attain the intended outcome or assess if the intended outcomes have been achieved.

In the curriculum development process described in this guide, the needs assessment process  provides the baseline data for a summative evaluation. The target audience needs are assessed. The curriculum is developed to meet those needs. The summative evaluation provides evidence if the identified needs have been met.

Procedures for developing an evaluation include:

1. Establish standards or criteria (i.e., the standard of measurement to determine the  extent to which an outcome has been attained). Review intended outcomes and baseline data during the needs assessment. Transform them into a criteria (a standard of measurement) to collect evidence. Consider the levels of need (knowledge, attitude, and practices) and the levels of learning (knowledge, attitude, skills, aspirations, and behavior).

Population Education Example:Criteria Intended outcome = Through participation in the curriculum activities, learners will be able to identify three ways overpopulation effects agricultural production. Criteria = ability to recall one or more of the six ways agricultural production is directly effected by overpopulation.

2. Gather evidence about the criteria.   Identify indicators: words, numbers, or things as proof of the extent to which the criteria is present.

Population Education Example: Evidence (Indicators) Criteria = ability to recall one or more of the six basic elements of agricultural production most directly affected by overpopulation. Evidence = Eighty percent of the learners were able to recall three of the six major impacts of overpopulation on agricultural production. Ten percent recalled all six and six percent recalled two.

There are three primary methods used to gather evidence:

1.  Question Learners: Group discussions   Questionnaires   Informal conversations Exposition: learners write Interviews: personal, telephone, structured, & unstructured. Learners keep a running  log or diary of their experiences 2.    Observe Learners 3.    Examine Documents

Before gathering data, consideration should be given to the type of data needed for the  reports generated. "Once again, start with the end in mind", know where you are going and the data needed support your intended outcome. It is time to step back, anticipate what you want and need to say about the impact the curriculum has had or could have if additional resources can expand the effort. Planning how to report impact must be based on the knowledge already gained from formative evaluations and pilot testing the materials and training program.

 A report can include several types of information based on the evidence in data. Decisions  about what to include or the focus of the report must also be determined before gathering the data. A useful guide to the kinds information reported is Bennett's (1976) seven categories of evaluation evidence:

Categories of Evaluation Evidence: End results. The changes or actions by people and communities.  Emphasis is on the economy, environment, or social and cultural adjustments and improvements Practice Change. The focus is on specific actions the learner is now doing (e.g., accounting practices, farm practices, business procedures, nutritional practices, or relationships within a family). KASA change.   This category refers to changes in knowledge, attitudes, skills, and aspirations. The measurements may be validated or they may simply be opinions on the extent of change. Reactions. The views of the learners often are useful in a report. These views can reveal feelings about the value of a program. People   involvement.  Data on the number of people participating in a  program can be used in a variety of ways in a report. Numbers can be described in terms of student-contact hours, as a percentage of the total population, of social-economic groups, or of professional groups. Activities. The inclusion of information about what was made available through various learning opportunities is useful in a report. This type of information illustrates the efforts of the programmer to assist people through educational opportunities. Inputs. This category includes the staff and other resources that are put into the program.

After evidence (data) is gathered, it is tabulated and analyzed to determine trends. The  objective is to gather information to judge the value of the curriculum or program. The reference, Chase (1987), How are we doing?, includes suggestions on how to design questionnaires, interviews, and analyze data, both qualitative and quantitative.

3. A judgement is made about the value of the curriculum or program by comparing the evidence (presence of indicators) with the criteria.

Population Education Example: Judgement. Evidence = Eighty percent of the learners were able to recall three of the six  major impacts overpopulation has on agricultural production. Ten percent were able to recall all six and six percent recalled two. Four percent did not respond. Judgement = Evidence suggests learners understood the relationship between overpopulation and agricultural production given the comparison between the  pre-test (needs assessment) results where only 29% of the learners could identify two or more of the six major impacts of overpopulation on agricultural production compared to 80% of the learners who recalled three or more of the impacts after participation in the curriculum activities.

Once the value of the program has been determined, by the evidence supplied by the indicators,  the next step is to report the value of the program to clients, stakeholders (shareholders), funding sources, and policy makers.

STEP 12: REPORTING AND SECURING RESOURCES

Sometimes there is no next time, no second chance, no time outs. Sometimes it is now or never."         

This step includes two important aspects of the curriculum development process: 1) reporting curriculum impact and 2) securing resources for future development, expanding training and  the scope of the program through the curriculum and/or the development of additional curriculum.

Reporting

 Perhaps one of the most neglected, yet one of the more important steps in the curriculum development process, is reporting the value and impact of the curriculum and the program it supports. The evaluation provides the evidence of the program's value and the impact it has  had on the lives of the learners. The report is vehicle to communicate that information.

Functions of reports are to:  "Provide a basis for further program (curriculum) development  and improvement. Reporting the effectiveness of a program will allow various clientele and support groups to suggest new directions for programs, new topics, and needed changes in format, promotion, and methodology. Provide the necessary information or data for support and defense for continuing or expanding programs where such action is appropriate. Individuals who hold financial decision-making responsibilities are supportive of quality or success. They want continuous evidence that they are making correct decisions. Provide the basis for promotion and public relations. Findings and conclusions for evaluations are useful in creating a positive image for the institution and in the promotion of future program efforts. A strong promotional effort is essential for maintaining the interests and commitment to continuing education (Boyle, 1982)."

Similar to the curriculum development process, the first consideration in the reporting process  is to identify the target audience for the report. The rest of the process is focused on what is being communicated, to which audience, and for what purpose. Additional tips for preparing an evaluation report include:

Tips for preparing reports: Know who you are reporting to and why. Clarify the purpose of the report. Clarify what you want to report.   More than one report may be necessary if the audiences are quite different. Consider: clients, members of the community, funding sources, and policy makers. Keep the report short, to the point, and well organized. Do not clutter it with  information not relevant to the central message. The main ideas should be clear. Relate the information to a few questions that need to be answered for the reader and to recommended actions. Put the most important information up front. The summary should be at the  beginning (for those who will not read the entire report). Start each part or section with the most important information. Make the report interesting and appealing. Use simple visuals, graphs, and  illustrations. The style and layout of the report is important no matter how simple it is. Explore using more than printed formats, (e.g., use video tapes). Use clear and concise wording. Keep the reader in mind. Define basic terms that may be unfamiliar to the reader. Include anecdotal remarks or comments by learners to help make the report "come alive."

The content of the report will depend on the evaluation evidence (refer to Categories of  Evidence (Bennett) and the targeted audience of the report. Be sure to recognize the whole curriculum development team, including the test facilitators and youth participants.

Reports should be interesting to read as well as useful to promote and explain the program, it's  impact, and to secure resources. But if it is not disseminated and used for what it was intended, it is a waste of time and effort to gather the data and prepare it. One of the most important strategies is your plan for using die report effectively to accomplish the outcomes you intend.

Securing Resources

 An area of primary importance in the program and curriculum development process is the  project's source of funding. Whether funds are secured from non-governmental or governmental organizations, justification and accountability are essential. The evaluation and reporting strategies are focused on providing evidence the curriculum developed is justifiable in terms of cost-benefit analysis and expenditures.

 The same basic principles used throughout the curriculum development process apply to  securing additional resources. A list of the key components of a funding proposal includes:

Funding Proposal Components: Know your audience. Know the priorities of the funding source and how your project meets their priorities (needs). Help them understand what benefits they get from funding the project. Clearly communicate the needs of the targeted audience and the intended outcomes. Outline the expected subject matter (content) and methods to help the targeted audience achieve the intended outcomes. Link budget items to each phase of the project: planning,  implementation (producing products, testing and revising, recruiting and training volunteers, and promoting the program and/or product), evaluating, and reporting. The cost of each phase is the basis of the budget and need for funding. Show "in-kind" values (i.e., projected costs of volunteer services and/or staff salaries) as well as direct and indirect costs. Outline your evaluation strategy. Show how you will measure the project achievement by the intended outcomes. The evaluation section should be very clearly communicated. Include a reporting and communications schedule with key components identified.

This completes the curriculum development process and provides a foundation to "begin all  over again." The curriculum development process is much like program development. It is a continuous cycle where the completion of one cycle leads to the next either because of new issues and needs or for revision and updating. The processes and procedures outlined in this guide are a systematic approach to curriculum development. It is my intention that it will support programs addressing major issues and problems. It is my expectation that it can and will be adapted to specific situations and conditions, modified, and improved upon with use, feedback, and evaluation. It is my aspiration that it will support the efforts of the many dedicated and talented people who are planning and implementing programs aimed at impacting major issues and problems, especially those of youth. There is no greater task, nor better measure of a society, than that of preparing youth to transition into a rewarding and productive life. There are many opportunities!

"It takes a whole village to raise a child"

An African Proverb