Unit 7. Evaluating Your Course |
|
Official
References
|
The main goal of evaluation is to improve learning. In classroom settings, evaluation centers on the quality of instruction, the environment (equipment, lighting, comfort level, etc.), and the course materials, as well as knowledge and application test scores. With distance learning, there are no instructors present, the environment changes with each student's situation, and tests are generally developed to the knowledge or comprehension level. The main thrust of evaluating the value and effectiveness of a distance learning course is through subject-matter expert review, student tryouts and surveys, and criterion-referenced test scores. But evaluation doesn't stop there, as you well know. The effectiveness of the training your students receive is continually assessed in a variety of ways. This information enables you to maintain your course. |
Specific information on evaluation policy and techniques can be found in:
AF Handbook 36-2234, Instructional System Development
AF Handbook 36-2235, Vol. 5, Information for Designers of Instructional Systems
Be sure to comply with any other operating instructions that apply to your school or major command.
 USAF Photo |
Types of EvaluationThere are three types of evaluation you need to know:
|
You will be using mostly formative evaluation techniques until you near the end of development of your course.
Any type of evaluation that is done before a course is implemented or activated is considered formative. Segments of the instruction, e.g., a unit or a lesson, are reviewed, and then you make specific curriculum decisions before the course is finalized. The objective is to find and correct problems in the early stages—saving everybody time and money.
When you finish the development stage of production, you're ready to conduct a validation, which is the final activity of the formative evaluation stage. Reassess the type and number of evaluators (individuals and small groups) you will need to "try out" your IMI. Include a representative sampling of your audience, as well as subject matter experts (SME) and other course managers. Be sure to plan ahead for a time and place. Give your evaluators a schedule and a checklist (sample developed by IT elements) of what they should look for. Tell them how to identify individual screens that need correcting.
Then, go through all the checklists from your evaluators and list the changes you need to make. Be aware that you may not be able to make all the improvements suggested. Time, software, and your own expertise can be limiting factors. Correct inaccurate information or missing links first. Fix the other stuff if you have the time.
Next, you probably should burn a CD of the entire course so you can proceed confidently to the next step, summative evaluation.
 |
Don't forget to schedule extra development time in your schedule to make corrections found during this phase (yes, there will always be some!). Of course, any deficiencies or problems found should be corrected before the course is implemented. |
The objective of summative evaluation is to determine the total effect of the instruction, including affective domain variables. You collect summative data after the course is finalized but before it is activated. In effect, you're performing a beta test.
When you're near the end of development of the course, the distinction between formative and summative evaluation begins to fade. In its purest sense, summative evaluation involves student tryouts in an operational setting. Especially if your course has never been offered by distance learning, you should gather a representative sampling of your students in an appropriate setting and measure such variables as how long it takes to complete the course, where students had trouble navigating or understanding instructions, what parts were confusing, etc.
|
|
During student tryouts, use
student test scores, student surveys
(questionnaires), and even observation techniques
to evaluate your course. Here are some samples: |
Notes for Observers
Now is the time to finalize your course. This is your last chance to fix all the errors and eliminate all the bugs, to be sure the course runs properly and teaches your subject matter. Resist the temptation to add new graphics or video at this point—you may be introducing errors.
This type of evaluation begins when students enroll in the course and continues through the life cycle of the course. At this phase, you look at internal assessments, such as student questionnaires, AF Forms 17 (student critiques), test results, and test item analyses, and then decide what improvements are needed in the course.
You also can use external feedback from the field in the form of inspection (IG) reports, standardization/evaluation team findings, and training quality reports to determine if your graduates are meeting job performance requirements. You should keep metrics on the types and number of problems your students experience.
The Evaluation Plan you drafted when you started your IMI project guides you through the process. It should include information on how you plan to conduct formative, summative, and operational evaluations. Under formative evaluation, you should include such activities as storyboard approval by an instructional systems specialist (ISS), editing, and review by SMEs. Summative review involves use of the technology. Here you want to involve an ISS, SMEs, other instructors, a sampling of students in small group trials, and, of course, your project manager. For operational assessment, you should list course exam scores, test item analyses, surveys, etc.
Keep a record of findings and changes made during the various stages of evaluation. Update your Evaluation Plan to reflect those findings and changes, and resubmit your plan to AFIADL when you submit the master CD to be activated.
