For questions on the content of this page, send E-mail to:
DSN: 596–4420 Comm: (334) 416–4420
Pagemaster
DSN: 596-4277 Comm: 334-416-4277
Webmaster
DSN 596-3795 Comm 334-416-3795
Maxwell AFB-Gunter Annex AL 36118-5643
The parts of a multiple-choice item
Guidelines for writing multiple-choice items
Sending formulas, graphics, or tables for the URE or CE
Alternate method of marking text support
Other options in marking text support
5–2. Writing Self-Test
Questions and Answers
to Self-Test Questions
Acceptable self-test question formats
Answers to self-test questions
5–3. Maintaining and Evaluating Course Examinations
Course author response to item analysis
Rekeying URE items and editing CE items
Maintaining course examinations affected by errata and supplements
SELF-TEST questions, unit review exercises, and course examinations are feedback and evaluation tools. STQs give students immediate feedback and self-evaluation on small amounts of information. UREs continue the teaching process by giving students a review of entire units. CEs evaluate student learning and provide feedback to the student and to test developers on the effectiveness of the course. Part of your job as a course author is to write good multiple-choice and self-test questions. Further, you must maintain questions in printed courses and in the course examinations. You’ll sometimes have to correct, rekey, or delete questions based on statistics and student inquiries. This unit explains both the creative and the maintenance processes so that you can keep a top-notch course in the field.
Multiple-choice items, like topical statements, address the most important points, and you must write them at the proper level of proficiency. Multiple-choice items must be related directly to their corresponding topical statements. This is critical for content validity.
If a topical statement says students must convert binary numbers, don’t write an item asking students to identify or add binary numbers—ask them to convert. In the box below is an example of a good match between topical statement and multiple-choice item.
|
If your topical statement is: |
433. Converting binary and decimal numbers |
|
Your multiple-choice item stem might be: |
What is the binary equivalent of 634(10)? |
The multiple-choice items you send in your U0.DOC file are the source for both the UREs and the CEs. The URE highlights key teaching points. In fact, the unit review exercise might be the most valuable tool for a student to get a true feeling for what is most important.
The essential word in the term “unit review exercise” is review. In our courses, review exercises continue the instructional process—not by introducing new points, but by reinforcing essential information and identifying gaps in student learning.
The multiple-choice questions that we use consist of—
· A stem.
· Four options.
· One keyed response.
· Three distractors.
These terms have specialized meanings in test development.
The stem is the statement
of requirement. The stem may ask a question, direct students to do
something (imperative), or begin a sentence that students must complete by
selecting the right word or phrase (incomplete statement). These examples show the three stem types.
How much force is needed to slide a 100-pound box along a concrete floor if the coefficient of friction between the box and the floor is 0.4?
a. 25 pounds.
b. 40 pounds.
c. 60 pounds.
d. 100 pounds.
Compute the force needed to slide a 100-pound box along a concrete floor if the coefficient of friction between the box and the floor is 0.4.
a. 25 pounds.
b. 40 pounds.
c. 60 pounds.
d. 100 pounds.
Given a 0.4 coefficient of friction between a 100-pound box and a concrete floor, the force needed to slide the box along the floor is
a. 25 pounds.
b. 40 pounds.
c. 60 pounds.
d. 100 pounds.
In a multiple-choice item, options are plausible answers to or completions of the stem. We use four options in all multiple-choice questions. Only one option satisfies the requirement of the stem—the keyed response. The other options are distractors.
Write one option that is clearly the correct response. Make sure the text supports it.
To an unknowledgeable person, each of the other three options must look as if it could be the correct response, though it is not. In fact, the value of your multiple-choice items depends largely on your skill in writing convincing but incorrect distractors.
Use the guidelines and examples on the next few pages to help you write good test items. The guidelines are not all-inclusive; but they can help you to send good multiple-choice items. Always review your items carefully and, if possible, have someone else review them. You may not see problems because of your familiarity with the subject matter.
You must never put essential information in introductions and summaries; therefore, you must never write items on introductions and summaries. Develop multiple-choice items on lesson text only.
Your topical statement must drive the type of multiple-choice item you write. This relationship is basic to good instructional systems development. Your ISS, of course, looks for text support in the lessons, but the ISS also looks for that all-important direct relationship between the objective announced in the topical statement and the multiple-choice item:
Objective of topical
statement--multiple-choice item
Course authors often have trouble with this connection and send us items that do not address their corresponding topical statements. Obviously, if you don’t send us appropriate items, you can experience production delays while we wait for you to write new items and send them to us or while the ISS rewrites your items and coordinates them with you.
To understand better what you need to do, look at the topical statement and the two multiple-choice items below. Note that the original item did not relate directly to the topical statement. The writer asked the student to tell how the barrel assembly on the Hamilton Standard propeller is secured. This may be important, but it’s not the objective of the lesson. The item in no way measured whether or not the student learned the function of the barrel assembly, as specified in the topical statement.
009. Functions of the assemblies on the
Hamilton Standard propeller
What the item writer said:
The barrel assembly is secured to the propeller shaft by
a. a hub nut.
b. lockrings.
c. an extension.
d. a splined coupling.
What the item writer should have
said:
On a Hamilton Standard propeller, the function of the dome assembly is to
a. prevent the blades from decreasing pitch if pressure is lost.
b. provide a stop for the dome piston steel sleeve.
c. transmit engine torque to the propeller.
d. change the propeller blade angle.
Make sure students can find the correct
response to each item in the lesson text, either clearly stated or implied, or
that the student can logically deduce or calculate the answer on the basis of
the text.
Psychologists and professional educators say that information presented in a logical sequence, logically linked, is a great deal easier for students to learn and to recall, whether in total or one piece at a time. Probably none of us has any trouble agreeing with them on that.
It may be a little harder to see, but it is equally true, that multiple-choice test items are more effective if they are presented in the same sequence in which they are supported in the text. The student can mentally scan the pertinent text from the beginning to find the answer. This scanning process not only reinforces the answer in its immediate context (so very important in understanding a procedure, for instance) but also reinforces the text as a whole.
Nearly all course authors send in materials that properly group test items by lesson number. Unfortunately, many authors fail to sequence test items within lessons. Whatever the reason for it, this lack of sequence makes it harder for the student to recall information, so reinforcement is weakened. Lack of sequence also hampers the student’s review of the text (more reinforcement) after answering the URE, and it hampers our efforts to verify your work. We ask that you sequence test items within lessons.
The item stem should present a definite problem that is meaningful without the options. The student should not have to read options to find out what a question means. For example, a student reading the first item stem below would not know whether to respond with what the forms do, where to find them, or what.
What the item writer said:
Instructions for preparing AFTO Form 781.
What the item writer should have said:
Which technical order has instructions for preparing AFTO Form 781?
The student doesn’t have to read the options to know what the second stem is asking.
A student may know just what an item refers to when the item is referenced to a specific lesson and appears in the text, in sequence with other items, but will the student know the implied reference if the item is out of context on a CE?
What the item writer said:
The oil cooler blower is driven by the…
Out of context the stem is general. Does the oil cooler blower refer to ground equipment or to a blower in a helicopter? The student needs to know which type the author has in mind.
What the item writer should have
said:
The oil cooler blower on the H3 helicopter is driven by the ...
The rewritten stem is specific, and the student no longer must guess which type of blower is under consideration. If, on the other hand, your course covers only one specific system or aircraft, you do not need to include that fact in each stem.
Define every acronym and abbreviation each time you use it in a test item (U0.DOC file). Editors at AFIADL will remove all of the definitions in the URE (since they were defined in the unit text), but the definitions will appear in the course examination items.
Put enough, but only enough, information in the stem for the student to be able to answer the question. Items having pronouns with doubtful antecedents, misplaced sentence elements, and words or phrases that can be misinterpreted are unfair to the student. In this example, the writer loaded the item stem with needless words and the stem ended up with a vague pronoun reference:
What the item writer said:
A functional grouping of closely related Air Force positions on the basis of similarity of education, training, experience, and other abilities required to perform them is referred to as ...
What the item writer should have said:
A grouping of tasks that require similar qualifications of airmen who do the tasks is known as ...
But overemphasis on brevity may lead to the omission of needed information:
What the item writer said:
To solve for voltage, what must you do to the original formula?
What the item writer should have
said:
To solve for voltage, what must you do to the original formula for Ohm’s law?
Distractors that are obviously wrong narrow the choices and increase the chances of correct guesses. For instance, in this item, the distractor “balloons” is not believable and the student can rule it out at once:
What the item writer said:
The sensing elements for the vertical reference platform of the automated astrocompass are
a. accelerometers.
b. bubble units.
c. pendulums.
d. balloons.
What the item writer should have
said:
The sensing elements for the vertical reference platform of the automated astrocompass are
a. accelerometers.
b. bubble units.
c. pendulums.
d. synchros.
If you fail to do so, differences other than the correctness or incorrectness of the options may give away the correct response. Item writers tend to make correct responses longer than incorrect ones. In this item, the correct option, d, stands out by its length alone:
What the item writer said:
The best reason for not writing a program using absolute addresses is that it
a. takes too long to write the program.
b. takes too long to debug the program.
c.
is too hard for a new programmer to write.
d. would cause complications if it became necessary to add other instructions somewhere in the midst of the program sequence.
What the item writer should have
said:
The best reason for not writing a program using absolute addresses is that it
a. takes too long to write the program.
b. takes too long to debug the program.
c. is too hard for a new programmer to write.
d. is hard to add new instructions to the program.
If the stem has such words as best, most, least, not, except, or never, italicize them. If other words in the stem are critical to student understanding, italicize them also.
Negatives in the stem can be confusing. Note that in the above sample items, the negative phrasing causes you to read more carefully to be sure you understand the question.
Be grammatically consistent throughout the stem and its options. For example, a or an as the last word in a stem can betray the right choice.
What the item writer said:
A group of tasks requiring similar qualifications of airmen is known as an
a. career field.
b. manning document.
c. Air Force specialty.
d. specialty description.
What the item writer should have
said:
A group of tasks requiring similar qualifications of airmen is known as
a. a career field.
b. a manning document.
c. an Air Force specialty.
d. a specialty description.
Often,
only one answer is text-supported, but other options could be correct. For example:
Good counseling helps develop
a. good communication.
b. community involvement.
c. better working conditions.
d. understanding and respect for others.
A key word in both the stem and an option can reveal the correct answer. In this item the word task gives away the correct option:
What the item writer said:
An “a” proficiency code on an STS item says a
student should be able to name parts, tools, and simple facts about a task. The
letter a is on the proficiency code
key under the heading of
a. subject knowledge.
b. task knowledge.
c. performance.
d. principles.
What the item writer should have
said:
An “a” proficiency code on an STS item says a student should be able to name parts, tools, and simple facts about a job. The letter a is on the proficiency code key under the heading of
a. subject knowledge.
b. task knowledge.
c. performance.
d. principles.
Use catchall options with caution. Students generally assume (correctly) that the catchall option is the correct answer.
An item having such options tends to test reading skills rather than specialty knowledge.
If you hide a negative word in a contraction, the student may overlook the negative and answer the question incorrectly.
Students are used to having numbers ordered, from low to high or from high to low. It is a nuisance not to have them ordered.
What the item writer said:
The square root of 9409 is
a. 97.
b. 87.
c. 107.
d. 77.
What the item writer should have
said:
|
The square root of 9409 is a. 77. b. 87. c. 97. d. 107. |
or: |
The square root of 9409 is a. 107. b. 97. c. 87. d. 77. |
Review existing items and verify that each item addresses the topical statement. Replace items that fail to do so. Use the item analyses to check the performance of existing CE items. Improve or replace weak items. Often you can make a weak item stronger by rewording the stem or changing an option. Sometimes, you need to rewrite the lesson text to support the item better.
NOTE: Before using an old version of a URE or CE as a source for test items, check to see whether any of the items have been deleted from the old test. If items have been deleted, find out why and make adjustments, as needed.
How must you punctuate item stems?
|
Stem Type |
Punctuation Type |
|
Question |
Punctuate a question with a question mark. Capitalize the first word of each option. |
|
Imperative |
Use a period at the end of an imperative statement. Capitalize the first word of each option. |
|
Open stem |
Omit punctuation at the end of an open-stem question (incomplete statement). Do not capitalize the first word of an option unless that word is a proper noun. |
What are the rules for typing options?
|
Punctuation |
Put a period at the end of each option. This is critical. |
|
Returns |
Put a hard return at the end of each option. This is particularly important after the “d” option. |
Send a minimum of two items per lesson—two items is a bare minimum. Send more items per lesson if a course is short, if a lesson is long, or if information is concentrated.
WARNING: Never send fewer than two items per lesson. Never send fewer than 75 items per volume. Sending too few items can cause us to stop production on your project.
Do not send more than 200 items for a volume. Since we never make tests longer than 124 items, we cannot use more than 200. If you have a wealth of material, concentrate on writing the best 200 items possible. If you must write more than 200 items, consider revising your volume.
You
may want to copy the checklist below to keep by you when you’re writing multiple-choice items. The
checklist is based on the guidelines just discussed.
|
Checklist for Writing Multiple-Choice
Items |
|
|
|
Is this item supported by lesson material—not on information in introductions? |
|
|
Does this item relate directly to the learning objective expressed in the topical statement? |
|
|
Is the text support clear? |
|
|
Is this item the next logical question, given the text support? |
|
|
Does the stem state a central problem? |
|
|
Can this item stand on its own in a course examination? |
|
|
Is the item clear, brief, and direct? |
|
|
Are the distractors convincing? |
|
|
Are the options parallel in category, structure, and length? |
|
|
Did I italicize key words (best, most, least, not, except, never, etc.) in the stem? |
|
|
Did I state the problem in a positive way, avoiding confusing negatives? |
|
|
Is each option grammatically consistent with the stem? |
|
|
Is only one option a correct response? |
|
|
Have I made sure not to repeat key words from the stem to an option? |
|
|
Did I avoid using “all of the above” and “none of the above”? |
|
|
Did I avoid using “both a and c” and “both c and d”? |
|
|
Did I avoid using contractions? |
|
|
Did I arrange numeric options in order? |
|
|
If this item is from an old URE or CE, is it a good item? |
|
|
Is the stem punctuated properly? |
|
|
Is there a period at the end of each option? |
|
|
Are there at least two items per lesson and at least 75 items for the volume? |
| Are there no more than 200 items in all? | |
You must send your multiple-choice
items in a test item bank.
See appendix (contents of file u0.doc) The file name for the bank is U0.DOC (that is
U-ZERO.DOC). The U0.DOC is not based on an AFIADL template. It is a straight
Word file. The list given here presents the mechanics of developing the test
item bank.
Create a new file. Name the file U0.DOC. This file can have formulas created in Microsoft Equation Editor.
On the first line, type the course number and volume, such as 2A177–01.
On the next line, type topicid, the first topical statement number, and a period. Use uppercase or lowercase, but use zeros for numerals, not capital O’s.
On the next line, type beginitem ans=a (b, c, or d) endid.
On the next line, type or copy from another file the item stem and the four options. Make sure that Word's automatic numbering feature is turned OFF.
On the next line, type enditem.
Leave the next line blank.
Repeat appropriate steps above for the remaining test item bank questions.
You may type comments or instructions in between “enditem” and “beginitem” using brackets ([[[), as in the examples that follow. Leave one blank line.
The beginning of the test item bank (U0.DOC) should look something like this example:
2A177–01
topicid001.
beginitem ans=b endid
Which of these is not a benefit of stock funding
depot-level reparables (DLR)?
a. Reductions in levels of
required inventory.
b. Increased carcass return rates.
c. Decreased carcass return rates.
d. Increased material availability.
enditem
[[[use next item on CE
beginitem ans=c import=i endid
Under the depot-level reparables (DLR) stock fund concept, what price is charged to your unit funds when a serviceable asset is issued from supply?
a. Standard price.
b. Carcass price.
c. Exchange price.
d. None; there is no charge.
enditem
topicid002.
beginitem ans=d endid
Under the depot-level reparables (DLR) stock fund concept,
what price is charged to your unit funds when an unserviceable asset is issued from supply?
a. Standard price.
b. Carcass price.
c. Exchange price.
d. None; there is no charge.
enditem
If you wish, you may show the relative importance of each test item in its parameter line:
· Import=I for an important item.
· Import=C for a critical item.
Do not mark routine items in your U0.DOC file. Items not marked “C” or “I” will default to “R.” Here are some examples of routine items:
· Responsibilities of other people and organizations.
· Most definitions.
· Very general or very specific information about a topic.
· Operation of something when the student is not directly responsible for procedure, process, or maintenance.
This designation is reserved for information, concepts, procedures, or processes that are important for the student to know and that relate in a direct way to the student’s job.
Example:
beginitem ans=a import=i endid
This designation is reserved for procedures or processes that, if not followed, could cause serious damage or harm to equipment, property (like software), or people. It’s probably something you want the student to commit to memory. A “critical” item is so important that you want to assure coverage in the self-test questions, URE, and CE. Use this designation very rarely.
Example:
beginitem ans=b import=c endid
You may find formulas, graphics, and tables to be helpful in testing. Be sure you know how to present them.
Use
your keyboard to write a simple
formula
(a2 + b2 = c2)
into the stem or an option of an item in the U0.DOC file. Use Microsoft
Equation Editor to build more complicated formulas and insert them directly
into the U0.DOC file.
You may be able to use your keyboard to create simple illustrations for use in a stem or option in your U0.DOC file—angles, for instance: Ð.
You must not insert the complicated illustrations we call graphics into your U0.DOC file. Instead, send them as you would normally (hard copy, electronic copy, reference in the legend). If you want to use a graphic in the URE that you used in the text, simply refer the student to the applicable figure number in the stem of the question. We will not duplicate a figure that appeared in the text in the URE portion of a unit. If you also want that item to be used in the CE, so indicate by putting instructions in brackets between the items in the U0.DOC file. The ISS will then link or import the graphic in the proper place on the CE.
Example:
enditem
[[[use next item on both URE/CE
beginitem ans=a endid
In figure 1-4, the bias for transistor Q1 is provided by
a. R1.
b. C1.
c. L2.
d. T3.
enditem
To use a graphic that you did not use in the text, send a hard copy and an electronic copy, and give a reference in the legend using T–1 for the first graphic, T–2 for the second, and so forth.
Example:
enditem
[[[next item URE only (or CE only)
beginitem ans=a import=i endid
In figure T-1,
the bias path for transistor Q2 is traced from
a. ground through C1 and R1.
b. Vcc through
L1 to ground.
c. Q1 to R2 and C2.
d. Q2 to ground.
enditem
Do not insert any tables (whether developed by the graphics shop or by you in Word) into your U0.DOC file. If you refer to a table from the text in a URE item, put a note in the text beside the legend. This alerts the ISS not to delete the figure reference.
Figure 1–5. IMSC mode switch versus ADI displays. [[[used in URE
If you send new tables for the URE, give each table a test figure number (T–1, T–2, etc.), even if you key it in yourself in a separate Word file.
To identify text support of your multiple-choice item, put the hidden word TEST in your Word file immediately after the end of the sentence or sentences having the support. To do this, use the Test Mark button on the toolbar called Test Related Stuff to put in TEST automatically.
To use the menu, click on Format, Font, and select Hidden before you type in the word TEST in all caps. When you finish typing TEST, type a space, and deselect the hidden font using the same drop-down menu. To see hidden text, click on the Show/Hide button (¶) on the Standard toolbar.
To use keyboard shortcuts to invoke a hidden font, press Ctrl+Shift+H, type in TEST, and press Ctrl+Shift+H again to turn off the hidden font. Again, to confirm you have typed the hidden text correctly, you may click on the Show/Hide button (¶) on the Standard toolbar.
This is not
a requirement. Generally, the marking
procedure outlined above will meet your needs. This alternative merely adds the
use of highlighting color and a three-digit number to correlate each item in
the U0.DOC file to its support in the unit file.
To use this method,
you can adapt the procedures just given or make three shortcut buttons for your
toolbar. The buttons you will need are for the Hidden command, the command for
the Color to highlight your test references,
and the Black Color command. Use the Black Color button to change to black type
after using the Highlight Color button. To
build your toolbar buttons, follow these steps:
Click on Tools
on the menu bar.
Click on Customize.
Click on the
Toolbars tab.
Click on All
Commands in the Categories window.
In the Commands
window, click on Color.
In the lower right
corner under Color, click on Black.
Move back to the
Color command. Click, hold, and drag the
command (using your mouse) to the toolbar area where you want it; then release
the mouse button.
Click on Edit
on the Custom Button menu.
Click on the black
color, and use the left mouse button to color in the picture area or draw a
pattern.
When completed,
click on OK.
Do steps 5 through 9
for the highlight color you’ve chosen. (Pick a color—other than black,
obviously—that will show up well.)
Move through the Commands
window and click on the Hidden command; now click, hold, and drag the command
(using your mouse) to the toolbar area where you want it; then release the
mouse button.
Click on a premade
button and click on Assign, or click on Edit and make a button.
(Most people use the fifth button in the third
row of the premade buttons.)
When completed,
click on Close.
Now you’re ready to go to work. First click on the
Show/Hide button (¶)
on the toolbar so that you can verify your entries as you work. Type a
three-digit number in hidden text at the end
of the stem of each
item in the U0.DOC file. Start with the first test item in lesson number 001
and continue to 999, as needed. Type TEST and the proper three-digit
number in hidden text at the end of the sentence or sentences containing the
correct answer to each item in the unit file. (Use the Find function in
the Edit drop-down menu to look up the text support for each
test item.) If the correct answer is spread out over two or more places in the
lesson, type TEST and the number in hidden text
in each place.
When you’re entering the text support marker in the unit file,
click on the Hidden button and then the Highlight Color button, and type TEST
and the three-digit number. You will have to click on these two buttons for
each entry you make, for your computer will automatically return to normal
editing mode when you click in another position in the file. (If you put two
markers after the same sentence, you will not have to click the Hidden and
Highlight Color buttons for the second marker. The second marker will pick up
the attributes of the first one.) If you must insert additional text immediately after a highlighted
three-digit number, first click on the Hidden button and the Black Color button
to return to normal editing mode. Otherwise, the added text will be
highlighted.
If you click on the Highlight Color button or Black
Color button without first clicking on the Hidden button, you will insert characters into the unhidden
text. If you click on the Hidden button without then clicking on the Highlight
Color button, you will insert hidden black characters. As a fail-safe, you may
choose to page through your U0.DOC and unit files in normal (unhidden)
mode after you are through inserting markers.
How will your counterparts at the Institute use
this method to verify text support of your
test items? First, they will open the unit file and click on the Show/Hide button
(¶). Then, to find text support for a specific item, they will type its
three-digit number in the Find What window. To find text support for all of
the items, one after the other, they will go to the top of the file, type TEST
in the Find What window, and click on Find Next repeatedly. Each
time the computer finds TEST, they can compare the text support against
the test item identified by the corresponding three-digit number.
If you
want to identify each item with a three-digit code but don’t want to go to much
trouble to do it, just leave off the color business:
(1) give each item its own three-digit number in hidden text at the end of the
stem, and (2) type TEST and the same three-digit number in hidden text
at the end of the sentence or sentences supporting the item. This way, you lose
the high visibility of the markers and you can’t find them so quickly just
scrolling through the file, but the Find function in the Edit
drop-down menu gets you unerringly to the next
hidden TEST anyway.
If you use this alternate method and want to highlight the
marker, you may find it easier to enter TEST plus the three-digit number
in each appropriate place in the text, use the Find function to locate
each TEST, and apply the Highlight Color by clicking the Highlight
button. (The Find function highlights TEST.) Black numerals following a highlighted TEST are
easy to read, so you will not really need to highlight them.
Here’s a shortcut for inserting a hidden, highlighted TEST: (1) Click the Hidden button (or press Ctrl+Shift+H), (2) click the Highlight Color button, (3) type TEST, (4) highlight (double-click on) TEST, (5) copy TEST (Ctrl+C if you don’t have a Copy button), and (6) paste in TEST each time you want to mark text support for an item (Ctrl+V if you don’t have a Paste button). This shortcut also works for pasting in a black TEST (or anything else), of course.
No matter which text support identification method you use, put each completed unit file into normal (unhidden) mode and search for TEST. If you have failed to hide a marker, the search function will highlight it for you. All you have to do to hide it is to click the Hidden button or press Ctrl+Shift+H. If you have not failed to hide any markers, your computer will search a file in seconds. Visually scanning the U0.DOC file in normal (unhidden) mode should let you find any errant markers there.
Once you develop topical statements and multiple-choice items, you develop self-test questions that exhaust the pertinent material in each lesson. You should sequence self-test questions in the same order in which the text supports them.
Self-test questions may be short-answer, matching, or problem/situation
questions. They should help students
reach the desired level of proficiency. Some questions may deal with lower
proficiency levels if they enable students to
reach the target level. You may choose to use figures with your self-test
questions. The example here shows the relationship of topical statements,
multiple-choice items, and self-test questions.
|
Topical statement: |
719. Converting binary and decimal numbers |
|
Multiple-choice
item stem: |
What is the binary equivalent of 634(10)? |
|
Self-test
questions: |
1. How can you convert mixed decimal numbers to the equivalent binary form? 2. What method should you use to convert decimal fractional numbers to binary fractional numbers? What number should you use as the multiplier? 3. What is the decimal equivalent of 10011010? 4. What is the binary equivalent of 1992? |
We use only these types of self-test questions—
· Short-answer.
· Matching.
· Problem/situation.
Do not send self-test questions in completion (fill-in-the-blank) or true-false format, or in any variation of these formats. Multiple-choice items are reserved for the URE and CE.
A short-answer
question asks
students to recall facts, ideas, or principles; make comparisons; define terms;
or give examples. Limit answers to a word, a brief list, a phrase, or a short
statement. When you write short-answer questions—
· Make each question clear and concise.
· Use the active voice.
· Address essential information only.
· Avoid contractions.
Suppose you want to write a short-answer question on AC and DC motors used to drive pumps. You want students to compare the two types. Instead of asking a general, open-ended question, define the type of response you get by asking for a certain type of comparison.
Do not ask this question:
What’s the difference between the AC motor and the DC motor as power sources for pumps?
Limit the response necessary by asking this:
What are the advantages of using an AC motor for driving pumps?
The first question is too broad. Good students can write at length on it. The second question tells the students exactly where to begin and where to stop. It sets limits to the response. Of course, there are many types of short-answer questions. Here are some examples of short-answer questions for a specified topical statement:
|
010. Purpose and procedures of SORTS
reporting |
|
Use matching questions to
show comparison, identification, or definition of basic facts. Matching
questions have two columns of related topics. Students match items from column
A that best relate to items listed in column B. Apply these rules when you write matching items:
· Give simple, direct instructions. State whether options may be
used once, more than once, or not at all.
· Use homogeneous item groupings.
· Put longer items in column A; put shorter items in column B.
· Use parallel syntax. (Any item in column A should seem to go
with any item in column B.)
· Use plausible distractors, but do not try to trick students.
· Use the buttons on the special matching toolbar (Carole’s
Matching Question Tools) to format matching exercises.
The example here is a good matching question. It addresses the topical statement by asking students to identify the functions of push buttons and indicators. You can use the special toolbar for matching questions for formatting most of your matching questions.
211. Functions of paper tape reader push buttons and indicators
Match each push button and indicator in column B with its function in column A. Items in column B may be used more than once or not at all.
|
Column A ____ (1) Stops the paper tape reader.
Matching Col A style ____ (2) Connects power to the unit. ____ (3) Permits an operator to load
tape. ____ (4) Engages the rewind
operation. ____ (5) Glows when the tape ends. ____ (6) Allows continued operation. |
Column B a.
Power. Matching Col B style b.
Operate. c.
Load. d.
Forward. e.
Rewind. f.
Backspace. g.
Halt. |
NOTE: Column headings are italic, 10-point Times New Roman, left.
To format the answers to a matching exercise, use the Answer style for the first line of the answer and the Subordinate Answer style for subsequent lines. For example:
211 STQLO style
1. (1) a. Answers style
(2) b. Subordinate
Answer style
Use problem/situation questions to measure students’ understanding of principles or ability to solve problems or apply principles. These questions go beyond recall or recognition. They require students to consider various factors and arrive at solutions—whether they deal with mathematical or simulated real-life situations.
When you provide a situation that asks students to draw from everything they learned in the lesson, you help them apply their knowledge and show they understand the information as a whole. Another advantage of the problem/situation question is that you can measure learning in several ways. You can change the variables to alter situations so that the students can make decisions about the same facts by evaluating different conditions. You can create problems to cover such areas as mathematics, troubleshooting, and planning. This kind of question is effective, efficient, and flexible. Use these guidelines to develop problems:
· Make the problem realistic, practical, and applicable to your
career field.
· Be sure the information is accurate.
· Give enough data for students to find a solution.
· Do not introduce unnecessary data as distractors.
Here is a math problem in a self-test question:
412. Converting decimal numbers to binary,
octal, and hexadecimal notations
1. Convert these decimal numbers to equivalent binary, octal, and hexadecimal numbers.
|
Decimal |
Binary |
Octal |
Hexadecimal |
|
a. 80 |
|
|
|
|
b. 132 |
|
|
|
|
c. 144 |
|
|
|
|
d. 256 |
|
|
|
The answer to this self-test question should be the same table completed.
The mathematical problem, in particular, forces students to do operations accurately and in a definite sequence. The simplest form of mathematical problem provides a body of data and requires the direct application of one or more related formulas. For example, look at these two sets of self-test questions:
613. Estimating labor for a simple plaster construction job
How many labor hours are needed to plaster a 12-by-12-foot room having walls 8 feet high and a door and window allowance of 87 feet?
How long should two people take to apply 1,000 square feet of stucco to a concrete masonry wall?
814. Statistically analyzing test scores
You have given a course examination to your students. When you tabulate their scores, you get this frequency distribution:
|
Scores |
Airmen |
Determine: |
|
60–69 |
4 |
(1) N. |
|
70–79 |
7 |
(2) Mode. |
|
80–89 |
8 |
(3) Median. |
|
90–100 |
5 |
(4) N/2. |
Solve for the elements listed in the “Determine” column of this table.
These examples ask questions students can solve by using formulas or applying principles. They are, in a sense, cut and dried. Manipulating a few simple formulas cannot solve most problems, including those involving mathematics. They are questions that simulate real life and pose problems for which students must consider possible solutions, selecting one they think is best. Such real-life items adapt particularly well to troubleshooting and planning exercises.
We are using the term troubleshooting to include equipment, organizations, and people. Writing good troubleshooting and planning questions may be harder than writing other types, but here are some tips to make your task simpler and your results rewarding:
· Make the situation realistic.
· Make it relevant to the principles or ideas developed in the text.
· Make it new. Do not repeat situations discussed in the text.
· Be clear and specific.
· Make sure the situation gives a good basis for the solution, but limit it so that it does not pose too many problems.
· Develop several problems for each situation.
· Make each problem independent of the solution to any other problem.
The number of problems and the difficulty of the material determine the amount of information you give. Situation questions are based on actual or hypothetical circumstances or data. They can be simple or complex and can provide the basis for one or more problems.
Problem/situation questions have many advantages in our courses. They approach real life more closely than other types of questions and help you put the students in the job. Items that force students to use the text, the figures, and reasoning ability make good troubleshooting exercises. For example:
015. Finding
the cause and prescribing the remedy for malfunctions observed in an
operational test of propellers on a multiengine aircraft
Situation: You are conducting an operational test of the ...propellers on a multiengine aircraft. You find these conditions: (1) oil leakage on the barrel halves of a propeller; (2) one propeller operates normally to the full increase rpm blade angle, but the blades do not go beyond low pitch during reverse operation; and (3) one propeller does not stabilize at the required minimum rpm.
What unit could permit leakage on the barrel halves of the propeller?
For
condition 2, what are the possible sources of trouble? Which are most likely?
For condition 3, if you find the trouble to be in the stepmotor head, what should you do to correct the trouble?
Answers to self-test questions
Answers are essential to the teaching of self-test questions. Students need an easy way to check their progress and to find out what they need to restudy. Students get quick feedback and reinforcement when they check their responses with the answers.
Writing answers demands as much care as writing questions. Do not ask questions you cannot answer from the information in a single lesson. Just as your questions should have the same meaning for all students, so should your answers. If possible, have someone who knows your subject check your questions and answers. Follow these rules when you develop answers:
· Don’t repeat the question in the answer. If you are having the student complete a table, you must put the completed table in the answer.
· Don’t introduce new information. All lesson information should appear in the text of the lesson itself.
· If the answer needs more information than is in the text, put the information in the text or rewrite the question.
· Answer each question fully.
· If there is a sequence of requirements in the question, answer in the same sequence.
Put the answers to
self-test questions at the end of each unit, just after the last set of
self-test questions and just before the unit review exercises. Separate the
answers to self-test questions from the rest of the unit by a solid line and a
title, which you can insert by clicking on the Answers Heading button on the
custom toolbar.
The format for the answers themselves makes checking the answer to any specific self-test question easy for the student. First, insert the lesson number (topical statement number) on a line by itself, with no period. Then, to format the lesson number, click on the STQLO style button on the custom toolbar. The answers themselves should have no styles other than the Answer style or the Subordinate Answer style applied.
A couple of examples may clarify the format rules. A self-test question may call for the student to respond with an answer having several parts. For instance, if a self-test question asks this:
216. Movement
1. State the
three rules of movement.
Format the answer like this:
216
1. (1) Movement must be motivated.
(2) Movement begets movement.
(3) Movement must be
believable.
Another example of a need for a multipart answer is for a self-test question that has several parts to which a student must respond. If a self-test question is formatted like this:
317. Notification
6. Explain each of these exceptions to hazardous materials
emergency notification
requirements:
a. On-base.
b. Permitted and in compliance.
c. Excluded.
Format the answers like this:
317
6. a. Any release resulting in exposure to...
b. Any discharge, emission, or other...
c. Any other release excluded under...
See appendix for more examples of answer formats.
The CE is a tool to measure student
achievement. Studying the text, self-test questions, and the URE prepares
students for the CE.
Keep copies of CEs (hard copies and
disks), CE item
analyses, and CE answer keys in locked storage. Do not let unauthorized people inspect or review CEs. Annually,
inventory all CEs, item analyses, and answer keys. Inventory them again when you leave your job. Show
the date each document was received, the course number and form number of the
document, and the type and date of disposition of any CEs removed from your
holdings. Keep data on such removed CE documents for 1 year past the date of
disposition.
A statistical analysis of each examination, an item analysis, helps us judge both student performance and individual items on the examination.
The Institute’s computer scores answer sheets and collects data on performance. After the computer has processed the answer sheets from the first 51 students who take a new or revised edition of a CE, it produces an item analysis. An ISS uses the item analysis to identify possible faulty items. (The ISS sends you a copy of the item analysis for your action.) The computer produces a second analysis after accumulating 123 responses. The second product gives a more accurate analysis of how the examination is functioning and shows the effect of changes or deletions made after the 51 sample.
The item analysis is valuable. You need to understand it because you will get one for most CEs, with our comments, for your review and evaluation. You must use the item analysis to decide when an item has weak text support, a miskey, two correct answers, or other problems. Of course, you must use item analyses of CEs to evaluate items before you reuse them.
Here are explanations of the summary entries on an item analysis. Study these paragraphs, keeping in mind that the
format for this
summary may change from the one found on the item analyses in your
files. You will be most concerned with the number in sample (nr in sample),
number of items (nr items), average ease (avg ease), average discrimination
index (avg item
disc), number of failures (nr failures), and failure rate.
Nr in sample—the number of students, either 51 or 123, who took the examination. These particular numbers are used to calculate the discrimination index because they are divisible by 3. Scores from three subgroups of students (subgroups based on overall CE performance) are used in calculating the discrimination index.
Nr items—the total of items scored on the CE at the time of the computer printout. The number may be the original number or a reduced number because of deletions.
Reliability index—a statistical estimate of how consistent results should be on successive administrations of the CE. These are reliability guidelines:
· Poor: less than 79.
· Good: 80–89.
· Excellent: 90 and above.
Avg item disc—the average of the individual item discriminations. It is a measure of how well the examination discriminated between the better and poorer students. A high positive discrimination index shows many or all of the students in the highest subgroup answered the items correctly, whereas many or all of the lowest subgroup did not.
Mean—the average number of correct responses for all students who took that form of the CE.
Standard deviation—a measure of the variability of clustering of the raw scores. If there are many scores at both the high and low extremes of the range, the standard deviation will be higher than when most of the scores bunch together near the mean.
Standard error of measurement—a statistical indication of the amount of variation to be expected in the test scores (if the same students took this test again). This estimate, derived from the test reliability and the standard deviation, shows the range in which a student’s true score probably falls. For example, if a student achieved a raw score of 97 correct answers and the standard error of measurement was ±5, the student’s true score probably is in the range of 92–102. Obviously, the lower the standard error of measurement is, the better.
Pass/fail point—percentage expressed as the minimum number of items the student must answer correctly to pass the course. A score of 65 percent correct is passing for most CEs.
Range—the lowest and the highest scores achieved so far. The range is also reflected in the frequency distribution of the scores.
Nr failures—the number of students in the sample who failed this form of the CE. Students who fail one form of a CE may take another form. Students who fail both fail the course.
Failure rate—percentage of students in the sample that scored less than 65 percent on this edition of the examination. Edition failure rates are, of course, much higher than course failure rates, since each student may take another edition of the examination if he or she fails the first time. If the fail rate is higher than 35 percent, there is a problem—maybe with the examination or maybe with the presentation of the subject matter in the text.
The guidance here should help you to read the individual item statistics. Again, the format for item analyses may change, but the basics will remain. These are item analysis headings:
· ? NBR—Item number on the CE.
· ITEM EASE—Item’s ease index.
· ITEM DISC—Item’s discrimination index.
· A, B, C, D—Options for the item.
· COR ANS—Correct answer.
The first figure in the column under each letter (A, B, C, and D) is the number of students who selected that option. The second figure is the discrimination index of the option. If the upper third of the group of students tends to choose a response and the lower third does not, the index is high and positive. This is fine if that response is the right one. Ideally, all correct responses discriminate positively (+) and all incorrect options discriminate negatively (–). The maximum possible discrimination index is +1, and the minimum is –1; but these limits are almost never reached.
It is impossible for a very easy item or a very difficult item to have a high discrimination index. If all students answer an item correctly (100 percent ease), or if they all miss it (0 percent ease), the item has no discriminating power, but this fact, in itself, does not indicate a “bad” item. If the item addresses an important safety practice, you may wish all students to answer it correctly; your test item may serve as reinforcement of essential information.
An item with 0 percent ease may be fine, but simply miskeyed. (Acceptable ease for an item can go as low as 30 to 40 percent, if not too many items are in that range.) The discrimination index guidelines are as follows:
· Fair: +24 to +26.
· Good: +27 to +33.
· Excellent: +34 and above.
By looking for items with ease under 50 and for items that discriminate negatively, we frequently can spot miskeys and defective items. We can correct miskeys and delete faulty items.
Use the information to maintain your current CEs, but also use it in planning your next revision. If you plan to reuse items from your current CEs, check the ease and discrimination index for each item. Weed out or improve items that are too easy or too hard, items with weak distractors, and items flawed in other ways.
When you get item analyses, read your team’s comments. Your ISS normally highlights for you any items having one of these problem indicators:
· Ease index under 50. (Means question is quite difficult.)
· Negative item discrimination. (Means poorer students got this question right, while better students selected a distractor.)
These unusual statistics suggest a problem, but they are indicators only. Do not delete or change an item based solely on statistics. Instead, examine the item and its supporting text. Be sure to check highlighted items for miskeys, ambiguous stems or options, and multiple correct answers. Check the text support to be sure each item is supported clearly and the support is not contradicted elsewhere in the text. Based on what you find in your research, decide what (if any) action to take. Do not delete an item just because it’s difficult.
Here are the most common actions authors take in response to an item analysis:
|
Delete the item |
If text support is weak. Note the text deficiency so that you can rewrite it at revision time. If more than one option is correct. Note the conflict so that you can correct it at revision time. If there is no correct option. Note the deficiency for correction at revision time. |
|
Correct the item |
If there’s a minor wording error in the stem. |
|
Rekey the item |
If the wrong option is keyed. |
|
Delete, correct, or rekey URE items |
If some bad CE items appear also on UREs. |
Depending on the information you find when you look at the
tests and text, you may need to delete or rekey an item or send errata to the text. Maintaining CEs is a cooperative effort—we need your
help. When you get student inquiries that affect test items, coordinate the
action with your Curriculum team and follow up with written notification. We may call you about
changes to test items based on student inquiries, item analyses, errata, and supplements. Because we are now using the print-on-demand
process for CEs, we can edit CE files to correct misspellings and other minor errors.
Time does not permit us to do major rewrites of items.
NOTE: You may make only very minor word changes to a URE or CE item in print. Tell us in writing if you find miskeyed items and how to rekey them. You may fax your letter to us at DSN 596–3208.
Rekeyed UREs will not be
included in student course packages until the next prepackage is done at the
warehouse. Only then is a new scoring key generated. The Institute periodically
prepacks several months’ supply of course materials, the exact number depending
on projected enrollment.
You know the impact of errata and supplements on your testing base since you initiate those actions. Along with any errata or supplement you send to the Institute, send instructions on changes to examinations. Unit 6 discusses this subject in detail.
To delete, rekey, or change an item, please fax the changes to the Institute.
|
Fax instructions: |
1. Type your request on letterhead. 2. Send ATTN: DCC. 3. Use this fax #: DSN 596–3208. 4. Identify the exact examination to correct, including edit code. 5. Identify which CE items to delete and which to correct. 6. Identify any corresponding URE items to delete or correct. |
As you write your multiple-choice items and self-test questions, remember these points:
· Multiple-choice items in the URE continue and extend the instructional process—not by introducing teaching points, but by increasing the scope and complexity of students’ experience over the same information. These items highlight the key teaching points.
· Self-test questions should carry a major teaching load in our courses. These questions should identify for the students all the information they need to retain from a lesson segment. Self-test questions should exhaust the instruction.
· CEs measure the extent to which students have retained important teaching points. CEs also reinforce teaching through the feedback of results to the student.
When you maintain your course examinations, consider all the elements that may affect student performance. Problems in testing may come from the text questions or from the text.
For questions on the content of this page, send E-mail to:
DSN: 596–4420 Comm: (334) 416–4420
Pagemaster
DSN: 596-4277 Comm: 334-416-4277
Webmaster
DSN 596-3795 Comm 334-416-3795