Connect with UNB
Your assessment methods should assess the outcomes or learning objectives (depending on the instruction organizational model you’re using) at the level indicated by the outcome. For example, if the learning outcome was at the application level, then the instructional activity would be at that level (or lead up to and end at that level—perhaps discussing material presented in class, then use peer instruction, then scenarios). The assessment, in turn, should also use scenarios, either as assignments or in tests or an exam.
Multiple choice can be used for any learning level, from knowledge and comprehension to application and synthesis. Typically the higher level questions involve scenarios or case studies, and the answer options represent response options, with the distracters (incorrect answer options) based on common misconceptions, so as to be plausible and thus keeping the question difficulty at an appropriate level.
Objective questions, which have specific correct answers (typically multiple choice, true/false, fill-in-the-blank), tend to take a long time to create but a short time to mark. Subjective questions, in which students demonstrate knowledge and skills by making statements and supporting them with evidence and argument, tend to the be opposite—less time to create, more time to mark.
The ideal situation for objective questions is to have a test bank made up of a large number of questions on each test topic, and to randomly select some from each topic for a particular test or exam. It takes many years to build such a test bank. It will be tempting to use ready-made, publisher-supplied test questions. Do so only if you can say “yes” to all the following questions:
Research indicates that the optimal number of answer options for multiple choice questions is three—that is, there is no appreciable increase in question quality by having four answer options compared to three. Question quality depends on answer option plausibility, and it is difficult to create more than 2 plausible distracters. So, as the number of answer options increase, general knowledge and question structure logic may become sufficient to eliminate distracters and find the correct answer independent of subject knowledge, thus reducing question difficulty and quality (Rodriguez).
Suggestions: Take a few minutes after each class and/or preparation session to write a few questions or question ideas, while the information is fresh in your mind. Have students suggest questions, or write some as a class activity, from time to time.
Definition: “Distracters” are incorrect but plausible answer options in a multiple choice question.
Principle 1. The stem must present a single, definite, meaningful statement to be completed or answered by one or more of the answer options.
| This | NOT this | Explanation |
|---|---|---|
Which TWO of the following definitions of culture are the MOST accurate?
|
Culture is best defined as:
|
Directions in the stem should be clear and unambiguous; there should be no doubt about the nature of the task. In the NOT example, the superlative “best” in the stem implies one correct definition for the answer, yet the options all contain two definitions of culture. |
| Which treatment for AIDS is the MOST effective? | Which of the following treatments for AIDS has been found to be most effective? | The good example makes it clear that one answer is being sought. |
| Which of the following is the major cause of death in the U.S. for children over one year old?” | The major cause of death for children in the U.S. over one year old is which of the following? | In the NOT example, “Which of the following” appears at the end. WH questions, such as “which” or “what,” are easier to process when they are “fronted.” Also, “non-fronted” WH questions tend to become dense with information, which increases the structure complexity and thus the decoding task. |
| In the immune system, self-antigens are sometimes mistaken as foreign to the body. When this happens, antibodies form against the self-antigens. Which statement describes this inappropriate reaction by the immune system? | An inappropriate reaction by the immune system (in which antibodies form against self-antigens, mistaken as foreign) best describes: | The NOT example has embedded and reduced clauses, which require students to read/ re-read the question to understand what is being asked. |
Principle 2: Avoid irrelevant sources of difficulty, such as insufficient information in the stem, lack of clarity regarding the task, and linguistic/ structural bias, and unclear wording.
| This | NOT this | Explanation |
|---|---|---|
| Which of the following statements about the Maritimes is TRUE? A. Lumbering is the primary industry. B. The annual yearly rainfall is 60 cm. C. Annual income is higher than the US average. D. Halifax is the most populous city.* |
In the Maritimes A. Lumbering is the primary industry. B. The annual yearly rainfall is 60 cm. C. Annual income is higher than the US average. D. Halifax is the most populous city.* |
Need enough information in stem to know what is being asked. |
| A patient has respiratory disease. He has a shift to the left in the oxygen- hemoglobin dissociation curve. What does this finding indicate? | A patient with respiratory disease has a shift to the left in the oxygen- hemoglobin dissociation curve. The nurse recognizes that this finding indicates that… | In the good example, the intent of the question is easier to identify and it is clear that students must choose the best answer from more than one correct answer. |
| Mr. R. is scheduled for lithotripsy. The nurse develops a teaching plan for Mr. R. How should the nurse describe this procedure in the nurse’s teaching plan? A. Surgical removal of stones. B. Capture of stones via scope. C. Fragmentation of stones by electrical charge. D. Dissolution of stones with medication. |
Mr. R. is scheduled for lithotripsy. The nurse develops a teaching plan in which the procedure is described as the: A. Surgical removal of stones. B. Capture of stones via scope. C. Fragmentation of stones by electrical charge. D. Dissolution of stones with medication. |
Unnecessary use of passive voice in the NOT example. It is easier to decode active voice sentences. |
| Clients with which of the following conditions are MOST likely to experience water intoxication? A. SIADH. B. Head trauma. C. Lung cancer. D. Hypothyroidism. |
In which of the following situations should the nurse have a high index of suspicion for water intoxication? A. Persons experiencing SIADH. B. Persons who have experienced head trauma. C. Persons with a diagnosis of lung cancer. D. All of the above. |
In the NOT example, “high index of suspicion” is not common terminology. This is also an example of unnecessary nominalization, in which the noun form of a word (suspicion) is used instead of the verb form (suspect). Also see #5 and #8. |
| Which of the following statements is FALSE about the American Nurses Association? It…. A. Is a professional organization whose membership consists of physicians, nurses, and citizens interested in improving health care. B. Works to improve the quality of nursing practice. C. Identifies the appropriate academic credentials for entry into nursing practice. D. Fosters the development of nursing theory by promoting nursing research. |
All of the following are correct statements about the American Nurses Association EXCEPT: A. Is a professional organization whose membership consists of physicians, nurses, and citizens interested in improving health care. B. Works to improve the quality of nursing practice. C. Identifies the appropriate academic credentials for entry into nursing practice. D. Fosters the development of nursing theory by promoting nursing research. |
The most frequent type of “tricky stem” is the reverse multiple choice item, which asks for the one incorrect answer rather than the one correct answer. It is more difficult to process than the “correct-answer” format as the negative is often missed. Often it is used because it is easier to create one incorrect alternative rather than three plausible distracters. Reverse MC items should be used only when students need to know incorrect practices, not for the sake of question-making convenience. |
| Which of the following is characteristic of an adult who has the potential for abusing children? A. Experiences low levels of stress. B. Is socially isolated. C. Lives in a stable environment with good support. D. Had a trouble-free childhood. |
Which of the following would not be a characteristic of an adult who may have potential for abusing children? A. The person is challenged by chronic stress. B. The person is socially isolated. C. The person is in a stable environment with good support. D. The person was treated abusively as a child. |
Direct forms of verbs should be used to avoid confusion and ambiguity. In the NOT example, “would” suggests an unnecessary conditional and “may” is vague. |
Principle 3: All answer options must be grammatically consistent with the stem. Without grammatical consistency, the answer may be signaled inadvertently. If each option begins with the same word, include that word in the stem.
| This | NOT this | Explanation |
|---|---|---|
| An electric transformer can be used to: A. Store electricity. B. Increase the voltage of alternating current.* C. Converts electrical energy into mechanical energy. D. Change alternating current to direct current. |
An electric transformer can be used A. For storing up electricity. B. To increase the voltage of alternating current.* C. It converts electrical energy into mechanical energy. D. Alternating current is changed to direct current. |
All the answer options must use the same grammatical structure. In the NOT example, all the options should be changed into clauses beginning with "to." Then "to" can be moved to the stem. |
| A medication order reads: ‘Digoxin, 0.125 mg PO qod.’ What is the correct way for the nurse to give the drug? A. Rectally daily before bedtime. B. By mouth every other day. C. By mouth twice a day. D. Intravenously once a week after recording an apical rate. |
Digoxin, 0.125 mg PO qod.” The nurse correctly gives this drug: A. Daily before bedtime B. By mouth every other day C. Twice a day by way of the oral route D. Once a week after recording an apical rate |
Wording should be consistent between the stem and the options and among the options. In the NOT example, “By mouth,” used in Option B, and “by way of the oral route,” used in Option C, refer to the same way of taking the drug. In the revision, “by mouth” is used in both options. In addition, frequency words should be placed in the same order in all of the options. |
Principle 4: Avoid unintended wording or logic clues.
| This | NOT this | Explanation |
|---|---|---|
| The care manager role is demonstrated when the nurse: A. Helps a diabetic client learn to give own injection. B. Meets with the client’s family prior to discharge. C. Organizes and coordinates a client’s plan of care.* D. Changes a client’s wound dressing. |
The care manager role is demonstrated when the nurse: A. Helps a diabetic client learn to give own injection. B. Meets with the client’s family prior to discharge. C. Organizes and manages a client’s plan of care.* D. Changes a client’s wound dressing |
In the NOT example, the stem contains the word “manager” and option C uses the word “manage.” Students are more likely to select C because of the repeated word. In the good example, C uses the synonym “coordinates” instead of “manages.” |
| Which of the following signs and symptoms will a person with pheochromocytoma exhibit? A. Hypoglycemia, lethargy B. Hypertension, anxiety C. Hypotension , lightheadedness D. Diarrhea, abdominal cramping |
A person with pheochromocytoma will exhibit which of the following signs and symptoms? A. Hypoglycemia, lethargy B. Hypertension, anxiety C. Hypotension D. Diarrhea, abdominal cramping |
The NOT example asks for “signs and symptoms” in plural form. Option C, a distracter, is in singular form, making students more likely to eliminate it because it does not flow grammatically from the stem. |
| Which of the following statements is TRUE of nursing interventions with physically abused women? They: A. Recognize that the woman may believe she is the cause of the abuse. B. Make sure the woman has family that she can go to if there is an occurrence of abuse. C. Keep questions of the woman indirect so the woman does not feel discovered. D. Offer the nurse’s personal opinions of the woman’s situation to strengthen the nurse/client relationship. |
Which of the following is true of nursing interventions with physically abused women? A. Recognize that the woman may believe she is the cause of the abuse. B. Make sure the woman has family that she can go to if there is an occurrence of abuse. C. Keep questions of the woman indirect so the woman does not feel discovered. D. Offering personal opinions of the situation is useful to initially develop the nurse/client relationship. | Options are not grammatically consistent or parallel in form: D begins with the gerund, or “-ing” form of the verb, whereas A, B, and C begin with the base form of the verb. |
Principle 5: Keep reading efforts to a minimum by eliminating irrelevant and repetitious text. Such things sometimes give away the correct answer.
| This | NOT this | Explanation |
|---|---|---|
| John F. Kennedy was assassinated in the year A. 1961. B. 1963. * C. 1965. D. 1968. |
John F. Kennedy was assassinated A. in the year 1961. B. in the year 1968. C. in the year 1963.* D. in the year 1965. |
The good example moved the phrase "in the year" into the stem and placed the years in chronological order. |
| What is an important nursing intervention for someone experiencing tuberculosis? To teach the client to: A. Correct hypoxemia by increasing respiratory rate above 30/min. B. Take all of her medications and report to the clinic for regular check-ups. C. Wear an oxygen mask at all times. D. Practice the most effective breathing pattern for “floppy” lung disease. |
An important nursing intervention for someone experiencing tuberculosis includes: A. Teach client to correct hypoxemia by increasing respiratory rate above 30/min. B. Teach client the importance of taking all of her medications and reporting to the clinic for regular check-ups. C. Teach client to wear an oxygen mask at all times. D. Teach client the most effective breathing pattern for “floppy” lung disease. |
Principle 6: Avoid negative statements. If negatives must be used, CAPITALIZE, underline, or highlight the negative term.
| This | NOT this | Explanation |
|---|---|---|
| Which of the following statements about breast examinations is FALSE? A. Postmenopausal women need to do a breast exam. B. The breast tissue is palpated systematically in a clockwise motion. C. A lump in a man’s breast could be breast cancer. D. Percussion is used to further assess any palpable breast masses. |
Which description of the breast examination is true: A. Postmenopausal women do not need to do a breast exam. B. Palpate the breast tissue systematically in a clockwise motion. C. Male breasts are not examined because males can’t develop breast cancer. D. Percussion is used to further assess any palpable breast masses. |
If an item cannot be worded positively, negative words such as “not” or “false,” as well as “true,” should be bold-faced, underlined, and capitalized, to highlight the nature of the task. |
Principle 7: Use only plausible and attractive distracters. Implausible distracters will help give away the correct answer.
| This | NOT this | Explanation |
|---|---|---|
| The vessel which carries oxygenated blood from the heart to the body is called the A. Venule. B. Pulmonary artery. C. Vena cava. D. Ascending aorta. * |
The vessel which carries oxygenated blood from the heart to the body is called the A. Trapezius muscle. B. Forebrain. C. Patella tendon. D. Ascending aorta. * |
In the “NOT” example, only "D" is a "vessel." The corrected version distracters are all vessels in the body. [Note: “pulmonary artery” actually has venous blood because it is taking it to the lungs to be oxygenated.] |
Principle 8: Use as choices "all of the above" or "none of the above," sparingly, if at all.
| This | NOT this | Explanation |
|---|---|---|
| South Africa leads the world in the mining of: A. Bauxite. B. Diamonds. * C. Iron ore. |
South Africa leads the world in the mining of: A. Bauxite. B. Diamonds. * C. Iron ore. D. All of the above. |
D could be replaced by another mineral, or just left out. Students who know that one listed mineral is not mined in South Africa can also eliminate D, making the correct answer choice odds 50/50. |
| Which of the following characterizes a wound that heals by secondary intention (as compared to one that heals by primary intention)? Select all that apply. A. There is formation of more granulation tissue. B. Healing time is quicker. C. Less scar tissue results. |
Which of the following characterizes a wound that heals by secondary intention (as compared to one that heals by primary intention)? A. There is formation of more granulation tissue. B. Healing time is quicker. C. Less scar tissue results. D. None of the above. Or, D. All of the above. |
With “D. None of the above” as an option, students who know that more than one option is correct will eliminate it. “D. All of the above” would also skew results since students who recognize two correct answers will likely pick D without knowing about the accuracy of the third option. |
Principle 9: Ensure that only one response is considered correct by experts in the field. (The example provided is easy so we are all experts in the field and can evaluate its quality.)
| This | NOT this | Explanation |
|---|---|---|
| The ingredient used as a leavening agent is: A. Salt. B. Flour. C. Yeast.* D. Water. Or The leavening agent used in bread is: A. Baking powder. B. Baking soda. C. Yeast.* D. Arrowroot. |
The ingredient used as a leavening agent is: A. Baking powder. B. Baking soda. C. Yeast.* D. Arrowroot. |
All the alternatives are leavening agents. To make a correct question, the specific product to be created would need to be specified in the stem (e.g., The leavening agent used in bread is …) or three alternatives that are not leavening agents would need to be provided (e.g., salt, flour, water). |
Principle 10: Long correct answer: Correct answer is longer, more specific, or more complete than other options
| This | NOT this |
|---|---|
| They don’t have to be exact, and once in a while there may be justification for an exception on the basis of subject matter or logic (provided the length is no clue to the correct answer), but answer options should be close in length. | Don’t slavishly make every answer option always the exact same length, at the expense of clarity, natural expression, or even which distracters you use, just to make the question visually “perfect.” |
Principle 11: Questions should relate to specific learning objectives.
| This | NOT this |
|---|---|
| Questions should: • Be based on work covered in class or readings or assignments, which, in turn, were selected because of learning outcomes or objectives listed in the syllabus. • Test at the learning level indicated in the outcome (see preamble to this section). • Correspond in number to the relative importance and amount of time spent on them in the course. |
Don’t: • Find obscure items in chapters and readings and use them to try to trick students. • Use other people’s questions without checking their fit and quality first. • Ask a lot of questions about a few topics because they are easier to create. |
Principle 12: Use of cultural content that many would not understand.
| This | NOT this | Explanation |
|---|---|---|
| Common objects previously owned by celebrities are often auctioned off at very high prices. Which of the behavioural mechanisms below are people buying such high-priced objects exhibiting? A. Projection. B. Identification. C. Rationalization. D. Reaction formation. |
When Sotheby’s auctioned off items from the Jackie Kennedy Onassis estate, those who paid “top dollar” for items were most likely using the behavioural mechanism of: A. Projection. B. Identification. C. Rationalization. D. Reaction formation. |
There is culturally specific information that not everyone would know: a celebrity, a cultural event, and the idiom: “top dollar.” |
| The nurse is teaching cancer prevention. Which of the following should the nurse identify as the MOST documented carcinogen? | In teaching cancer prevention, one of the following should be identified as the best documented carcinogen. Select the number one culprit. | “Culprit” may be difficult for ESL students. “Best-documented” is subjective. It is also easier to decode active voice sentences. |
True-False questions are well suited for quick checking understanding of a large number of facts, but like multiple choice questions, could potentially be used to test any learning level. However the fact that there is a 50/50 chance of picking the correct answer makes it less effective for comprehensive testing at higher learning levels.
1. Relate items to learning outcomes or objectives stated in the syllabus
2. Each question should be a clear, direct (active voice), unambiguous question on one item (unless cause and effect are being tested, in which case two statements are appropriate)
3. Be sure there is no discernible answer pattern, random smattering, slightly more false than true because students who are guessing tend to pick true more than false.
4. Avoid trivial content and detail; negatives and especially double negatives; convoluted sentences; two or more statements in the question (unless cause and effect are being tested)
| This | NOT this | Explanation |
|---|---|---|
| Lucian Freud’s works are noted for an often discomforting examination of the relationship between artist and model. □ True □ False |
Lucian Freud, painter of Hotel Bedroom owned by the Beaverbrook Art Gallery, was Sigmund Freud’s grandson. □ True □ False |
Shouldn’t be trivial. |
| Lucian Freud’s works are noted for their psychological penetration, and for their often discomforting examination of the relationship between artist and model. □ True □ False |
Ask one thing at a time. | |
| Although born in Berlin to a German mother and Austrian father, Lucian Freud became known as a British artist. □ True □ False |
Lucian Freud was widely considered the pre-eminent British artist of his time. □ True □ False |
Arguable. |
| Lucian Freud’s typical paintings were thickly impastoed portraits and figures. □ True □ False |
Lucian Freud was known chiefly for his thickly impastoed portrait and figure paintings. □ True □ False |
“Known chiefly” is too vague. |
| Freud painted people in his life rather than professional models. □ True □ False |
Freud is not known for painting people who were not professional models. □ True □ False |
Double negative (see below). |
5. Avoid all-or-nothing (e.g., "all," "none," "never," "always,") or vague determiners (e.g., "generally," “most people think,”) since the former make answer guessing easier and the latter can make the answer you selected as correct a matter of debate.
6. True-false questions are more valuable if you ask students to write the converse, correct statement for each false one.
| Statement | True | False | Write correct, converse statement for false statements. |
|---|---|---|---|
| All dogs are mammals | |||
| All marsupials are kangaroos | |||
| All parasites are animals | |||
| No snakes are reptiles |
Bosher, Susan. (2003). Barriers to Creating a More Culturally Diverse Nursing Profession: Linguistic Bias in Multiple-Choice Nursing Exams. Nursing Education Perspectives. 24(1)
Carusetta, E. (undated). Designing Student Assessment Tools: Tests, Problems, and Essays. Student Engagement Wiki at the University of New Brunswick.
Lampe, S. & Tsaouse, B. (2010). Linguistic Bias in Multiple-Choice Test Questions. Creative Nursing, 16(2).
Rodriguez, M. C. (2005).Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research. Educational Measurement: Issues and Practice.
University of Waterloo Centre for Teaching Excellence teaching tips, Exam Questions: Types, Characteristics, and Suggestions.