GOOD QUESTION DESIGN FOR OBJECTIVE TESTING

em Avaliação da Aprendizagem
Colleen McKenna and Joanna Bull
CAA Centre, University of Luton
 
 
Recent rises in student numbers have led to a re-evaluation of the type and frequency of assessments. Marking large numbers of student essays, projects and other subjective assessments is time-consuming, labour intensive and prone to errors of consistency (Newstead and Dennis 1994). As a result, it is increasingly common for staff to incorporate objective testing within their assessment profile. Objective tests are those which require a user to choose or provide a response to a question whose correct answer is pre-determined. A typical question might require a student to
  • select a solution from a set of choices (MCQ, true-false, matching)
  • identify an object or position (graphical hotspot) or
  • supply brief numeric or text responses (text input).
Including objective tests within an assessment profile allows for more regular and efficient examining of students, because objective questions can be marked rapidly using computers, an OMR or staff members with no special knowledge in the area being assessed. Additionally, they eliminate the need for double (and triple) marking, thus saving time after the examination process.
Objective tests are especially well-suited to certain types of tasks. Because questions can be designed to be answered quickly, they allow lecturers to test students on a wide range of material. The use of CAA in the delivery of objective tests enables the provision of automatic feedback (in terms of scores, hints, praise, and guidance) to the student. Additionally, statistical analysis on the performance of individual students, cohorts and questions is possible. Furthermore, questions can be pre-tested in order to evaluate their effectiveness and level of difficulty.
The capacity of objective tests to assess a wide range of learning is often underestimated. Objective tests are very good at examining recall of facts, knowledge and application of terms, and questions that require short text or numerical responses. But a common worry is that objective tests cannot assess learning beyond basic comprehension. However, questions which are constructed imaginatively can challenge students and test higher learning levels. (Simas and McBeath 1992). For example, students can be presented with case studies or a collection of data (such as a set of medical symptoms) and be asked to provide an analysis by answering a series of questions. Well-written questions can test problem-solving in a range of subjects. If using a computer, students can be given electronic tools to manipulate or construct objects on a screen; they can also perform tasks based on simulation of fieldwork or laboratory experimentation.
There are, however, limits to what objective tests can assess. They cannot, for example, test a student's abilities to communicate, to construct arguments or to offer original responses. Tests must be carefully constructed in order to avoid the decontextualisation of knowledge (Paxton 1998) and it is wise to use objective testing as only one of a variety of assessment methods within a module. Nevertheless, in times of growing student numbers and decreasing resources, objective testing can offer a viable addition to the range of assessment types available to a lecturer.

Writing basic objective test questions

Writing good objective test questions takes practice and is often time-consuming at the outset. The following section considers four basic question formats (multiple choice, matching, true-false and text match) and offers suggestions for constructing each type. Much space is devoted to the multiple choice question (MCQ) and a number of the principles discussed in this section can be applied to the other question types. (For a description of a wider range of objective test question formats, see Appendix A).

Multiple choice questions

A traditional multiple choice question (or item) is one in which a student chooses one answer from a number of choices supplied. A multiple choice question consists of
  • a stem - the text of the question
  • options - the choices provided after the stem
  • the key - the correct answer in the list of options
  • distracters - the incorrect answers in the list of options

Example of a multiple choice question

As societies increase in complexity from folk to industrial, social control is more likely to be invested in the   (Stem)
a. family   (Distractor)
b. school   (Distractor)
c. state   (Key)
d. peer group   (Distractor)
e. religious structures   (Distractor)



(Question from Graduate Record Examination (GRE), Sociology Test 1997-1999. Website address: http://www.ets.org)

Suggestions for writing good stems

  • Present a single, definite statement to be completed or answered by one of the several given choices
A. Weak question:

Polysaccharides
a. are made up of thousands of smaller units called monosaccharides
b. are NOT found in the aloe vera leaf
c. are created during photosynthesis
d. can be described by the chemical formula: CHHOH

B. Improved question

Polysaccharides of the plant cell wall are synthesized mainly in
a. the endoplasmic reticulum
b. the cytosol
c. the plasma membrane
d. the Golgi complex
e. amyloplasts

In the top example, there is no sense from the stem what the question is asking. The second example more clearly identifies the question and offers the student a set of homogeneous choices.
  • Avoid unnecessary and irrelevant material
A. Weak question

Paul Muldoon, an Irish postmodern poet who uses experimental and playful language, uses which poetic genre in "Why Brownlee Left"?
a. sonnet
b. elegy
c. narrative poem
d. dramatic monologue
e. haiku

B. Improved question

Paul Muldoon uses which poetic genre in "Why Brownlee Left"?
a. sonnet
b. elegy
c. narrative poem
d. dramatic monologue
e. haiku

The top example contains material irrelevant to the question.
  • Use clear, straightforward language in the stem of the item. Questions that are constructed using complex wording may become a test of reading comprehension rather than an assessment of whether the student knows the subject matter.
  • Use negatives sparingly. If negatives must be used, capitalize, underscore embolden or otherwise highlight.
  • Put as much of the question in the stem as possible, rather than duplicating material in each of the options. (Gronlund 1988)
Weak question

Theorists of pluralism have asserted which of the following?
a. The maintenance of democracy requires a large middle class.
b. The maintenance of democracy requires autonomous centres of contervailing power.
c. The maintenance of democracy requires the existence of a multiplicity of religious groups.
d. The maintenance of democracy requires a predominantly urban population.
e. The maintenance of democracy requires the separation of governmental powers.

Improved question

Theorists of pluralism have asserted that the maintenance of democracy requires
a. a large middle class
b. autonomous centres of contervailing power
c. the existence of a multiplicity of religious groups
d. a predominantly urban population
e. the separation of governmental powers



Question from GRE Sociology test book - 1997-1999 Website address: http://www.ets.org.

Suggestions for writing good distracters

  • For single response MCQs, ensure that there is only one correct response.
  • Use only plausible and attractive alternatives as distractors.
  • Avoid giving clues to the correct answer.
A. Weak question

A fertile area in the desert in which the water table reaches the ground surface is called an
a. mirage
b. oasis
c. water hole
d. polder

B. Improved question

A fertile area in the desert in which the water table reaches the ground surface is called a/an
a. mirage
b. oasis
c. water hole
d. polder

Example A uses the article "an" which identifies choice b as the correct response. Ending the stem with "a/an" improves the question.
  • If possible, avoid the choices "All of the above" and "None of the above". If you do include them, make sure that they appear as correct answers some of the time.
It is tempting to resort to these alternatives but their use can be flawed. To begin with, they often appear as an alternative that is not the correct response. If you do use them, be sure that they constitute the correct answer part of the time. An "all of the above" alternative could be exploited by a test-wise students who will recognise it as the correct choice by identifying only two correct alternatives.
Similarly, a student who can identify one wrong alternative can then also rule this response out. Clearly, the student's chance of guessing the correct answer improves as they employ these techniques. Although a similar process of elimination is not possible with "none of the above", it is the case that when this option is used as the correct answer, the question is only testing the students' ability to rule out wrong answers, and this does not guarantee that they know the correct one. (Gronlund 1988)
  • Distracters based on common student errors or misconceptions are very effective.
One technique for compiling distracters is to ask students to respond to open-ended short answer questions, perhaps as formative assessments. Identify which incorrect responses appear most frequently and use them as distracters for a multiple choice version of the question.
  • Correct statements that do not answer the question are often strong distracters.
  • Avoid using ALWAYS and NEVER in the stem as testwise students are likely to rule such universal statements out of consideration.
  • Do not create distracters that are so close to the correct answer that they may confuse students who really know the answer to the question. "Distracters should differ from the key in a substantial way, not just in some minor nuance of phrasing or emphasis." (Isaacs 1994)
  • Provide a sufficient number of distracters.
You will probably choose to use three, four or five alternatives in a multiple choice question. Until recently, it was thought that three or four distracters were necessary for the item to be suitably difficult. However a 1987 study by Owen and Freeman suggests that three choices are sufficient (Brown 1997). Clearly the higher the number of distracters, the less likely it is for the correct answer to be chosen through guessing (providing all alternatives are of equal difficulty.)

Matching Questions

Matching items require students to match a series of stems or premises to a response or principle. They consist of a set of directions, a column of statements and a column of responses.
Example - matching test item

 
Directions: Match the quotation in column I with the literary school with which it is associated listed in column II. Items in column two may be used more than once.
Column I
Column II
  1. You can lead a horse to water but you can't make itHoldIts nose to the grindstone and hunt with the hounds.Every dog has a stitch in time. Two heads? You've beenSoldOne good turn. One good turn deserves a bird in the hand.
  2. I cannot see what flowers are at my feet,Nor what soft incense hangs upon the boughs,But, in embalmed darkness, guess each sweetWherewith the seasonable month endowsThe grass, the thicket, and the fruit-tree wile;White hawthorn, and the pastoral eglantine;Fast fading violets cover'd up in leaves;And mid-May's eldest child,The coming musk-rose, full of dewy wine,The murmurous haunt of flies on summer eves.
  3. frseeeeeeeefronnnng train somewhere whistling the strength those engines have in them like big giants and the water rolling all over and out of them all sides like the end of Loves old sweeeetsonnnng the poor men that have to be out all the night from their wives and families in those roasting engines stifling it was today Im glad I burned the half of those old Freemans and Photo Bits leaving things like that lying about hes getting very careless
  4. Twit twit twitJug jug jug jug jug jugSo rudely forc'dTereu
  5. A perfect Judge will read each Work of WitWith the same Spirit that its Author writ,Survey the Whole, nor seek slight Faults to find,Where Nature moves, and Rapture warms the Mind;
A. Romanticism
B. Modernism
C. Neo-classicism
D. Post-modernism
E. Humanism
F. Classical realism

Matching questions are really a variation of the multiple choice format. If you find that you are writing MCQs which share the same answer choices, you may consider grouping the questions into a matching item. Tips for writing good matching questions include:
  • Provide clear directions
  • Keep the information in each column as homogeneous as possible
  • Allow the responses to be used more than once
  • Arrange the list of responses systematically if possible (chronological, alphabetical, numerical)
  • Include more responses than stems to help prevent students using a process of elimination to answer question.
 

True/False questions

A true-false questions is a specialised form of the multiple-choice format in which there are only two possible alternatives. These questions can be used when the test-designer wishes to measure a student's ability to identify whether statements of fact are accurate or not.
Example True/False questions

T F   A poem with the following rhyme scheme could be correctly referred to as an English sonnet: abab cdcd efef gg.
T F   All eukaryotic genes are organized into operons.

True-false questions offer lecturers a very efficient method of testing a wide range of material in a short period of time. They can also be combined within a multiple-choice to create the more complex assertion-reason item. However, true-false questions do have a number of limitations:
  • Guessing - a student has a 1 in 2 chance of guessing the correct answer of a question.
  • It can be difficult to write a statement which is unambiguously true or false - particularly for complex material.
  • The format does not discriminate among students of different abilities as well as other question types.

Suggestions for writing true-false questions:

  • Include only one main idea in each item.
  • As in multiple choice questions generally, use negatives sparingly.
  • Try using in combination with other material, such as graphs, maps, written material. This combination allows for the testing of more advanced learning outcomes. (Gronlund 1988)
  • Use statements which are unequivocally true or false.
  • Avoid lifting statements directly from assigned reading, lecture notes or other course materials so that recall alone will not permit a correct answer.
  • Generally avoid the use of words which would signal the correct response to the test-wise student. Absolutes such as "none", "never", "always", "all", "impossible" tend to be false, while qualifiers such as "usually", "generally", "sometimes" "often" are likely to be true.

Text match response

The text match question requires a student to supply an answer to a question or complete a blank within a brief piece of text, using words, symbols or numbers.
Examples of text match question

a. Tony Blair is the leader of the ___________ party.
b. 235 x 23 + (9x5) = ________.

A possible advantage of this question type is that the student must supply the correct answer rather than identify or choose it. The likelihood that the candidate will guess the correct answer is lower than that of a multiple choice question. However, the short answer response questions can be difficult to phrase in such a way that only a single correct answer is possible. Additionally, if you are marking the assessments with computers, spelling errors may disadvantage students who know the right answer.
For a fuller discussion of question design see the CAA Centre web tutorial at http://caacentre.ac.uk/objections/index.shtml .
For more information about computer-assisted assessment see the CAA Centre website at http://caacentre.ac.uk .

References

Brown, G., with Bull, J., and Pendlebury, M. (1997) Assessing Student Learning in Higher Education. London: Routledge.
Gronlund, N.E. (1988) How to construct Achievement Tests. Englewood Cliffs: Prentice Hall
Isaacs, G. (1994) "About multiple choice questions" in Multiple Choice Testing: Green Guide, No. 16, HERDSA, Cambelltown, NSW, 4-22.
Newstead, S. and Dennis, I. (1994) 'Examiners examined: The reliability of exam marking in psychology'. The Psychologist 7:216-19
Paxton, M. (1998) 'A linguistic perspective on multiple choice questioning' in The Higher Education Research and Development Society of Australasia newsletter, April 20.1
Simas, R. and McBeath, R.J. (1992) 'Constructing multiple choice test items', in McBeath, R.J. (Ed.), Instructing and Evaluating in Higher Education, Educational Technology Publications, Engelwood Cliffs, New Jersey.

Appendix A - Objective question types

The following are examples of some of the question types appropriate for CAA:
Multiple choice questions (MCQs) are the traditional 'choose one from a list' of possible answers.
True/False questions require a student to assess whether a statement is true or not.
Assertion-Reason questions combine elements of MCQ and true-false.
Multiple response questions (MRQs) are similar to MCQs, but involve the selection of more than one answer from a list.
Graphical hotspot questions involve selecting an area(s) of the screen, by moving a marker to the required position. Advanced types of hotspot questions include labelling and building questions.
Text/Numerical questions involve the input of text or numbers at the keyboard.
Matching questions involve linking items in one list to items in a second list.
Sore finger questions have been used in language teaching and computer programming, where one word, code or phrase is out of keeping with the rest of a passage. It could be presented as a 'hot spot' or text input type of question.
Ranking questions require the student to relate items in a column to one another and can be used to test the knowledge of sequences, order of events, level of gradation.
Sequencing questions require the student to position text or graphic objects in a given sequence. These are particularly good for testing methodology.
Field simulation questions offer simulations of real problems or exercises.
Other question types require students to identify and/or manipulate images. Students may be asked to plot a graph, complete a matrix, draw a line or build up an image using parts provided.