1. They can be exceptionally rigorous
As Daisy Christodoulou sets out first here, here and then here, counter to our lay-person intuition multiple-choice questions can actually be very difficult to answer well, thus requiring truly concrete and in-depth knowledge. In the following example:
15. How did the Soviet totalitarian system under Stalin differ from that of Hitler and Mussolini?
A. It built up armed forces.
B. It took away human rights.
C. It made trade unions illegal.
D. It abolished private land ownership.
It wouldn’t be enough to know a bit about about Nazi Germany, you would have to also know something about the regimes of Stalin and Mussolini, and be aware of subtle differences between them. I think I know the answer to the question… but I’m not certain!
If you’re still unconvinced, just consider Tony Gardiner’s UK Mathematics Challenge, which was sat by pupils across the country on Thursday last week:
Or try searching for just a few example questions from the GMAT:
2. They can be marked quickly, or even instantly by computer, and then provide remarkably granular insight into pupil knowledge and progression
‘But can’t they just guess?’ In terms of a summative assessment of pupil progress, guessing won’t work. If the questions have five possible answers, then over time they will only achieve 20% through guesswork. A pupil could guess correctly, presenting the impression that they know a particular item which in fact they do not. Over time, however, if this question were allowed to recur as part of a long-term assessment strategy, then the correct guess would likely be spotted in the analysis.
As Joe Kirby and Bodil Isaksen also recently pointed out to me, if you allow for more than one correct answer, and don’t say how many correct answers there are, then the probability of correct guessing reduces dramatically. For example, if the question were “Which of these were considered wonders of the ancient world?” And five of the seven were given as possible answers, along with a further five which are incorrect, then the odds of a correct guess plummets to 0.3% if the pupil knew that there were five correct answers. I’m not even sure right now how to calculate the odds if they don’t have that information. Joe also noted that, as with the example question about totalitarianism, if your ‘wrong options’ are things that sound plausible, like ‘the Colosseum’ for example, then the questions become a further tool for learning as pupils make mistakes and realise that ‘it’s an error to think that the Colosseum was a wonder of the world.’
3 Once created, they can be set automatically
4. They enable pupils to easily leverage the testing effect and distributed practice, thus building memory storage strength throughout the course of their time in education
If multiple-choice assessment were part of an extremely frequent routine then it benefits from the double-whammy of the testing effect (acts of memory recall build the storage strength of that memory) and, if questions are allowed to disappear for a while before they reappear, then also distributed practice (the process of allowing retrieval strength to dip, so that when the act of recall is attempted the subsequent gain to storage strength is increased.)
They can also allow for pupils to take greater ownership of their learning. Some teachers are now setting up courses on Memrise which empowers pupils to leverage the testing effect and distributed practice from home.
5. When used as pre-tests, they can prime pupils for what is to come, leading to even greater learning
It’s been shown that testing before learning the material on the test can accelerate future learning of that material.
Multiple-choice tests have an advantage here because they offer an opportunity for pupils to see key words, language, phrases and ideas before they are presented to them in lessons, while still giving pupils the chance to feel like they can kind of do *something*, even if it’s just read then guess, whereas by contrast an open-ended question would quite obviously be impossible to answer.
Suggestions have been made that this pre-lesson priming leads to greater attention in pupils when those same key words and phrases from the pre-test appear in the lesson. Ironically the research has suggested an advantage to English and the humanities as a result of this, where most English and humanities teachers I’ve spoken to have intuitively assumed that pre-testing might be fine for maths, but has no place in their subject.
1. They take a long time to create
2. They require a great deal of expertise and skill to create well
Creating a good set of multiple-choice questions is not an easy thing to do. Ideally, the incorrect options should be selected with care, and should reveal common misconceptions or tease out fine detail nuances between minimally different concepts (e.g. totalitarianism under different leaders.) Just take a quick look at Dylan Wiliam’s work here on Hinge Questions to get a sense of how much expertise needs to go into designing these well:
You can see this kind of thinking at work on the UK Maths Challenge papers. If people have been inexpertly creating multiple-choice questions for many years, it would be no surprise that they now have a bad rep.
3. We value what we measure, and they restrict what we ask pupils to do
Dan Meyer has done some fascinating work analysing the verbs that Khan Academy uses in posing its maths questions. The most frequent verb is analyse, followed by calculate, both occurring roughly 20 times more often than the verbs estimate and argue.
He then looks at what pupils are asked to produce. Most frequent are ‘a number‘ and ‘multiple-choice response,’ roughly 30 times more often than ‘a transformation‘ or ‘a two-column proof.’
The reason this kind of analysis is important is because we value what we measure. In maths, for example, if people were forever expected to ‘select the right answer’ then perhaps we risk people thinking that mathematics is all about ‘getting the right answer,’ even when questions are as demanding and thought provoking as those of the UKMT.
So is this a tool of summative assessment, formative assessment, or a tool of learning?
All of the above.
Multiple-choice tests can be a summative tool (see TIMSS and NAEP), a formative tool (via granular question-level analysis) and a tool of learning (via the testing effect and distributed practice, and as this article notes: “the well-replicated finding that tests do not simply act as measurement tools, but also prompt leaning in their own right, [are under-appreciated.]” http://learninglab.uchicago.edu/Pre-Testing.html)
I get the sense that, in education over the years, people have tended to adopt an idea and then stick to it. Con number 3 is largely eliminated if multiple-choice questions are seen not as another panacea, but as another tool in an inevitably complex toolbox.
But should you use them? Well I don’t. I don’t only because I do not have enough time to write them, write them well, and write enough of them to leverage the pros. If you can find a way, though, I would strongly suggest that you add them to your repertoire. David Thomas managed to do so, as he notes here, through use of Quick Key. For my part, I’m wondering if I can use Quick Key to have pupils sit half a GCSE paper, and fill in the numbers according to how many marks they scored on each question, so that I can quickly create a QLA of every practice paper they sit…
Ultimately, I’m fascinated by the role multiple-choice questions could play as part of a well-thought-out, long-term education strategy, embraced by a teaching profession as a whole.