If you haven’t heard about it yet, buy this book:
You know very little about how assessments work.
We all know very little about how assessments work. I’ve been astounded by just how much there was to learn from one short book; how much I didn’t know. Some of the conclusions will feel familiar to you: it’s unfair that so much of a hard working child’s future and evaluation should be pinned on a single exam, that grade wasn’t necessarily a true reflection of what you know they’re capable of, right? Yes, and no, and this book reveals all the subtle shades of nuance between the yes and the no. It explains why your intuition is bang on, but also the limits of that line of reasoning.
Then, there are things that just will have never even occurred to you before. Here’s one that certainly never occurred to me! Consider the following test item, from the book, which shows a distance time graph for a runner:
The question might be “Explain what is happening at points A and B?” to which the correct answer would be “The runner is accelerating at A, and maintains constant speed at B.”
Two kinds of pupil could encounter this test item.
The first has studied distance time graphs in class; this is effectively a seen-item. They studied the relationships between distance and time shown here on the graph, and how that relates to speed = distance / time, and why therefore A must represent increasing speed, while B represents constant speed.
As they come to the test item they recall this knowledge to answer the item correctly. Therefore, the item tested this pupil’s ability to recall what they learnt in the past.
The second pupil has never seen a distance time graph before, but has learnt that speed, distance and time are related by speed = distance / time. This pupil will need to spend time thinking and reasoning through how that relationship is being represented on the graph here. They may or may not be successful, but either way, in this instance the item is a test of reasoning, not recall, since the pupil has never seen it before. It is an unseen-item.
For me, the ground-breaking realisation was that the same test item might be assessing two different things!
Not only does this vary between pupils with different experiences, but lets look back at the first pupil again. They might have understood what was discussed in the lessons, but on the other hand perhaps they didn’t, and eventually just learnt to memorise the fact that a curved lines represents acceleration while a straight line represents constant velocity, without any deeper understanding of why this is the case. On the other hand perhaps the same pupil did understand everything perfectly well in the lesson, but through experience comes to rapidly identify curves with acceleration and straight lines with constant velocity. In answering the question they might be able to very quickly respond by again relying on recall, the difference now being that if you asked that pupil to explain ‘why’ they would then be able to go on to justify their answer. But, this doesn’t change the fact that in responding to the test item their thought process was one of recall, not reasoning.
With this in mind, the test item gives us no way of knowing what we are really testing! Are we testing recall, or are we testing reasoning?
One analogy that had occurred to me in the past is responding to times tables questions. Is a child answering by recalling that 7 x 8 is 56, or are they doing it by taking the number 7, doubling it (14), doubling it (28) and doubling it again (56)? In this instance, one way of differentiating would be by time limit. Since recall is fast (almost instant) and all processing takes time, placing a very strict time limit on a times tables test would help to assess whether the pupil was making use of recall or processing to answer the questions.
Conversely, if you want to test for processing or reasoning then you need test items that force this as best as possible. For example asking a pupil to evaluate 8 x 23, which isn’t in the times tables, would necessitate at least some kind of processing (mental or paper-based). Albeit, it still would not be straight-forward to assess precisely what process was being brought to bear!
Including test items on topics that make use of learnt knowledge, but are not taught explicitly in the classroom (unseen test items), is more likely to guarantee that reasoning takes place. The AQA exam board seems to have started to do this more frequently in recent years, for example this item taken from their November 2014 paper:
A question like this has never come up before, and it will likely never come up again.
In summary, there is so much more to assessment than I think most of us have ever been taught. Please buy Koretz’ book. We need to know this.