This will make you think! What are you really testing…?

If you haven’t heard about it yet, buy this book:


You know very little about how assessments work.

We all know very little about how assessments work.  I’ve been astounded by just how much there was to learn from one short book; how much I didn’t know.  Some of the conclusions will feel familiar to you: it’s unfair that so much of a hard working child’s future and evaluation should be pinned on a single exam, that grade wasn’t necessarily a true reflection of what you know they’re capable of, right?  Yes, and no, and this book reveals all the subtle shades of nuance between the yes and the no.  It explains why your intuition is bang on, but also the limits of that line of reasoning.

Then, there are things that just will have never even occurred to you before.  Here’s one that certainly never occurred to me!  Consider the following test item, from the book, which shows a distance time graph for a runner:

Test Item

The question might be “Explain what is happening at points A and B?” to which the correct answer would be “The runner is accelerating at A, and maintains constant speed at B.”

Two kinds of pupil could encounter this test item.

The first has studied distance time graphs in class; this is effectively a seen-item.  They studied the relationships between distance and time shown here on the graph, and how that relates to speed = distance / time, and why therefore A must represent increasing speed, while B represents constant speed.

As they come to the test item they recall this knowledge to answer the item correctly.  Therefore, the item tested this pupil’s ability to recall what they learnt in the past.

The second pupil has never seen a distance time graph before, but has learnt that speed, distance and time are related by speed = distance / time.  This pupil will need to spend time thinking and reasoning through how that relationship is being represented on the graph here.  They may or may not be successful, but either way, in this instance the item is a test of reasoning, not recall, since the pupil has never seen it before.  It is an unseen-item.

For me, the ground-breaking realisation was that the same test item might be assessing two different things!

Not only does this vary between pupils with different experiences, but lets look back at the first pupil again.  They might have understood what was discussed in the lessons, but on the other hand perhaps they didn’t, and eventually just learnt to memorise the fact that a curved lines represents acceleration while a straight line represents constant velocity, without any deeper understanding of why this is the case.  On the other hand perhaps the same pupil did understand everything perfectly well in the lesson, but through experience comes to rapidly identify curves with acceleration and straight lines with constant velocity.  In answering the question they might be able to very quickly respond by again relying on recall, the difference now being that if you asked that pupil to explain ‘why’ they would then be able to go on to justify their answer.  But, this doesn’t change the fact that in responding to the test item their thought process was one of recall, not reasoning.

With this in mind, the test item gives us no way of knowing what we are really testing!  Are we testing recall, or are we testing reasoning?

One analogy that had occurred to me in the past is responding to times tables questions.  Is a child answering by recalling that 7 x 8 is 56, or are they doing it by taking the number 7, doubling it (14), doubling it (28) and doubling it again (56)?  In this instance, one way of differentiating would be by time limit.  Since recall is fast (almost instant) and all processing takes time, placing a very strict time limit on a times tables test would help to assess whether the pupil was making use of recall or processing to answer the questions.

Conversely, if you want to test for processing or reasoning then you need test items that force this as best as possible.  For example asking a pupil to evaluate 8 x 23, which isn’t in the times tables, would necessitate at least some kind of processing (mental or paper-based).  Albeit, it still would not be straight-forward to assess precisely what process was being brought to bear!

Including test items on topics that make use of learnt knowledge, but are not taught explicitly in the classroom (unseen test items), is more likely to guarantee that reasoning takes place.  The AQA exam board seems to have started to do this more frequently in recent years, for example this item taken from their November 2014 paper:

A question like this has never come up before, and it will likely never come up again.

In summary, there is so much more to assessment than I think most of us have ever been taught.  Please buy Koretz’ book.  We need to know this.



About Kris Boulton

Teach First 2011 maths teacher, focussed on curriculum design.
This entry was posted in Uncategorized. Bookmark the permalink.

3 Responses to This will make you think! What are you really testing…?

  1. chemistrypoet says:

    Interesting. There is a strong tendency to value reasoning over recall. Reasoning is more important, right? (This has truthiness). But, of course, yes and no. Reasoning is good, but if a person knows very little, then that reasoning can never be used in a meaningful way….unless problems are tackled in teams. In reality, people tend to know some things, and reason from there. I had problems remembering thermodynamic based equations, so I learnt the basics and practised deriving the equations….(it would have been more efficient to memorise them, but for some reason I struggled with that). Now, I solve problems with the aid of colleagues. We all have bits of knowledge, facts, lateral connections and a solution will often emerge (there are also often several possible solutions in the real world). But, students need to learn stuff (to instant recall), and how to do things (including approaches), and need to develop the habit of using what they know and reasoning to where they didn’t know. What should we value? All of it.

    • Kris Boulton says:

      Well… I suspect it’s rather that they serve different functions. Do you really want to reason how to get to work each day, or do you just want to recall quickly how to do it? How about this: do you really want to reason through how to tie shoe laces, or do you just want to quickly whirl your fingers around and have it be done? How about driving: do you really want to reason through how to correctly perform a gear change at the appropriate time, or do you just want to automate that process so you can get on with the business of paying attention to the road and surrounding environment?

      People have seen memorisation as the enemy for so long, and in so doing failed to appreciate its frankly fundamental importance. Not only are there times when, frankly, recall is *far* better than reason, but reasoning is itself *dependent* on fluent recall!

      When you make a new meal for the first time, it’s a bit of a pain. You have to think carefully about all the ingredients, making sure you have the right quantities, the equipment you’ll need, trying to get all the timings right etc. Once you make that same meal 5-6 times you just recall what to do – the whole process is much more fluid and effortless, and frankly, pleasant.

      But! What if you wanted to try a new twist on that dish? It would certainly be more desirable to understand its construction well enough to be able to change parts of it and predict the results. There might even be times when you prefer that intellectual culinary challenge to the ‘same ol’ ‘

      I suspect our poor rhetoric stems from the reality that recall is much simpler to understand and teach to than reasoning. Consequently, perhaps *all* that was taught in the past was recall, or thereabouts. People lamented the lack of attention to reasoning and the like, decried the ineffectuality of so-called ‘rote memorisation,’ and voila, we got stuck with a Zeitgeist consequent of a pendulum that swung so far in one direction that it’s missing half our brain.

      In short, they’re both equally important, but serve different functions, and if we really want to crack this education game we probably need to take both into account and treat each with respect!

      (I wrote this I think after the first line or two, thinking it would be shorter than it is – my usual fault. In the end all I’ve done is agree with you, using more words!)

Leave a Reply to teachingbattleground Cancel reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s