What the research says

Does Quizzing Help With Questions You Haven’t Seen?

Practice tests help students remember the answers they practised. But do they help with new questions? A review of 122 experiments found they do, under four conditions.

By the Giga Quizzes team4 min read
A rainbow-coloured bridge over a small river on a sunny day
Photo: Jan Małota, Unsplash, Unsplash License, cropped.
In this post
  1. 1The review
  2. 2The headline
  3. 3When transfer happens
  4. 4An honest caveat
  5. 5Designing quizzes that transfer

The most common objection to quizzing is that it trains students to answer the quiz. Practise a question, and of course you get that same question right later. But exams rarely repeat the practice questions word for word, and real life never does. The question that matters is transfer: does practising retrieval help with questions you have not seen, in a form you have not met? In 2018 Steven Pan and Timothy Rickard pulled together more than forty years of research on exactly that.

The review

Pan and Rickard collected 192 transfer results from 122 experiments in 67 published and unpublished articles, with 10,382 participants in all. Every study compared students who had taken practice tests with students who had spent the same time seeing the material again, and every final test asked something different from the practice.

Transfer comes in many kinds, and the review sorted them. In everyday classroom terms, a few examples of what counts as a new question:

  • A new format: a fact practised as a typed answer, then tested as multiple choice.
  • An application: practise “What does a catalyst do?”, then meet a question about why a particular reaction speeds up.
  • An inference: practise what a passage says, then answer something it implies but never states.
  • A new case: practise diagnosing one patient, then diagnose another with the same condition.

The review sorted every study by kinds like these, so that it could ask not just whether practice transfers, but which kinds of transfer it helps and which it does not.

The headline

Overall, practice testing did transfer: d = 0.40 compared with seeing the material again. That is a real, moderate benefit, and it came from questions the students had never practised.

But the average hides wide differences. Transfer was strongest in some kinds of new question and weak or missing in others.

Transfer was strongest to…Transfer was weakest to…
the same content in a different test formatthe same word pairs, rearranged
application and inference questionsmaterial that was studied but never tested
medical diagnosis problemsproblems with worked examples
Pan & Rickard (2018), from the review’s summary of results

The second column is worth reading carefully. Testing some facts from a lesson did not do much for the other facts from the same lesson. Quizzing helps what it touches.

When transfer happens

The authors turned their findings into four principles. They warn that most of the research comes from laboratories, but the principles are clear.

  1. Transfer is likeliest when the answer is the same. If the practice question asks for a term given its definition, and the exam asks for the same term given a real-world example, the shared answer carries over.
  2. Transfer grows when practice covers the idea broadly. Telling similar concepts apart, explaining an answer, or meeting the same concept through several questions at different levels all help.
  3. Transfer grows with explanations and restudy after the quiz. Feedback that explains, or a chance to go back over the material, makes it more likely.
  4. Transfer grows with accuracy on the practice test. The better students do on the practice, the more it carries to new questions.

The fourth principle has a practical edge. Very hard practice quizzes, where most students fail, may teach less than ones pitched so that most students can get most questions right.

An honest caveat

The review also checked for publication bias, the tendency for studies with positive results to be published more often. Once the authors corrected for it, the estimated transfer was much smaller when none of the helpful conditions above were present, often close to nothing. In other words, quizzing does not transfer automatically. It transfers when the quiz is designed for it.

That is less discouraging than it sounds. The helpful conditions are not rare or expensive: showing an explanation, asking about an idea in more than one way, and pitching questions at a level students can reach. Most good quizzes already do some of these. The caveat is a warning against the laziest version of quizzing, the same narrow question repeated until it is memorised, rather than against quizzing itself.

Designing quizzes that transfer

Each principle points to a choice a quiz writer can make.

Questions that reach beyond themselves

  • Ask about the same idea in more than one way: a definition, an example, a scenario.
  • Include questions that make students choose between two similar concepts.
  • Show an explanation after each answer, not just the letter of the right option.
  • Pitch most questions so the class can get them right, with a few harder ones for stretch.
  • Quiz the ideas you most want to last, because untested material gains little.

Two related studies go deeper into parts of this: explanation feedback and fact versus thinking questions. Together they make the same point as this review. A quiz is a tool for learning only if it is built to make students think about the idea, not just the question.

Sources

  1. Pan, S. C. & Rickard, T. C. (2018). Transfer of test-enhanced learning: Meta-analytic review and synthesis. Psychological Bulletin, 144(7), 710–756.

More from the blog

All posts →