Pre- and Post-Training Tests: How to Write Questions That Really Measure the Target Skill

What does a pre- and post-training test measure, and what does it not?

CDC’s training evaluation guidance says the best way to detect any change in learning is to assess it before and after the training, and that the test can include a demonstration of the skill. A post-test alone shows whether learners reached the required level of proficiency, but not that they changed, since they may have known the material already. ^[1]

The test has limits that should be clear to anyone reading the results. It measures what appeared in the test situation, that is, learning. For transfer of learning to the job, the same guidance describes delayed follow-up evaluation as the best method, once the learner has had the chance to apply it. ^[1] So do not present improved scores to senior management as an impact on job performance.

Start from the learning objective, not the question bank

The most common mistake is writing the test from the slides: a definition here, a list of steps there. The result is a test that measures recall while the programme promises a skill.

CDC recommends writing learning objectives by combining Bloom’s Taxonomy to choose the verb and cognitive level, the ABCD method to set audience, behaviour, condition and degree, and the SMART criteria. It states that an objective should be assessable through a test, an assignment or a demonstration. ^[2] CDC’s quality training standards require learner assessments to tie directly to the learning objectives, to use varied question formats and to give learners feedback on their answers. ^[3]

The practical rule: if the objective is “diagnoses why a transaction stalled”, a question asking the trainee to “define a complaint” does not measure it, however precise it is.

How do you build a test blueprint?

A test blueprint (or table of specifications) maps each objective to its level, assessment method and weight in the test. It stops easy-to-write questions crowding out the objectives that matter.

Three rules when filling it in:

• Weight follows the objective’s importance at work, not the time spent explaining it.
• The method follows the objective’s level: recall suits a short question, analysis needs a case, and performance needs a task or demonstration.
• Measure each objective with more than one question where possible, so the judgement does not rest on a single item that may be ambiguous.

From definition questions to performance questions

Suppose the objective is for the trainee to analyse a customer complaint. Instead of asking for a definition of a complaint, give a short case setting out the sequence of the transaction and the staff responses, and ask the trainee to identify the stage at which it broke down and the evidence for that in the case text.

A multiple-choice question can measure judgement if it is built carefully:

• The stem presents a situation and one clear problem, so a competent trainee could think of the answer before reading the options.
• Wrong options come from real, common mistakes that staff make, not from obviously false choices.
• No unintended cues: do not make the correct answer the longest or most detailed option, and do not repeat a word from the stem in it.
• One question per idea, so two objectives are not merged into a single item.

When do you need a practical task instead of questions?

If the objective is a performance, such as writing a reply to a customer, entering a transaction in the system or handling a difficult conversation, questions alone are not enough, and the test can include a demonstration of the skill. ^[1] In that case:

• Before training, write a list of observable criteria for acceptable performance, such as “quotes the transaction number” and “proposes a specific resolution date”.
• Keep the task conditions identical in both tests: time, available information and type of case.
• Agree with the assessors how each criterion is to be interpreted before marking, and try it together on one or two answers.

A short, well-defined task is better than a long one that is hard to assess; the aim is to see the target outcome, not to test everything in the programme.

Review the questions with a subject-matter expert to confirm the keyed answer and the clarity of the wording, then pilot them on a small sample of people similar to the trainees, and drop or revise any question that competent people get wrong or that everyone gets right without training.

The same questions or two parallel forms?

If you use the same questions before and after training, part of the score may rise because trainees remembered the questions or looked up the answers, not because they learned. The effect is smaller if correct answers are not shown after the pre-test.

The stronger option is two parallel forms: each question in the first form has a counterpart in the second that measures the same objective at the same level, using a case that differs in detail but is similar in difficulty. You can also counterbalance the forms: half the group takes form A before and form B after, the other half the reverse, so improvement cannot be put down to one form being easier.

Use the pre-test as a teaching tool too. Assessment during training, whether a short knowledge check or observed practice, shows what needs reviewing before moving on to the next part. ^[4] Pre-test results tell the trainer which objectives trainees have already mastered, so they can be shortened, and which need more time.

Hypothetical example: reading results by objective

Hypothetical example: in a report-writing programme, 20 participants took two parallel forms based on a work case. In each case they had to select the supporting evidence and write a conclusion a decision-maker could act on, and two assessors marked the answers against fixed criteria for clarity and soundness of reasoning. The results were:

The overall score looks acceptable, but analysis by objective shows that conclusion writing barely improved. The team now reviews the conclusion-writing part of the programme: did trainees get enough practice and feedback on it? It also checks that the two cases really are similar in difficulty before judging the programme.

What should you check before comparing scores?

• Pair the two scores for each trainee while protecting privacy, using a code rather than a name where possible, and exclude anyone who did not sit both tests from the comparison.
• Record any change in test conditions: time allowed, whether references were permitted, paper or online.
• Review each question separately: a question most trainees still get wrong after training means either that the related part was not taught well or that the wording itself is ambiguous, so check both possibilities before judging the programme.
• Beware small groups: in a group of ten, one trainee can shift the average noticeably.
• Give each trainee feedback on their performance by objective; this is part of the learner assessment standard, not an optional extra. ^[3]
• If a pre-test is not possible, CDC accepts a retrospective self-assessment as an alternative, but it remains self-report. ^[1]

Conclusion

Build the test from the learning objectives and a clear blueprint, make the questions cases that mirror work tasks, and use two parallel forms of similar difficulty. Then read the results by objective, and present them as evidence of learning that still awaits evidence of application.