Some organisations handle the design of assessment exercises internally. This can be difficult to get right without a good level of experience and expertise in-house.

If you are handling the design of assessment exercises internally here are some tips to getting it right!

Written exercises

Mistakes to Avoid Unwanted Impacts
The information provided in background documents doesn’t match with what the assessors are instructed to look forYou need to provide all the information you would expect to be referenced and used during the exercise. It is not fair to make assumptions about what candidates do and don’t already know as this may vary. In poorly designed exercises, assessor guidelines can include points which the candidate cannot be reasonably expected to have known.
There is insufficient detail provided within the exercise for the candidates to be able to answer in fullAssessors measure performance against indicators which do not match the information provided
Tasks are not sufficiently in-depth, varied or carefully constructedCandidates are not steered to provide a wide enough range of behaviours/ performance so assessors are unable to reliably rate some of the constructs (elements of the competencies) which are nevertheless included in the score sheets. this creates an unreliable assessment and potential for invalid results
Insufficient guidance of what good performance should look likeThis leads to reduced reliability of assessment as assessors may base evaluation of ‘good performance’ on their own opinions, which will vary
Insufficient adaptation of generic Behaviourally Anchored Rating Scales (BARS), Leadership Quality or other similar frameworkIt can be useful to use measurement frameworks as guidelines to be adapted to match the specific exercise. However, the adaptation should not deviate them from their original meaning, or risk unintentional overlap with other constructs.
Constructs are indistinctAssessors end up rating the same behaviour multiple times as it isn’t clear how different constructs can be differentiated
Constructs are included without any steer within the exercise material to encourage the candidate to include thisCandidates will fail to provide evidence of a particular construct as it was not apparent from the exercise that this area was relevant. Assessors end up scoring a ‘1’ for no evidence when it was actually not the candidates fault
Indicators are unrealistic/ a top score would require more than candidates are likely to be able to deliver in the time available.In order to meet high requirements in one competency, performance on others may suffer, creating skewed results
Unrealistic subject matter or pitched at the wrong levelThe assessment won’t be accurate if the exercise content isn’t appropriate or if it is too difficult or too easy for the level being assessed

 

Role-plays

Mistakes to AvoidUnwanted Impacts
Too much for the candidate to doThe assessors won’t be able to make a decent evaluation if the candidates have to use a scattergun approach to cover everything expected
Too much script for the role-playersThere isn’t the time for the candidate to contribute so the assessors are left with little to evaluate
Too much to read and assimilate/ too much complexityCandidates can end up reading from their notes to cover all essential information rather than demonstrating their natural behaviour in practice
Constructs overlapThere is too much emphasis on one element of the role-play performance e.g. problem solving or interpersonal aspects, at the expense of areas which are less obvious, but still need to be assessed. Less experienced assessors will struggle to differentiate between the constructs so the assessment will be of global performance rather than broken down into distinct areas. This will make results misleading and unreliable
Too many things covered within one indicatorThis is where adapting the BARS is important because many elements of the constructs do overlap and could confuse the assessors, rendering their assessment less reliable
Too much information in the candidate instructions and assessor guidelinesThis makes is more difficult for the assessors to separate out the behaviours they are observing and draw conclusion about what they ae indicative of
The content of the indicator does not reflect the essence of the constructThis makes the assessors job more difficult and their ratings less likely to properly reflect the performance they witnessed
Lack of detail to the assessor and role-player guidelinesSome candidates will try to model some of the positive behaviours by ‘saying the right things’ but in a superficial way. Role-player guidelines need to be robust enough to challenge this in a consistent way, and assessors need the complexity in their scoring criteria for them to be able to see where holes in the performance exist

 

If you would like a free review of your assessment exercises then contact us by email so we can help you feel confident you are designing exercises to meet validity/ reliability standards.