With the increasing use of assessment processes for selection and promotion, designing your own exercises rather than using off-the-shelf products, or hiring consultants to design bespoke exercises, can seem like a logical and cost-effective step. Before starting to design your own exercises, it’s worth considering a few possible pitfalls. Here are our 8 best guidelines for creating reliable, valid and fair assessment exercises*.
(*Why is this important? Assessment exercises used for recruitment, selection and promotion have to be fair (not discriminating in a biased way), reliable (accurate and consistent in the way they measure) and valid (measuring relevant factors), because if they aren’t, there is no justifiable reason for using them)
What type of exercises can be used?
There are a variety of exercises commonly used for assessments and these can include role-plays, written exercises (including reports, in-trays, case studies and analysis exercises), fact-finding exercises, presentations/ briefings and group exercises. Psychometrics including ability tests and personality questionnaires may also be used, and situational judgement tests (although these are not recommended for designing in-house).
What standards need to be applied?
Exercises should meet standards set out in the BPS Standards for the Design and Delivery of Assessment Centres and the ISO 10667 standards, including (but not limited to): the use of trialling, piloting, post centre evaluation and use of behaviourally anchored rating scales (BARS).
Make sure your exercise is measuring against the competencies/criteria required
When writing exercises, it can be tempting to start by writing the scenario. However, this is likely to lead to exercises not covering all the behaviours required and measuring other behaviours which are not required!
Careful preparation is therefore key and exercise designers should begin by carefully analysing the role the exercise will be used to assess, mapping this to behaviours and then considering how those behaviours could be elicited in an exercise situation. Many organisations will have their own competency frameworks, but it is essential to ensure that the competencies being applied are relevant to the particular role and that the behavioural indicators outlined in the competency framework are capable to being assessed within the confines of an assessment exercise. It should be considered that several areas within any framework might be difficult to assess objectively; e.g. areas such as resilience and creativity can be subjective and open to debate, and incorrect use of these can inadvertently result in indirect discrimination.
Construct the exercise to manage a wide range of likely behaviours elicited
Exercise designers should consider all the possible interpretations of the materials and possible behaviours these might elicit. This can be guided by adding certain ‘steers’ to exercises, but be careful not to make these too obvious. The alternative is leaving the exercises too open-ended, which makes it difficult to predict what responses you may get. This becomes difficult when constructing your score sheets, as your marking instructions should account for all the possibilities to help your assessors mark consistently as much as possible.
Piloting the exercises is essential to ironing out any glitches, and working out if there could be variations in the way instructions are interpreted or the way potential responses might be formed.
Make sure your exercise is able to distinguish appropriately between strong and weaker candidates
This might sound obvious! But it’s surprisingly common problem for exercises to fail to distinguish between candidates and most end up scoring in the middle.
- This is usually because the BARS have not been calibrated correctly against the behavioural indicators, so as a consequence, candidates are unlikely to score at the highest or lowest ends of the scale.
- An alternative problem can occur when the BARS are too sensitive. In this case small differences between the candidates can result in fairly large differences in how the candidates score. In this case good candidates may be rejected and there is potential for grievances being filed by candidates over an unfair process.
Don’t try to make your exercise assess too much
Consider how many competencies and how many behavioural indicators can reasonably be assessed within a given exercise. With limited time frames allocated to candidates to complete tasks, there is only so much they can fit in. Some competencies/ indicators might be consistently missed, either because there wasn’t enough ‘steer’ in the exercise, or the instructions inadvertently gave the impression certain actions (which link to certain competencies) weren’t a priority. The results can give a misleading impression that the candidate wasn’t strong in a certain area, when this was actually more to do with the exercise design. Experience and piloting can help prevent this problem, which is important, because, again, candidates can feel unfairly treated in situations such as this, which has an impact on morale, candidate acceptance of the process/ results, and HR workload related to repercussion of these.
Pitch it at the right level
If you don’t have loads of experience with design, it’s surprisingly easy to make exercises unduly complex and misjudge whether your instructions are suitably clear and specific. An exercise which is too difficult risks having good candidates gain a poor result and is also likely to result in disengagement and hostility to the process in general. An exercise which is too easy may mean all candidates passing the exercise and weaker candidates being put in post, or alternatively have more successful candidates than available posts, which can result in disengagement. So you will need to make sure that exercises are clearly aligned with the competencies at the correct level. It is also important that exercise timings are realistic-these should allow the candidate sufficient time to complete the exercise, however too much time may result in candidates misunderstanding the task required or you’ll have so much evidence being generated that the exercise becomes unreliable to assess.
Create relevant exercises
It is important that candidates are able to see the relevance of the exercise to the job. AT VCA we create fictitious Fire & Rescue Services in our exercises which are popular with candidates as they feel more equipped to deal with tasks and to understand the context. A caution with doing this is to make sure the exercise does not require inside knowledge which only some candidates may possess. All knowledge which is needed to complete the exercise must be written within the exercise.
If candidates are struggling to see the relevance of the exercise, then they may disengage with the process and their performance in the exercise may not accurately reflect their skills. If this occurs, candidates are also more likely to challenge the process.
Make it easy for assessor to score consistently
This can be a common problem with exercises and can result when marking instructions are not clear and/or the scales for the exercise are not correctly matched to likely behaviours. Related problems can also arise when role-player guidelines are not written incredibly precisely, and this results in levels of inconsistency across role-plays.
Be mindful of adversely impacting on particular groups
It is important that the exercise respects diversity and isn’t likely to have direct or indirect adverse impact on groups with protected characteristics such as gender, sexual orientation, religion. It is also important to check that the exercise will allow those with a disability to complete it without any disadvantage and consideration should be given to neurodiversity. Careful checking should be given to language used, tasks and knowledge required for these tasks. The skills and attributes being assessed may also be discriminatory if these require a higher level of skill than is required for the role. For example, assessing personal interaction skills at a higher level than would be required in the role may result in indirect discrimination against those with ASD.
If you need any help with your exercise design, to talk through ideas or have the peace of mind of a check-through to help you with final tweaks, please do feel free to contact us for an informal chat.
