Simulations Rock

May 1, 2009 – As published in the Rocket-Hire Explorer

By Ken Lahti, Ph.D. - Director of Product Strategy & Innovation, PreVisor

Most people don’t like taking tests. And who can blame them? Work and applying for work can be tough enough without being asked to sit still for 60 minutes or more and answer a bunch of multiple-choice questions. This discomfort is compounded by the lack of transparency in many tests, where test-takers aren’t sure what they are being evaluated on and may not understand or even believe that the questions are job-related. Even if it’s for a good cause – like gaining insight into one’s competencies for career planning or as part of applying for a job – testing usually just isn’t that fun. While there are tremendous benefits to employers to get the kind of objective and predictive Talent Measurement information that only tests can provide, the candidate experience is pretty lacking.

On the other hand, simulations rock. Simulations are a form of testing where the candidate or employee is asked to actually do something. That is, rather than simply writing or speaking responses (e.g., essays or interviews) or choosing an answer on a multiple-choice test, simulations ask the test taker to complete a task. A simple example is a typing test in which the test-taker views some written material and is asked to type it as quickly and as accurately as they can. A more complex example would be a “day in the life” leadership simulation that puts the test-taker in the role of a manager and has them deal with incoming emails, voicemail, and visits from their direct reports. Interestingly, even though they are still being measured and evaluated with simulations, people seem to enjoy taking these kinds of tests. They are much more engaging than traditional tests, and in some cases they are almost game-like.

Classic rock
Simulations have around for decades under various guises. They have been called work sample tests, performance assessments, and skills tests. Again, think of the typing test (e.g., for admin/clerical jobs). Candidates for factory work have been asked for decades to put pegs into holes in a pegboard to understand their hand-eye coordination and likelihood for success in assembly roles. For jobs where physical abilities are important for performance (e.g., firefighter), employment assessments historically and even today have involved physical tasks that simulate actual work activities (e.g., carrying a rescue dummy).

Why is there such enduring value for simulations? Well, if you think about it, the very best and most assured way of evaluating people’s ability to successfully perform in a given role… is to give them the job for a while and see how they do. For more complicated jobs, you might need to wait months to get a good reading on whether the person stacks up. Clearly, this would be expensive… in more ways than one.

The shortened version of this would be to take certain job elements and replicate them in a controlled environment – the simulation. Candidates go through the simulation activities and, in essence, prove that they can do the job (or at least, key parts of the job). As it turns out, this is a good model and a decent approximation of the job-tryout that provides objective and highly predictive information about future on-the-job success.

Electronica
In modern testing, simulations have moved online. While some simulations are still done as in-person exercises (e.g., coaching a subordinate via live role play), most employers are looking to take advantage of automation wherever possible in their testing programs.

Historically, the downside of simulations has been the cost of administration. Role players and/or simulation scorers are often expensive resources; even if these roles are played by existing staff, those folks need to stop work to assess candidates. In addition, there is considerable work required up-front to implement the program and train users, and ongoing investment is required to recalibrate the scoring procedures periodically to ensure consistency and comparability between applicants.

Online simulations solve these challenges. For example, PreVisor’s Front Line Manager assessment includes an online Coaching simulation. In the Coaching simulation, test-takers play the role of a current manager dealing with some tricky issues with their subordinates. Rather than use live role players, the system presents video clips of a subordinate, and the test-takers interact by indicating the most appropriate and least appropriate responses given what is being discussed. The system automatically scores these responses and provides an accurate measure of candidates’ potential for success in coaching subordinates. Similarly, PreVisor’s online Inbox exercise allows candidates to demonstrate their prioritization and decision-making skills in a real-life managerial scenario dealing with emails, voicemails, and the occasional office visitor. The system scores the simulation in real-time without the need for costly raters, observers, etc.

Post-modern
The leading edge of simulation design for use in employment testing combines the high-fidelity job-relevant test taker experience with modern testing technologies to maximize efficiency and prediction and get the best of both worlds.

For example, PreVisor’s Contact Center Virtual Scenario combines simulation elements – such as listening to and typing customer information in a real-world scenario and navigating multiple computer screens – with traditional test components such as situational judgment test questions embedded directly into the experience. A candidate who is immersed in the contact center simulation solving a customer’s billing issues or helping them book travel doesn’t suddenly stop and say “wait, this looks like a test” when the question pops up “how would you respond.” They simply choose the best response and continue on without missing a beat. Similarly, while they may feel challenged in finding relevant information and solving problems on-screen, they don’t necessarily feel like they are taking a cognitive ability test because it is embedded in the simulation.

Thus, the combination of performance-based simulation measures (typing speed and accuracy) with traditional competency-based tests embedded within results in a highly predictive, efficient, and yes engaging assessment tool.

There are always exciting and innovative things being developed in the science of assessment. Simulations have both the classic coolness and the progressive hipness to endure as an important part of employment testing programs. Expect to see more simulations and even greater use of interactive technologies for more jobs across all levels of the organization in months and years to come.