Unproctored Internet Assessment

A Look at Real-life Data from a Large Retailer

Businesses today view unproctored internet-based assessment as a promising, cost-effective methodology for efficiently screening applicants due to the widespread availability of technology and Internet access for job applicants. However, there are many concerns with moving assessment administration from a proctored, controlled setting to an unproctored setting. These include issues of privacy, test security, comparability of testing environments, and authentication of respondents, to name a few.

Assess Systems recently had an opportunity to collect data at a large retailer and compare differences between candidate scores in proctored and unproctored assessment conditions (the client transitioned from on-site proctored testing to an integrated applicant portal via their career website).

The data presented below represent candidate assessment data using the Select for Retail Sales Associate Survey (RSA) during the client’s peak hiring season – from mid-August to December. Select for RSA measures work-related personality characteristics (Performance Index), basic abilities (Retail Math), work ethic/personal integrity (Integrity Index), willingness to do tasks required in the role, and a positive response indicator (an unlikely virtues measure to detect candidates that may be ‘putting their best foot forward’ when completing the assessment).

Findings

In year 1 (proctored assessments) the client administered over 48,000 assessments. In year 2 (unproctored assessments integrated into an automated, online applicant tracking system) over 98,000 assessments were completed. When comparing candidate data under these two conditions we found:

The overall candidate screen out rates for the two conditions was very similar. For proctored assessments the screen-out rate was approximately 28%. The unproctored assessment overall screen out rate was approximately 26%.
The Performance and Integrity Indices showed very similar scoring trends between the proctored and unproctored groups. Presented here is a chart of the distribution of Performance Index scores under each condition.

performance index

It appears that the slight upward trend in passing rates was mainly due to improvements in the Retail Math score when applicants took the assessment from home. Not surprisingly, when applicants could complete the assessment without someone watching over their shoulders, their collective scores rose. Many other test vendors have noted that ability tests administered in an uncontrolled setting are suspect (and should be weighted accordingly, if considered in the selection process at all). If used in this manner, a confirmatory test of skills should be administered in a proctored setting later in the process.
Additionally, as seen in the following graph, slight differences were seen in the Positive Response Indicator between the unproctored condition (where 15% of candidates flagged as positive responders) and the proctored condition (where 11.6% flagged as positive responders). This suggests that in unproctored conditions candidates may be more likely to try to present themselves favorably. While not used as a screen out measure, the Positive Response Index is used to identify candidates that may be “impression managing” when they complete the survey (i.e., trying to look better than they really are). Candidates identified as positive responders should be more closely vetted in the selection process.

positive response index

Conclusions/Lessons Learned

Throughout this transition the client was able to maintain the quality of hire by effectively screening candidates. Similar trends were seen in proctored and unproctored environments for the overall recommendations provided. In the client’s view, the slight discrepancy in personality assessment scores between the on-site, controlled setting and web-based administration was acceptable, given the significant process improvements associated with the web-based assessment.

Differences in test scores in two areas suggest revisions to the current process. First, the increase in retail math scores in an unproctored setting raises legitimate concerns over the quality of measurement for ability assessments in these conditions and the value of this information for the selection process. Best practices in this area suggest that a) candidates proceeding to the next step in the process could be administered a confirmatory measure of their math skills under more controlled conditions or b) assessment of math skills could be moved to a later stage in the process where monitoring could occur. Second, as mentioned above, Positive Response scores were higher in an unproctored setting suggesting that steps should be taken to encourage applicants to respond candidly to the assessment. Again best practices would include confirmatory testing under controlled conditions or a more extensive interview process to confirm the assessment findings. In both areas candidates should be instructed that secondary, confirmatory assessments may be used at a later stage in the process.

Ultimately, this comparison shows little difference in personality assessment scores (the core of this particular assessment) between proctored and unproctored settings, suggesting that concerns over unproctored testing can be minimized when combined with appropriate security and candidate verification techniques.

Rating: 5.0/5 (1 vote cast)

Tags: Unproctored Internet Assessment
Posted in Selection | No Comments »