Home > Upcoming
Luncheon > Past Luncheons
July 2008 Luncheon Summary At the special July Luncheon, presentations were made by graduate students Megan Shaw (George Washington University) and Tiffany Bludau (George Mason University). Their papers were selected by a PTC/MW committee from the papers submitted by students in response to the 2008 Call for Student Presentations. Tiffany M. Bludau Today, more applicants than ever are using the Internet to apply to new jobs, and more organizations are hiring on-line. Both employers and applicants are using the Internet heavily in finding the best personnel and best jobs available, but it is important to understand the implications of web-based recruitment systems and applicant reactions to them. When applying for positions in organizations over the Internet, many applicants are faced with two types of applications: web forms or actual files they are allowed to fill out and return to the prospective employer. With web forms, applicants fill out various fields, are not allowed any extra room to discuss qualifications, and interact only through the computer with their prospective employer. When given actual files to download, applicants are free to modify the file and insert other relevant information about themselves and then mail in or e-mail their application directly to the prospective employer. This research addresses applicant reactions to these commonly used web-based application processes and extends the literature by looking at different web-based methods. According to Gilliland’s (1993) model of applicant reactions, the formal characteristics of the selection system and the interpersonal treatment one receives are two contributors to applicant perceptions of fairness. Gilliland’s model relies on various organizational justice theories which lead to an overall model of applicant perceptions. Later on, these fairness perceptions affect reactions during and after the hiring process. The opportunity to perform is suggested as a formal characteristic which refers to the applicants’ opportunity to express themselves before any decisions are made (Thibaut & Walker, 1975). Applicants want the opportunity to display their skills and abilities to prospective employers and let employers know all of the necessary relevant information about them. Arvey and Sackett (1993) note that this can be interpreted as “voice” in the selection domain. I was interested in whether prospective employees felt that they are allowed input into the process if given the opportunity to download the application and modify it as they see fit for their application. If applicants feel that these modifications allow them some input into the selection process, they will report that downloadable application forms are fairer than web-based forms. Therefore, I hypothesized main effects for the type of form or application one receives when fairness and favorability of the method is considered. Related to voice is the two-way communication between applicant and employer. This aspect differs from opportunity to perform in that it deals directly with interpersonal experiences. Martin and Nagao (1989) found that when applicants were applying for an upper-level position, they expressed more anger and resentment when they did a computerized interview or paper-and-pencil interview as compared to when they had a face-to-face interview. Similarly, I hypothesized main effects for the status of job one applies for when fairness and favorability of the method is considered. Method Participants and Procedures Participants were directed to one of four websites in the 2 (type of application) by 2 (type of job) design matrix. Two websites were web forms to complete with follow-up measures. The other two websites directed participants to an application file which they were required to download, complete, and then e-mail back to the experimenter. Once students completed their application, they were directed to follow-up measures online. In order to encourage participant motivation, a $50 gift certificate to a local bookstore was offered to the best applications submitted. Measures Application. Regardless of condition, participants filled out the same standard job application regarding past jobs held, availability, education, etc. The only difference between the two application forms was that one was only available online (web form), with a limited space for text, whereas the other form was available in a Microsoft Word file (web application file) and was easily modified. Within these two conditions, two different conditions were presented. Participants were presented with a prompt for either an entry-level or a managerial-level position. Dependent variables. Participants in the study were asked of their perceptions of fairness of the application form using the four-item measure used by Bauer, Truxillo, Sanchez, Craig, Ferrara, & Campion (2000) to assess fairness based on their opportunity to perform. They were also presented with three general procedural fairness questions from Gilliland (1994). Participants were also asked if they generally agree with and favor the method they were given on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Demographics. Age, race, and gender information was also collected to assess if there were any moderating effects of these variables. Results Data were collected for 268 participants, with 65 to 70 participants in each cell. Participants were 76% female and 54% Caucasian. To test some my first set of hypotheses, I ran an independent samples t-test across application type. No significant differences were found, but when the sample was split on the type of job applied for, significant differences across forms were found for procedural fairness (t = -2.38, p < .05) and the favorability of the method (t = -1.79, p < .10) for lower-level applicants, lending partial support to the hypotheses. No significant differences were found across application form or job regarding one’s chance to perform. For procedural fairness, a main effect was not found for application form, but a main effect was found for the job level, F(1, 264) = 2.78, p = .09, eta-square = .01, and the interaction between application form and job level was found to be significant, F(1, 264) = 3.84, p < .05, eta-square = .01. This interaction is displayed in Figure 1 below.
Figure 1. Procedural fairness by application and job type. For the favorability of the selection method, a main effect was not found for application form, but a main effect was found for the job level, F(1, 264) = 2.83, p = .06, eta-square = .01. Means across job level and application form are displayed in Table 1. Table 1. Means across groups
Discussion These results partially support my hypotheses regarding differences across forms, but support was not found for my hypotheses across varying levels of applicants. The type of application form affected attitudes for lower level applicants, but not managerial applicants. Given that many organizations tend to rely on web-based applications and web-based forms to fill entry-level and lower-level vacancies, these results can be useful to practitioners in creating web-based selection systems. For some organizations, the practice of using web-based forms may not be practical (large pools of applicants), and for this initial test, effect sizes were low, so more work should assess these types of systems and how to improve them. More research needs to consider the applicant’s perceptions of fairness with web-based selection systems given that many organizations tend to use online systems to fill entry-level positions, especially since the largest difference in perceptions was among the lower-level applicants. References Arvey, R. D. & Sackett, P. H. (1993). Fairness in selection: Current developments and perspectives. In N. Schmidt & W. Borman (Eds.) Personnel selection: 171-202. San Bauer, T. N., Truxillo, D. M., Sanchez, R. J., Craig, J. M., Ferrara, P., & Campion, M. A. (2000) Applicant reactions to selection: Development of the selection procedural justice scale (SPJS). Personnel Psychology, 54, 388-420. Gilliland, S. W. (1993). The perceived fairness of selection systems: An organizational justice perspective. Academy of Management Review, 18, 694-734. Gilliland, S. W. (1994). Effects of procedural and distributive justice on reactions to a selection system. Journal of Applied Psychology, 79, 691–701. Martin C. L., & Nagao, D. (1989). Some effects of computerized interviewing on job applicant responses. Journal of Applied Psychology, 74, 72-80. Thibaut, J. & Walker, L. (1975). Procedural justice: A psychological analysis. Hillsdale, NJ: Erlbaum. November 2007 Luncheon
Download Kenya N. Wiley August 2007 Luncheon Summary Robert Hausmann, Nathan D. Ainspan, and Mark Tregar The War in Iraq, the aftermath of Katrina, and other recent events have highlighted questions about the way that the Federal government conducts and oversees its procurement and acquisitions, especially since the government has to acquire more goods and services with fewer resources. From FY 2000 to FY 2005, government purchasing increased 75% while fewer professionals are available to award, administer, and provide oversight to these contracts. Since the mid-1990’s, the federal acquisition workforce has declined by nearly 50% through personnel reductions. Further, with half of the workforce eligible to retire in the next four years and an acute shortage of mid-career professionals available to replace them, the Federal government has realized that it needs to strategically manage its acquisition workforce. The Defense Acquisition University (DAU) has been tasked to understand and address this shortage in the military realm. DAU provides practitioner training, career management, and services to enable the military’s Acquisition, Technology, and Logistics (AT&L) community to deliver timely and affordable capabilities to the nation's warfighters. As it works to prepare the military acquisition professionals, DAU is also partnering with the Federal Acquisition Institute (FAI) to share knowledge and best practices with the civilian acquisitions sector as well. DAU asked the Center for Naval Analyses (CNA) for assistance in developing competency models of their workforce, and then use these models to conduct a competency assessment of the workforce. CNA worked closely with DAU's Acquisition Workforce and Career Management Office to develop a competency model for the Contracting career field – one of the largest of the 13 career fields in Acquisition community. From September 2006 through February 2007, CNAC met with senior functional leaders and over 300 subject matter experts (SME's) in several regions across the country to identify the competencies required for superior performance on the job from entry to senior level. The SME's used CNA's online web-based COMPASS tool to develop the list of critical competencies and respond to questions about the applicability of the list they generated. 25 critical competencies emerged from this discussion, ranging from technical aspects of contracting to non-technical competencies such as customer service and problem solving. Meta-Analysis: An Overview Michael McDaniel During this workshop, Dr. Michael McDaniel provided an in-depth presentation on meta-analyses. A meta-analysis is a type of study where a researcher collects results from past studies and aggregates the data in an attempt to draw conclusions from a body of literature. To demonstrate the benefits of meta-analysis studies, Dr. McDaniel conducted an exercise where all workshop participants had to select ten poker chips from a bag containing poker chips of different colors. Based on the assumption that each participant represented an independent sample, the exercise illustrated the benefit of cumulating evidence across numerous samples in order to account for sampling error. Following the exercise, Dr. McDaniel discussed the two types of questions answered in a meta-analysis: (1) What is the magnitude of the effect in the population; and (2) What are the sources of variance across studies. He then provided a review of effect sizes, including an explanation of how to calculate standardized mean differences (d) and the importance of gathering studies for a meta-analysis that report an effect size, as well as its standard error. Dr. McDaniel then walked through the five stages of conducting
a meta-analysis: As part of this discussion, Dr. McDaniel stressed the importance of having a clearly defined explicit question, as this provides guidance for how to proceed with gathering studies and establishing inclusion criteria. He also provided strategies for conducting a thorough and systematic search for studies. Dr. McDaniel then discussed two different camps of meta-analysis: the Hedges and Olkin tradition and the Hunter and Schmidt tradition. Some of the major differences between the two camps are summarized in the table below.
Dr. McDaniel concluded the workshop with a discussion on publication
bias. Publication bias occurs when studies are missing from the
literature that systematically differ from published studies. He
presented four major sources of publication bias: To examine whether publication bias exists, Dr. McDaniel demonstrated Duval and Tweedie’s Trim and Fill method (see Duval & Tweedie, 2000a, 2000b), which uses a funnel plot to examine studies’ sample size and the correlation magnitude. To the extent that the funnel is symmetrical, one can infer a lack of publication bias. However, if the funnel is asymmetrical, the trim and fill method “trims” the asymmetric studies from the funnel plot identify the unbiased effect and then “fills” the plot by re-inserting the trimmed studies as well as their imputed counterparts. References The State of the Profession: I/O Psychology Lois Tetrick It was my pleasure to speak to the Personnel Testing Council of Metropolitan DC (PTC/MW) at the April Luncheon. The purpose of my presentation was to update PTC/MW members about initiatives the Society for Industrial and Organizational Psychology (SIOP) is pursuing to strengthen and advance the profession of I/O Psychology. In addition, I hoped to receive input from PTC/MW members as to what SIOP is doing and could do. To give you an idea of who SIOP is, we have approximately 3800 members including fellows, members, associates, international affiliates, and retired members and we have just over 3100 student members and student international affiliates. Based on SIOP statistics, approximately two-thirds of our members identify their primary affiliation as being a non-academic institution. Of the one-third of our members whose primary affiliation is an academic institution, slightly over half are in Psychology Departments and slightly less than half are in Business Schools. To continue to accommodate and serve our diverse and growing membership, SIOP is making efforts to develop an institutional research board. The purpose of this board will be to help us coordinate and integrate the various databases and survey data that we collect throughout the year. We are also working on procedures as to how these data may be accessed and by whom. In the past year, SIOP has been involved in formulating a strategic plan. The primary strategic goals are for SIOP to be (1) the visible and trusted authority on work-related psychology, (2) an advocate and champion of I/O Psychology to policy makers, (3) the organization of choice of I/O professionals, and (4) the model of integrated scientist-practitioner effectiveness that values research, practice, and education equally and seeks higher standards in all three areas. We have initiatives relative to each of these goals. Relative to visibility we are concerned about increasing our visibility
within the field of psychology and beyond. To increase our visibility
within the field of psychology SIOP has become a member of the
Federation of Behavioral, Psychological, and Cognitive Sciences
(FBPCS) and have increased our involvement in the American Psychological
Association (APA) and the Association for Psychological Science
(APS). SIOP has five representatives to the APA Council of Representatives.
Our APA Representatives are collaborating with other divisions
who share our interests and goals. We also have two SIOP members
on the APA taskforce for the revision of the Model Licensure
Act, which we hope will serve the interests of I/O practitioners. Lastly, SIOP is beginning an initiative to define and clarify the
scientist-practitioner model as it applies to I/O Psychology.
I believe that this initiative will identify a set of behavioral
guidelines relative to training and education, professional recognition,
and other purposes. One area where the results of this initiative
may be very useful is in establishing proficiencies for I/O Psychology.
APA recognizes four specialties in psychology – clinical, counseling, school and I/O.
Each of these specialties, except I/O, has an associated list of proficiencies.
It appears that it is time, maybe past time, for us to promulgate proficiencies
in our specialty. TSA’s New Performance Management System: Elizabeth Kolmstetter As organizations mature, the systems they use to manage and develop
their human capital must adapt and change to meet business need
and demand. In the case of the Transportation Security Administration
(TSA), the last five years have represented one of the most rapid
growth curves in the modern Federal Government. Growing from an
agency of about a dozen individuals in November of 2001 to nearly
60,000 nine months later presented the first formidable challenge
for an agency chartered with protecting the American public in
the air, on land, and in the sea. Less than five years later, TSA’s
primary human capital challenge has shifted from the recruitment
and selection of airport screeners to the development and retention
of those screeners.
The PASS framework was built to be applicable to all positions
within TSA and has been rolled out initially to 45,000 security
screeners. PTC/MW Business Meeting & Presidential Address: Theodore L. Hayes, PTC/MW President The depth and breadth of information about test-takers that can
be captured in a brief Internet-based assessment allows for integration
of organizational goals, hiring manager objectives, and test-taker
self-awareness. In short, all interested parties have increased
access to information about psychological capital from their varying
perspectives. Moreover, there is increasing pressure on assessments
coming from trans-border access and re-deployment of assessments
as part of the organization’s self-branding as an “employer
of choice.” In response, test developers must increase assessment
validity and maintain fairness. Assessment designs that incorporate
greater breadth of constructs and inference – such as basic
individual differences, meta-analyses, and transnational sensitivity – will ‘win’
in the global testing marketplace. Conversely, situationally specific
designs with limited inferences will ‘lose.’ This talk
provided an outline of some of the issues that are most important
for PTC/MW’s test and assessment experts. Finally, there
was a review of how predictions made in past President’s
Columns panned out over the year (answer – like clockwork). One way that organizations are re-aligning the assessment function with broader organization goals is to search for ways to increase “person-organization fit.” This is a growth area but one that is experiencing concomitant pains. In some cases, as with the “what type of spy are you?” program, the message sent is not clear or necessarily helpful. In other cases, as with the U.S. State Department’s self-assessment checklist, there is greater pre-applicant self-awareness created at no ongoing cost to the organization. Another alignment of assessment functions with the three cultures involves taking responses to non-cognitive predictors seriously. That is, rather than scoring assessments or assuming that someone is “faking” when he/she responds to a personality or biodata assessment, there is a wealth of self-disclosure that could be better utilized to increase prediction and ease the on-boarding process for a new employee. This may be more salient when the applicant and the organization are mis-aligned in terms of geography or cultural expectations. A final alignment would come in the form of taking meta-analysis results seriously. There is a wide gap between meta-analysis marketing – which is not science – and meta-analysis results, which are increasingly the way behavioral scientists communicate. It was argued that PTC/MW members should be able to distinguish marketing from results, and should capitalize from the hundreds of published (not to mention unpublished) studies available to improve the quality of assessment. The presentation closed with the observation that there has never been a better time to be a personnel assessment professional. The demands and rewards of measuring and understanding human performance at work have never been greater. PTC/MW is the country’s premiere local organization for assessment professionals, which means that PTC/MW’s Board has an important responsibility to raise and maintain very high standards. Testing 2.0? Issues and Controversies in the Corporate Market for Selection Testing Doug Reynolds, Ph.D. A combination of trends and technical advancements has pushed the use of
selection tests toward new operational conditions. Beyond the use of the Internet
for delivery, these factors include: the need to integrate multiple assessment
systems, the introduction of regulations regarding Internet applicants, the
constant push to eliminate adverse impact, and the pressure to meet multiple
purposes with a single assessment. Each of these issues has placed new pressure
on the testing market. This presentation will briefly cover these trends and
provide current examples of deployment scenarios that were not possible in
the recent past. Consequences will be examined from case examples as well
as from research conducted across deployment conditions, including results
from research on an operational test where candidates were coached on a non-cognitive
measure. Strategic Recruiting and I/O Psychology: Today’s Tools for Applicant Attraction and the Role of I/O Psychologists in Making Recruitment Work Rich Cober Recruiting and selecting top talent is an organizational imperative in our global economy. Today’s companies have more tools at their disposal than ever before to do the job of advertising career opportunities. Many of these tools have been developed from the technology afforded to practitioners due to the expanded use of the Internet for employee recruitment. This presentation provided (a) a framework for employers trying to make sense of today’s talent marketplace, (b) an overview of the tools and channels available to attract today’s talent, and (c) some thoughts on the role that I/O psychologists should play in helping to shape recruiting strategy. Despite mixed signals in the United States economy, the employment marketplace remains very strong. Wages have held relatively steady or increased in recent reports, the national unemployment rate stands around 4.7%, and unemployment in the Metro DC area is under 3%. Finding skilled workers in this type of market is a challenge, particularly for those organizations focused on building talent pipelines with active job seekers. Therefore, the skills of recruiters and strategies for leveraging recruitment channels are becoming increasingly important (despite the trend in HR to outsource some functions, including aspects of recruiting). As companies develop their talent strategy – an understanding of how job seekers search for employment opportunities and are affected by employment marketing materials can go a long way to building a winning talent acquisition solution. For guidance in how best to market to different groups of job seekers – lessons can be learned from the I/O Psychology, Marketing, and Strategic HR research/ professional literatures. One effort to build a theory of job seeker behavior based on findings from these literatures was shared during the presentation (Cober, Brown, Keeping, & Levy, 2004). This model suggests that the emotional reaction a job seeker has on first accessing an employment-related web site (e.g., excitement) will affect their willingness to use that site to acquire employment information and potentially apply for jobs. Once using the web site, job pursuit intentions build within the individual to the extent that the job seeker finds information that is relevant to his/her needs in a relatively painless fashion. To the extent that the online talent dance unfolds in a manner that peaks a job seeker’s interest in an organization, they will apply for employment and be willing to deal with whatever early selection hurdles an employer demands. Understanding job seeker behavior on the Internet is critical for recruitment strategists given current use of online channels for job pursuit. A survey that I conducted with Doug Brown from the University of Waterloo earlier this year found that 51% of new hires in 2005 were sourced from the Internet (Cober & Brown, 2006). Knowing that the Internet is being heavily leveraged by job seekers begs the question of which specific online channels are being used and how their use impacts the bottom line of hiring efforts. The big job boards such as Monster are certainly prevalent players in the marketplace (and seem to be re-trenching themselves in print media – see Workforce Management’s October 23, 2006 issue). However, considerable discussion has taken place in the professional literature regarding the true ROI of the big boards – trying to distinguish hype from substance. In reaction to some of this discussion, more niche boards such as Dice.com have emerged to service the online hiring needs of specific professional communities. Even our own SIOP web site features a niche board that provides an important place for I/O Professionals to learn about employment opportunities. Social networking sites have emerged on the scene during the past couple of years. These sites include Linked-in.com, MySpace.com, and Jobster.com. Companies have long valued the employee referral as a critical source of new hires, and many employers report that referrals are their most robust source of new hires today. Social networking sites have the promise of combining the relationships that feed referrals with the tracking technology behind the web to provide for a dynamic online source of talent. Vertical search engines such as Indeed, JobCentral, and Google are also making plays for relevance – quite convincingly. These sites allow job seekers to enter key words in one search engine, and then view listings from across job boards and employment web sites. Recently Craig’s List filed an injunction against one of these engines to stop accessing postings from its site for searches. The direct connection that these sites promise with employment pages across the web could change the way job search is facilitated, or perhaps become a controversial engine if more boards begin to protect their online space against such technology. Finally, most employers, if not every employer, has their own web space – and for major organizations, the employment portal of that space is one of the most heavily viewed areas of the web site. Effectively leveraging that space is critical for successful hiring. The options available to employers for recruiting have not totally translated to the greater efficiency and effectiveness in hiring practices once anticipated, and perhaps promised, by the first generation of leaders in the online recruiting industry. While improvement has been realized in both the efficiency of processing and effectiveness of targeted hiring efforts, increased variety has presented increased complication in how best to use online sources for hiring and in determining the value of sources for sustainable recruiting activity. The competency of I/O professionals in statistics and research methods can provide organizations with great insight around the utility of different recruiting options. However, the presence of I/O in the recruiting/staffing departments of organizations is not as commonplace as one might think. A real opportunity for I/O exists in helping companies to craft hiring strategies based not on the marketing prowess of recruiting vendors, but on the testing and demonstration of results that can be achieved through the use of such vendors. References Cober, R. T., Brown, D. J., Keeping, L., & Levy, P. E.
(2004). Recruitment on the net: How do organizational web site characteristics
affect applicant attraction? Journal of Management, 30(5), 623-646. Current Legal Challenges in Employment Testing Kathleen Lundquist R. Lawrence Ashe, Jr. Summary by: The well-attended June workshop focused on the recent key features of employment litigation cases with highlights on both the positive and negative actions by applied psychologists. After reviewing the concepts of prima facie discrimination, business necessity, and alternative practices with lesser impact, Mr. Ashe presented dozens of new employment case decisions in brief with an eye toward trends in the salient decision-points in the court’s reasoning. Two trends in particular stood out. First, consultants and researchers are doing a better job of grounding their work in job analysis, which leads to a more logical correspondence between the test and proposed criteria – the job-relatedness argument. Because testing programs are increasingly more logically job-related even when adverse impact results, this leads plaintiffs to propose that alternative predictors should have been chosen. Consequently, another trend in court decision-making is for courts to demand that plaintiffs outline what alternatives would provide comparable job-relatedness or meet business necessities rather than simply proposing that such measures may exist. This requirement requires research evidence rather than rhetoric, which can be a burden for plaintiffs. It is not all “a walk in the park” for defendants, though. In several cases reviewed by Mr. Ashe, courts rejected defendants’ over reaching claims for test implementation based on superficial job analyses (especially ones based largely on job descriptions) or cut-off scores that had no sensitivity to business necessity. Finally, in some cases defendants undermined their own claims of business necessity by resorting to random selection within a band – thus implying the potential validity of an alternative selection strategy. Dr Lundquist noted that plaintiffs have changed their behavior based on better test development and implementation. Practitioners and consultants have notably improved the quality of their work regarding job analysis-test content linkages, job relatedness, cut score setting, and evaluations of potentials with lesser adverse impact. This makes other organizational decision-making points – performance reviews, promotion decisions, etc. – more inviting targets for plaintiffs’ attorneys. Dr. Lundquist made three substantive suggestions for organizational practitioners and consultants. First, it is critical to establish test programs that are grounded in job analysis as well as sound test-development practices and standards. This means test content, format, and cut scores need to have a logical relation to the job functions. Second, practitioners must audit process to ensure that there is ongoing validity, fairness, and standardization. Finally, organizations need to (over) communicate to applicants concerning test-based decisions in order to defuse misunderstandings or claims of disparate treatment in the evaluation process. PTC/MW thanks the Human Resources Research Organization (HumRRO) (www.humrro.org)
for sponsoring the June workshop. Moving Beyond DOT and O*NET: R.J. Harvey, Ph.D. At the luncheon, Dr. R.J. Harvey shared his views about O*NET and discussed what he believes is a better strategy for dealing with the loss of the Dictionary of Occupational Titles (DOT) — one based on defensible but practical job analysis combined with the empirical methods of linking job analysis results to worker-requirements using job component or 'synthetic’ validation. Dr. Harvey started the luncheon by reviewing the history of the O*NET. Developed in the 1990s, the O*NET was created to replace and significantly expand on the DOT, a definitive listing of occupational titles and work-related information (e.g., worker functions) maintained by the U.S. Department of Labor (DOL) since 1939. Although originally intended to serve as the most comprehensive source of occupational information in the U.S., Dr. Harvey noted there were number of issues that greatly limited the O*NET’s value in this regard. These included, but are not limited to:
To address these issues, and as an alternative to the O*NET’s approach to job analysis, Dr. Harvey argued that we need to return to proven job analysis “basics.” Specifically, such an approach would:
The last of these is especially critical. Dr. Harvey emphasized that job component validation represents the most viable and defensible approach to establishing and validating these linkages. Besides being rooted in behaviorally-based job analysis, job component validation carries a number of advantages. By taking a job component validation approach, one can:
To illustrate some of these ideas, Dr. Harvey concluded the presentation by summarizing his work to construct and populate a national occupational database using his own Common Metric Questionnaire (CMQ). For a copy of Dr. Harvey’s presentation, click here: (and then you implement): The Art of Implementing Good Science Amy Grubb Suzanne Tsacoumis Consider the following situation: You have just completed the design and development of an assessment process for which you have collected a wealth of validation evidence. Following professional, technical and legal guidelines, you are proud of the end result and look forward to its use. Now what? Although the test development may be the most intriguing part of the effort, a key ingredient to the efficacy of a human capital program is in its implementation. In fact, if not executed well, the validity and integrity of the system could be jeopardized. Successful implementation requires a balance between the technical rigor of the development and validation with a mind toward organizational change and the use of research and communication skills to persuade appropriate usage. During this session, Drs. Amy Grubb and Suzanne Tsacoumis addressed the administrative, technical, and communication considerations and challenges from both the internal and external consultant perspective, including practical tips and recommendations for getting tests to be used operationally. The session also included a lively discussion among attendees who shared their own implementation issues and success stories. Administration Considerations:
Issues and Tips Administration issues are at the core of implementation and tend to reflect the most obvious concerns. Among the key issues to be considered are:
In addition, the anticipated applicant flow has a direct impact on many of the decision points noted above. It is imperative to consider administration issues well in advance of the development of the assessment methods since they impact the nature of the techniques as well as create the infrastructure for the assessment process. A few administration tips discussed during this session included:
Technical Considerations:
Issues and Tips There also are various technical issues associated with the implementation of a testing process. For example, will a cut score be established and if so, how? Alternatively, an organization may want to consider top-down selection or using a banding approach, both of which could be used in combination with a minimum standard, if desired. If more than one measure is used, the organization must determine whether it will use a multiple hurdle approach or a compensatory model. Additionally, if different assessment methods are used, consideration must be given to the weights associated with each tool. Some of the technical tips noted include:
Communication Considerations:
Issues and Tips The final broad category of implementation issues is communication. Often acknowledged as a critical, yet poorly executed component, effective communication should focus on providing all the requisite information tailored to the appropriate audiences and delivered in a timely manner. The audiences may include stakeholders such as hiring officials, executives, and special interest groups, as well as any individuals impacted by the development and implementation of the system. The communication should be mapped-out as part of a comprehensive plan and should emphasize two-way communication. Additional communication tips include:
Concluding Thoughts Drs. Grubb and Tsacoumis concluded their talk by stressing the importance of flexibility since it is impossible to anticipate all potential issues and roadblocks. Further, it is important to recognize that each element of the implementation framework is interdependent. To the extent possible, the implementation strategy should be defined prior to initiating tool development. Then, at every decision point, one should ask: (1) Who is going to do this? (2) How are we going to do this? and (3) When is this going to be done? Both “good science” and an organizational change perspective
are prerequisites for successful implementation and good return on investment. Developing a National Certification Program: Issues and Challenges Gavan O’Shea This presentation described the American Institutes of Research’s (AIR) effort to develop a nationwide certification program within the electrical industry, with a specific focus on several key milestones during the development process where the work’s national scope raised interesting issues. These milestones included the development of the job analysis survey, sampling, and data analysis plans, the creation of test specifications, and the item writing process. Each is discussed briefly below. Job Analysis Descriptor Review
Job Analysis Survey Design In light of these concerns, AIR designed the JA survey to reflect such potential differences. Specifically, prior to rating a given duty, task, or KSA statement, respondents were asked whether they performed the duty/task (or required the KSA) as a part of their current job. If participants answered “no,” they moved on to the next descriptor. If they answered “yes,” they provided more detailed ratings for that descriptor (e.g., importance, frequency, difficulty, acquisition requirements). While allowing AIR to investigate potential variation in occupational requirements, this type of design raised a number of important considerations:
Once the survey design phase was completed, AIR employed a stratified multi-stage sampling plan to create a sample and field the JA survey. Specifically, this consisted of selecting electrician training sites during the first sampling stage, and electricians within those sites at the second stage. Because the number of electricians is not distributed equally across the U.S., the selection of training sites was weighted to reflect this geographic disparity by selecting relatively more sites from those regions with more electricians. Development of Content Specifications Finally, Dr. O’Shea briefly described the process employed to develop the certification test items. The subject matter experts who wrote these items were experienced instructors, and many had considerable expertise writing end-of-year multiple choice test items within the context of an electrician training program. Once this group completed an item-writing training session, they were assigned a target number of items to write within a content area (e.g., bending conduit) that reflected their area of expertise. A rigorous review process was then employed to create items suitable for pilot testing. Dr. O’Shea focused specifically on the challenges raised during the review process by the items’ technical content. This situation dictated close collaboration with the client: while AIR’s own internal item edits focused on item clarity and readability, the client’s technical expertise was utilized to locate any content-related flaws. Additionally, AIR worked closely with the client to verify that their structural improvements did not change the technical correctness of a given item. Another technical challenge faced during the review process involved ensuring the consistent use of terminology and abbreviations across items. In closing, Dr. O’Shea emphasized that the goal of the presentation was to discuss some of the challenges encountered during the test development process and to describe how we dealt with them. In his experience, the strategies and solutions to challenges faced when developing certification programs are inextricably linked to the program’s unique goals, structure, and context. The presentation focused on key aspects of work done to date, though there is still much work left to do (e.g., implementation, maintenance, revision). What Do Federal Employees Say? Results from the Federal Human Capital Survey Leslie Pollack Summary by: For the January Luncheon, Dr. Pollack shared the methodology and findings from the 2004 Federal Human Capital Survey (FHCS), conducted by the U.S. Office of Personnel Management (OPM). First conducted in 2002, the FHCS asks Federal employees for their views on leadership quality, performance culture, and talent capacity within the Federal workforce. Overall, results of the 2004 survey indicate that Federal employees:
Information on and snapshots of results from the 2004 FHCS can be found at www.fhcs2004.opm.gov. Dr. Pollack reported that nearly 150,000 Federal employees responded to the
88-question survey—a response rate of 54%. Following the administration
of the FHCS, OPM conducted research on response rates and reasons for nonresponding. articipants tended to stop responding at a greater rate during the first half of the survey than the second half of the survey. In addition, more people stopped responding once they reached the demographic questions at the end of the survey. In fact, the total number of participants who stopped taking the survey when they reached the demographic section outnumbered all of the participants who opted out during the rest of the survey. As part of this research, OPM also conducted a nonresponse study that examined why employees did not respond to the FHCS and potential differences between FHCS responders and nonresponders. Dr. Pollack shared that although only preliminary results are available, nonresponders reported “being too busy” as the primary reason for not completing the survey. Other reasons reported by nonresponders include “the results won’t be used to change anything,” “the survey is not important,” and “the results of the survey are never shared with employees.” Development and Implementation of a Cross-Occupational Biodata Measure: What We’ve Done and Where We’re Heading Julia McElreath Julia McElreath, 2005 President of PTC/MW, is a Personnel Research Psychologist with U.S. Bureau of Customs and Border Protection (CBP), Department of Homeland Security (DHS). Her work for the DHS focuses primarily on developing and validating entry-level selection systems. Prior to her position with the DHS, Julia worked in assessment and selection for the former INS and the FBI in Washington, D.C., and for Michigan Consolidated Gas Company in Detroit, MI. Julia earned her Ph.D. in Industrial/Organizational Psychology at Wayne State University in Detroit, MI. This talk was organized into two parts: the first part was a discussion of the recent work on biodata that has been conducted at Customs and Border Protection (CBP). Topics of the biodata discussion included the purpose of the study, background on uses, advantages, and development of biodata, methodology for the recent research, analyses and results, and future directions for biodata at CBP. The second part of this talk was an update on the current and future state of PTC/MW, including gratitude for the 2005 board and committees, accomplishments for the year, and upcoming 2006 events. The impetus for the recent work on biodata at CBP was the formation of the Department of Homeland Security (DHS), which raised a number of assessment-related issues. Some of these issues included which occupations were to be served by the assessment development group, what assessments have been used in the various occupations and in the legacy agencies, and what assessments were appropriate to use as DHS moved forward. With respect to biodata in particular, the goal was to retain a biodata measure in entry-level selection batteries for a variety of occupations, and to the extent possible, use the same biodata measure across occupations. Several questions needed to be addressed in order to meet this goal, such as what measures were available, how well these measures have worked in the past, and which measure is likely to work best in the future. Biodata in general has demonstrated incremental validity over cognitive measures and has demonstrated lower adverse impact than more cognitively loaded predictors. Also, biodata measures tend to allow for a more well-rounded assessment of applicants than cognitive measures alone. For these reasons, it was desirable to retain a biodata measure in the selection batteries across occupations. This biodata study started with identifying biodata measures and data sources. Four biodata measures were identified: A-Empirical, B-Rational/Empirical, C-Combined, and D-Best Empirical. Validation data (sample sizes ranged from 348 to 757) and applicant population (population sizes ranged from 3,370 to 26,628) data were identified for six occupations (Border Patrol, Customs Inspectors, Immigration Inspectors, Special Agents, Immigration Enforcement Agents, and Adjudicators). Criterion measures in the validation data included training scores from the training academies, scores on a work simulation exercise, and supervisory ratings of performance. Both the work simulation exercises and the rating scales were developed specifically for the validation studies. Validity, reliability, and effect sizes were calculated where possible and appropriate. It should be noted that not all measures (predictors or criteria) were available in all data sets (for example, training data were available for Customs Inspectors, but work simulation exercise scores were not). Cross-occupational indices (such as weighted average validities) were then calculated. Weighted average validities for the four biodata measures ranged from .26 to .36 for training scores, from .17 to .25 for work simulations, and from .18 to .26 for performance ratings. Reliability estimates for the biodata measures were .90 (Biodata A-Empirical), .74 (B-Rational/Empirical), .88 (C-Combined), and .87 (D-Best Empirical). Further analyses included estimates of effect size, which were uniformly low, as expected, and ranged from .03 to .43 for the White-African American comparison, from .00 to .36 for the White-Hispanic comparison, and from -.12 to .11 for the male-female comparison. A rank ordering of the effect sizes revealed that the B-Rational/Empirical measure had the lowest effect sizes across comparisons. The results from the validity, reliability, and effect size analyses were all taken into consideration when choosing the most appropriate biodata measure for operational use across occupations. Biodata measure B (Rational/Empirical – sorry, Henry!) was selected to be included in the predictor batteries across the occupations included in these analyses. Future research will include the collection of applicant data in three occupations. The biodata measure that is being administered includes the items for measure B (Rational/Empirical), but also includes the items for measure D (Best Empirical – yea, Henry!), and construct-based biodata scales that have shown promise in an initial double-cross validation study in one occupation. Also, there is a new validation study being planned for late 2006 in one occupation, and all the relevant biodata measures (B-Rational/Empirical, D-Best Empirical, and construct-based scales) will be included in that effort. Part two of this talk focused on the state of the PTC/MW organization. First, Julia thanked the 2005 board members (Ellen Kollar, Ted Hayes, Martha Hennen, Tim McGonigle, Jennifer Hurd, and Chris Parker) and committee chairs (Ellen Kollar, Pat Curtin, Cassi Fields, Lia Meyer, Ben Porr, David Hamill, and Lance Seberhagen) for their dedicated service, enthusiasm, and wisdom. Julia then gave a brief overview of some accomplishments for the year, including a terrific calendar of speakers, four volumes of the new PTC/MW Quarterly as well as the monthly electronic updates, a Bemis award winner for the second year running (Ilene Gast), online ballots for the 2006 elections, an overhaul of the PTC/MW website, and a balanced budget, just to name a few. Finally, Julia offered congratulations to the 2006 board members – Ted Hayes, Cassi Fields, Jennifer Hurd, Greg Beatty, Sue Hay, and John Ford. The Merit Principles Survey 2005 and Related Research at the Merit Systems Protection Board John Ford John Ford is a Research Psychologist in the Office of Policy and Evaluation at the U.S. Merit Systems Protection Board (www.mspb.gov). MSPB's Office of Policy and Evaluation (OPE) conducts studies to assess the health of Federal merit systems and to insure they are free from prohibited personnel practices. This presentation reviewed several studies recently released by OPE and discussed preliminary findings of the government-wide Merit Principles Survey 2005. Results of MSPB research studies are formally presented to the President of the United States and the U.S. Congress. These studies are also designed to inform other stakeholders, such as agency leadership, Federal HR practitioners, public sector managers and supervisors, and public policy researchers. The following MSPB reports were briefly discussed. They are available from the "Studies" section of MSPB's web site at www.mspb.gov/studies/mspbstudiespage.html.
OPE's current research agenda includes studies of the conflict resolution process used by first-line supervisors with their employees, fairness and equitable treatment of Federal employees in protected classes, factors that affect Federal employees' level of engagement in their work, and the effective use of training in developing employees. MSPB encourages interested researchers and practitioners to provide input about these topics during 2006. The presentation included a discussion of the 2005 Merit Principles Survey (MPS). MSPB has conducted this survey every 3-5 years for the past two decades. Each administration of the MPS tracks the incidence of prohibited personnel practices among Federal employees, assesses the degree to which merit principles are followed, and gathers information to support OPE research studies. Nearly 37,000 full-time civilian Federal employees completed this year’s web-based MPS. They were randomly sampled from 14 agencies and represent viewpoints of employees at the non-supervisory, team leader/supervisor, and manager/SES levels. Just over a fifth of the MPS 2005 participants (23%) will be eligible to retire in the next year. MPS 2005 results show that Federal employees understand the missions of their respective agencies (95%) and the great majority believes that their agency’s mission is important (94%). On an individual level, employees understand how their work contributes to the agency’s overall mission (92%) and find the work they do meaningful (86%). Most also believe that their work teams provide high quality products and services to the public (80%). They are less certain that their jobs are secure (60%) or that they have the resources they need to do their work (64%). Although Federal employees are committed to their roles, they are less satisfied with how well they are supported in these roles. In spite of this relative dissatisfaction, many Federal employees like their jobs (71%). A greater number are motivated by such factors as pride in their work (98%) and their duties as public employees (90%) than by direct rewards, such as a hypothetical cash award of $1,000 (71%) or increased chances of promotion (70%). Fewer employees (61%) are satisfied with their pay, or with the recognition and awards they receive (40%). These findings portray a Federal workforce that is well motivated by personal factors, but less satisfied with support and rewards from their employer. The MPS 2005 also gathered information about employee training needs. Although most survey participants report that they receive sufficient training to do their jobs (64%), a sizeable minority would like additional training to improve their job performance (46%). Many participants responded to the opportunity to write a description of the type of training they need most. They clearly expect that development of these skills should be funded by agencies through formal training (81%) and on-the-job developmental experiences (77%). However, only one third (33%) have communicated their training expectations to their agencies in career development plans, and only half (48%) see their supervisors as a source of opportunities to improve their skills and performance. Better communication between employees and their agencies may be needed if Federal employees are to receive the training they need. Further information about the results of the MPS 2005 will be made available in the MPS 2005 project report and in MSPB's "Issues of Merit" newsletter. Transportability of Physical Performance Tests Todd Baker and Deborah Gebhardt Drs. Baker and Gebhardt discussed some of the relevant issues related to physical performance tests and the means of establishing transportability evidence to support such measures. Dr. Gebhardt kicked the discussion off with an overview of the physiological bases of tasks performed on the job. This part of the presentation acknowledged that understanding the physical demands of a job requires knowledge of the physiological systems of the body, how these systems are involved in task performance, and how working conditions may impact these systems and the physical demands of the job. Dr. Gebhardt highlighted six physiological components that can be involved in performing job tasks at varying levels of physical demand. These were: muscular strength, muscular endurance, aerobic capacity, anaerobic power, equilibrium, and flexibility. Each physiological component was defined, along with working conditions that might impact each component. Examples of tasks requiring each ability were discussed. Muscular Strength was defined as the ability to produce a force and power to resist a stress. This component involves lifting, pushing, pulling and holding, and includes upper and lower body and trunk muscle groups. The following conditions impact demonstrations of muscular strength:
Muscular Endurance was defined as the ability to apply force continuously/repeatedly for 1+ minutes. This involves lifting, pushing, pulling and holding, and includes upper and lower body and trunk muscle groups. The following conditions impact demonstrations of muscular endurance:
Aerobic Capacity was defined as the ability of the respiratory and circulatory systems to transport and convert oxygen into energy (e.g., adensosine triphosphate). Aerobic capacity or cardiorespiratory endurance is involved in tasks performed continuously over long periods of time (e.g., 5 -20 minutes or greater) and with a specified intensity. Conditions that impact demonstrations of aerobic capacity include:
Anaerobic Power was defined as exerting all-out physical effort for brief periods of time (e.g., 5–90 seconds). This ability involves the use of stored energy to perform short duration tasks (e.g., run 200 yards, run up 3 flights of stairs, restrain resistive suspect). Conditions that impact demonstrations of anaerobic power include:
Equilibrium was defined as keeping or regaining one’s balance or staying upright when in an unstable position. This includes maintaining balance when changing direction or when forces are being applied to knock one off-balance. Conditions that impact demonstrations of equilibrium include:
Flexibility was defined as the ability to use a range of motion at the joints and the elasticity of the muscles and tendons to stretch when bending, twisting, or reaching out. Conditions that impact flexibility include:
This overview helped the audience to understand and appreciate task requirements at this level of analysis. After providing this overview, Dr. Gebhardt turned the presentation over to Dr. Baker, who discussed transporting the validity of a physical performance test. Dr. Baker reviewed the standards of evidence needed to transport a test as outlined in the Standards for Educational and Psychological Testing (1999), the Principles for the Validation and Use of Personnel Selection Procedures (2003), and the Uniform Guidelines on Employee Selection Procedures (1978). He summarized the relevant requirements as requiring:
Various approaches to establishing whether jobs are “substantially similar” were described. These approaches are based on similarity in tasks, abilities (KSAs), working conditions (ergonomic/environmental parameters of job tasks), and physiological measures, among others. Suggested questions to consider in deciding what approach to use and how to interpret the similarity information included, “What level of similarity is acceptable?” and “Does similarity apply to the entire job or just components related to the test (in this case, physical aspects of each job)?” The relative strengths and weaknesses of taking each approach to establishing job similarity were delineated, and some practical methods of conducting each type of analysis were provided. For example, in establishing task similarity using a common list of tasks, the job analyst can determine the number of common and unique essential tasks between two jobs and express the similarity as the percent of overlap in essential tasks between the jobs. % Overlap = Common Tasks * 2 In computing ability (KSA) similarity, Dr. Baker suggested the following method. Collect SME ratings of essential tasks on behaviorally anchored rating scales where anchors have known ability requirements. Calculate the mean ability requirement across all essential tasks for each job, and compare the profile of ability requirements between the two jobs of interest. To establish similarity using ergonomic/environmental parameters, specific, pertinent ergonomic questions are asked about each job (e.g., what requirement does each job have to run distances?). Responses to these questions in terms of ergonomic parameters are established and compared to evaluate similarity. Finally, to establish similarity based on physiological measures, such measures (e.g., VO2, force, torque) are considered for each job task and comparisons of the resulting data are made. No one measure of similarity may adequately determine whether jobs are similar, so multiple methods should be used. An example of compiling these data via multiple methods for four specific jobs was provided. The objective of this project was to evaluate job similarity and transport validity evidence for the job of Nuclear Security Officer across four organizations. Job analysis, task data, KSA data, working conditions, and physiological measures were all analyzed to review the evidence supporting job similarity. The combined evidence allowed Human Performance Systems, Inc. to evaluate evidence supporting test transport through comprehensive understanding of job similarity. Another method for evaluating job similarity is the Bland-Altman approach. Borrowed from the medical field in which it is used to determine the similarity between new and established medical instruments, this approach analyzes the difference in values of ratings collected about each job. Means and standard deviations of the difference in rating values for each job are plotted and the resulting scatterplot is reviewed to assess similarity. One strength and several weaknesses of this method, as compared to the previously described calculations, were identified. Dr. Baker concluded the presentation by identifying several critical components needed for the transportability of tests. These components are:
Development of an Internet-Based Tool to Assess the Competencies of Acquisition Professionals Lance Anderson Vinay Nilakantan The presenters discussed their work on a project for the General Services
Administration (GSA), for which they developed an Internet-based competency
assessment tool for GSA’s 1102 Series Acquisition Workforce: the Applied
Learning Center (ALC). A summary of that presentation is provided below.
In addition to these issues, acquisition professionals need guidance on where to improve their skills and they need recognition for continued development. GSA’s solution to the above-listed challenges is the ALC, an online,
competency-based assessment tool that is being used to measure skills on 15
competencies. The purposes of the ALC are to measure skills gaps, assess the
competency of GSA’s Acquisition Workforce, and provide a way for eligible
GSA associates to meet the OPM 1102 Qualification Standard. The ALC items were developed through a six-phase process that included conducting a job analysis, conducting a background materials review, drafting assessment items, gathering subject matter expert ratings and feedback on items, administering beta tests, and determining target scores. The keys to the success of the item development were ensuring that the items reflected the content of the job (competencies used were identified by the Federal Acquisition Institute), use of high quality, up-to-date resource materials, and structured review and refinement of multiple versions of each item by at least seven independent subject matter experts. The items were then assigned to one of three phases of assessment, where
each phase has particular organizational and employee-based objectives. Exhibit
1 describes these three phases. The Objective Assessment consists of multiple-choice questions covering 11
competencies and GSA-specific knowledge, most of which are based on case studies.
A total of 495 questions are included on this assessment. Associates could
not “fail” the assessment; feedback is presented in terms of high,
medium, low skill gaps. The Full Assessment consists of multiple-choice questions covering 16 competencies,
most of which rely on case studies. This phase includes over 320 items. Users
passing the Full Assessment can earn college credit, training credit, and,
for 1102’s that are grade 13 or above, a waiver to the 1102 OPM Qualification
Standard (OPM, 1999).
The ALC objective assessment has already reached virtually all 1102’s:
96 percent of the 1102’s participated in its initial operational use,
just completed in July.
For more information, contact Lance at landerson@caliber.com or Vinay at
VNilakantan@meridianksi.com. Certification = Employment Test? Dr. James Sharf Dr. Sharf provided an overview of some of the issues arising from recent activity in Washington regarding the definition of an applicant and the implication of such efforts on the work of I/O Psychologists and others. His particular focus was on the impact of these proposals on licenses and certification requirements and the definition of such minimum qualifications. In an internet connected world information about job vacancies is immediately made available to a world wide pool. Federal EEO enforcement agencies have proposed defining “e-Applicants” and challenging internet recruitment. Dr. Sharf pointed out that employment recruitment practices frequently include licensing and certification “minimum qualification” requirements to receive further consideration. There has been much discussion regarding the definition of an applicant from the standpoint of the Uniform Guidelines on Employee Selection Procedures. EEO enforcement agencies have proposed the Adoption of Additional Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures as They Relate to the Internet and Related Technologies. Under these proposals, the evaluation of such licenses and certification used as part of many recruitment strategies would constitute an employment exam subject to review under the Uniform Guidelines. When these additional questions and answers are adopted, minimum qualifications including licensing and certification will be newly burdened under the Uniform Guidelines. While employment tests are defended as “job related,” minimum qualifications are not because they include knowledge and skills expected to be previously mastered. Licensing and certification exam developers have not had much input from I/O psychology because such externally administered minimum qualifications have not been challenged as “employment decisions.” Employers choosing to use such licensing and certifications as part of their recruitment strategy will likely take on the burden of reviewing their use as part of their employment process. Prospects for greater I/O involvement in licensing and certification exam development and defense are looking green. Assessing Leadership Potential at the FBI Dr. Amy Grubb Dr. Seymour Adler The FBI has created a new system for evaluating Special Agents for promotion to positions at the middle management levels (GS-14 and GS-15 levels). The primary business need of the organization was to develop a system that assessed overall managerial skill and potential. The FBI had been publicly criticized for its lack of management and leadership. Therefore, the Director of the FBI mandated that leadership become a core value for the organization to help it fulfill its mission and public obligation. Empirically, there was overwhelming evidence from a number of job analyses that it was managerial and leadership skills, more than technical skill (e.g., skill in White Collar Crime, or Counterterrorism, or Cybercrime, etc.), that defined success as a manager at the FBI. Eight core managerial competencies emerged as the keys to success, and these served as the foundation for the new promotion system. The set of target competencies addressed leadership, interpersonal abilities, planning and organizing, and initiative, among other areas. The new process was also intended to reduce subjectivity and increase the perceived fairness of the system. A core element of the new system is the Leadership Skill Assessment (LSA), which uses assessment center methodology delivered telephonically to evaluate target managerial competencies. The FBI enlisted the assistance of Aon Consulting to develop, validate and administer the LSA. To help assure credibility and cultural acceptance, the FBI required that the LSA be face valid—fully customized to the demands of middle-management positions at the Bureau. FBI executives and the project liaison were actively involved in the development of simulation exercises, set specific behavioral standards for each competency and sub-competency on each simulation exercise, and helped pilot and content validate the program. Two separate LSAs were developed, for the GS-14 and GS-15 levels respectively, and two parallel forms were developed for each LSA. Each version of the LSA is comprised of multiple role-play scenarios and background materials that allow the participant to prepare for, and perform in, the live telephone exercises, interacting with a team of Aon Consulting’s trained professional assessors. Once the LSAs were developed, the greatest challenge of project roll-out was delivering a rigorous assessment process in a relatively short time frame (initially three months of test administration) for a relatively large, de-centralized population (over 3,000 applicants) spread across the country and the world (over 50 locations). Telephone delivery of the LSA allowed the FBI to implement standardized assessment in the face of these enormous logistical challenges and minimize the time that participants, all Special Agents, were taken off their critical day-to-day responsibilities. Delivery of the LSA by a remote and centrally managed team of professional AON Consulting assessors also allows the FBI to maintain complete security on the identities of all participating FBI Special Agents and to minimize any biases in the assessment process. Aon assessors are unaware of the participant’s name, FBI ID number, tenure, race, or current position. In short, telephonic administration of the LSA allows for the benefits of unbiased, anonymous assessment in a timely and cost effective manner while maintaining strong psychological fidelity to the nature of the target jobs. All assessment ratings are entered directly into an on-line system and all exercises are automatically recorded using digital recording technology. Aon assessors rely on the evaluation standards set by FBI subject matter experts. Scores on each sub-dimension and competency dimension are generated mechanically by the automatic averaging of the many ratings provided by independent assessors across a number of exercises. Quality assurance activities included capturing and re-rating exercises randomly from the digital recording system, and statistical process control of on-line ratings down to the level of the individual exercise, dimension, and assessor. This allows for the identification and correction of relative leniency, halo, and/or skewness. In addition, assessors are monitored to ensure consistency in each assessor’s role-playing. All quality assurance data is used to provide feedback and coaching to assessors. Several key factors are required for success in a major undertaking of this scope and importance. Buy-in has to be high-level and broad. In this case, senior leadership supported the process and Legal, HR, and employee associations were involved throughout. Second, outside consultants need access to knowledgeable job experts to assure that simulation scenarios are realistic without being excessively technical and capture the agency’s culture, and that assessment guidelines reflect the standards of the organization. Finally, program design has to reflect the exigencies of implantation. Here, close cooperation between the design and implementation teams, within and across the FBI and Aon Consulting, produced a tight partnership that enabled implementation against an imposing array of logistical challenges. Assessing Test Fairness Using Multiple Regression Herman Aguinis Dr. Aguinis presented a method of assessing test fairness using multiple regression. He started by defining a moderator variable and explained the importance of the moderator variable. A moderator variable is when the relationship between X and Y depends on Z (the moderator). The relationship between X and Y can be causal or simply based on covariation. For example, in personnel selection, the relationship between abilities (X) and performance (Y) depends on motivation (Z, the moderator). Or, the relationship between a test and a criterion may depend upon gender or ethnicity. In other words, the X-Y relationship may differ depending on the value of Z (e.g., 1 = Female, 0 = Male). Understanding moderator effects can also be important for general management in showing how an organizational intervention is effective at similar levels for workers at different ages. Dr. Aguinis presented the formula for calculating the moderated multiple regression (MMR) model: | ||||||||||||||||||||||||||||
