Validation of the Orpheus Minor Scales in a working population

By John Rust

Orpheus is a broad spectrum 190 item work-based combined personality and integrity questionnaire that generates scores on sixteen scales - seven minor scales, five major scales, and four audit scales. The minor scales of Orpheus are Proficiency, Work-orientation, Patience, Fair-mindedness, Loyalty, Disclosure and Initiative, and are designed to assess the integrity traits of carelessness, poor work orientation, anger, resentfulness, disloyalty, lying and inertia, within a framework traced back to Prudentius in the Fourth Century AD. The major scales are Fellowship, Authority, Conformity, Emotion and Detail and are based on the `big five' model of personality. The study reports a validation of the Orpheus Minor Scales against 7 adjective check lists on a sample of 380 respondents in employment settings in the United Kingdom. General issues relating personality and integrity testing are also discussed.

Orpheus is a 190 item questionnaire which assesses 12 traits comprising 7 minor scales and 5 major scales (Rust, 1996). The minor scales of Orpheus are Proficiency, Work-orientation, Patience, Fair-mindedness, Loyalty, Disclosure and Initiative, and are designed to assess the integrity traits of carelessness, poor work orientation, anger, resentfulness, disloyalty, lying and inertia. The five Orpheus major scales are based on the 'big five' model of personality, interpreted within the context of work-related behaviors, beliefs, attitudes and interests. The scales (Fellowship, Authority, Conformity, Emotion and Detail) represent social, organizational, intellectual, emotional and perceptual aspects of personality respectively, and are based on the 'big five' traits of extraversion, toughmindedness, openness-to-experience, neuroticism and conscientiousness.

Sackett et al (1989) described personality scales used in the assessment of integrity as "disguised purpose tests" to distinguish them from overt tests of integrity, which he called "clear purpose tests". Integrity testing is reviewed in the Tenth Mental Measurement Yearbook (Conoley & Kramer, 1989). In addition, the American Psychological Association (Goldberg et al., 1991) and the US Office of Technological Assessment (1990) have both examined the use of integrity tests, and these reports are summarized by Camara and Schneider (1994). Personality based integrity tests are generally similar in form to any other occupationally based personality test. However, it has been argued that integrity tests are as good or better predictors of overall job performance as classical personality tests (Ones et al., 1993; Schmidt et al., 1992; Collins dc Schmidt, 1993; Sackett & Wanek, 1996).

O'Bannon et al. (1989), Sackett et al. (1989), Ones et al. (1995) and Sackett & Wanek (1996) list some of the constructs and target behaviors used in integrity tests. These include responsibility, long-term job commitment, consistency, proneness to violence, moral reasoning, hostility, work ethic, dependability, depression, energy level, disciplinary problems, violence on the job, excessive absenteeism and tardiness, and theft. Rust (1997) summarizes these target behaviors within a broad spectrum formulation of integrity based on a seven trait model derived from the psychological theory of the classical scholar Prudentius, who published his "Psychomachia" in the Fourth Century AD. To obtain the conceptual framework for the seven minor scales of Orpheus, a survey of the integrity testing literature had been carried out and the results classified and categorized in terms of the Prudentius model of personality. Integrity traits reflect behaviors and attitudes that are generally held to be undesirable, however these were reversed for category definition purposes. The resultant traits were Proficiency, Work-orientation, Patience, Fair-mindedness, Loyalty, Generosity and Initiative. However, during the development of the scales it became clear that Generosity was too highly correlated with social desirability to be assessed independently by this technique and this scale was replaced by Disclosure, which is an inverse social desirability or lie scale.

In the present study Orpheus was administered to 380 participants in a variety of employment settings in the UK, together with a 94 item adjective check list designed to assess the same seven traits with a view to obtaining data on the validity of the Orpheus Minor Scales.

METHOD

Orpheus was administered to 380 respondents in a variety of occupations within 20 organizations in the UK. The occupations sampled included accountants, drivers, engineers and scientists, human resource personnel, insurance underwriters and claims negotiators, managers, police officers, sales and marketing staff, secretaries and clerks, security staff, teachers and trainers. The mean age of the sample was 30.67 years with a standard deviation of 11.01.

The split-half reliabilities for the 5 major scales and the 7 minor scales, together with the scale intercorrelations from the 423 respondents in the standardization sample (Rust, 1997), are given in Tables 1 and 2. Intercorrelations between major and minor scales are not given as this would involve duplication of items (29 of the items appear in both major and minor scales).

A 94 item adjective check list was constructed. Twelve pilot items were included for each of the 7 Prudentius traits. This was administered to 198 staff employed in a major UK security company. Each set of 12 adjectives was factor analysed to yield a short 8 item interim scale for each trait, with equal numbers of positive and negative items. The order of the checklist items was randomised before presentation. Response categories were on a five point scale from `not at all like me' to 'very much like me'. The checklists were:

Checklist 1 (Carelessness): 8 adjectives, the highest factor loadings being for Absentminded, Forgetful, Careless and Thoughtless and the lowest for Wise and Serious.

Checklist 2 (Absenteeism): 8 adjectives, the highest factor loadings being for Irresponsible and Unkempt, and the lowest for Determined and Tireless.

Checklist 3 (Hostility): 8 adjectives, the highest factor loadings being for Stormy, Wild, Aggressive and Angry, and the lowest loadings being for Patient, Accepting and Non-violent.

Checklist 4 (Supervision): 8 adjectives, highest factor loadings being for Harddone-by, Self-righteous and Scapegoated, and lowest factor loading being for Just, Ethical and Impartial.

Checklist 5 (Disloyalty): 10 adjectives, the highest factor loadings were for Opinionated, Egotistical and Arrogant, and the lowest for Dutiful, Unassuming and Obedient.

Checklist 6 (Greed): 8 adjectives, the highest factor loadings were for Unappreciative, Resentful and Envious, and the lowest for Benevolent, Altruistic and Compassionate.

Checklist 7 (Inertia): 12 adjectives, the highest factor loadings being for Disheartened, Discouraged and Down-hearted, and the lowest for Inventive, Creative and Ingenious.

RESULTS

The correlations between the seven checklist scales and the Orpheus scales are shown in Table 3.

In the correlation matrix, non-significant correlations are notable for their relative paucity. However, some of this surfeit of significance can be attributed to response bias artefacts. The acquiescence effect will not be among these as this has been eliminated through within-subject standardisation for both Orpheus and the adjective check list, and by balance of positive with negative items for every scale. However, Prudentius adjectives are more likely to be affected by social desirability. To counterbalance for this influence, the correlation matrix was recalculated as a partial correlation matrix which removed the influence of P6: Disclosure, a robust measure of socially desirable responding. This is shown in Table 4.

It can be seen in Table 4 that partialling out of the social desirability effect eliminates some of the smaller correlations, but leaves most unchanged. The pattern of intercorrelations found for the `big five' conform to expectation. They generally confirm the interpretations given in the literature for `big five' profile scores.

The data were further analysed to examine the discriminant validity of the Orpheus minor scales, particularly with respect to the `big five' scores from the Major Scales. One of the claims for the `big five' model is that it is a complete description of personality. If this was the case then we would not expect the minor scales to add anything to the total picture over and above the information already incumbent in the `big five'. We can evaluate this claim by partialling out the effect of all five major scales from the correlation matrix between Orpheus and the Prudentius checklist scales. This partial correlation matrix is shown in Table 5.

DISCUSSION

While the ideas of Galen of Pergamon are generally known to personality theorists, there is rather less familiarity with those of his contemporary, Aurelius Prudentius of Caesar Augusta (today Saragossa in modern Spain). Prudentius argued that a continuous battle takes place throughout life between man's animal nature and his ability to reason. In the Psychomachia various aspects of the human character are personified by warriors in battle. The battle epitomizes a process of evolution from the chaotic processes of childhood towards a unity in old age in which reason should reign supreme. However this evolution rarely proceeds smoothly. The developing human must make choices on this journey through life, and can become fixated by these choices which develop into habitual modes of responding. If poor choices are made then we must live with the consequences and will not become fully rational. Possible choices on this journey are epitomized by two armies, the vices and the virtues. In face, the Psychomachia anticipates many of the ideas of contemporary psychotherapy, including Freud's "libido" and Roger's "self-actualizing tendency" as well as cognitive behavior therapy, by almost two millenia. Unlike the biological traits derived from Galen, Prudentius' traits include an element of choice on the part of the subject. Undesirable personality characteristics develop through habits which may be difficult to shift, but can be changed with effort and thus are ultimately our own responsibility. Because the model recognizes that humans operate within an ethical environment it is far more able than that of Galen to take account of everyday concerns.

The Orpheus minor scales had been designed to assess the strengths and weaknesses of the human character. In contrast to the `big five' model, there is a recognition that most assessments of personality in everyday life are made in terms of the consequences of particular characteristics or actions for others. Because a character weakness in one occupation may be neutral, or even a strength, in another, the minor scales are to be recommended for use only where relevant to a particular work setting. For example, entrepreneurs will often take risks in order to learn from their mistakes, whereas this approach would not be desirable for airline pilots. Thus, Proficiency assesses the degree of care that is likely to be taken in carrying out a task and is of relevance to occupations in which mistakes can have particularly severe consequences. Work-orientation assesses work ethic and can be important where absenteeism may present a problem or where staff are required to work long hours in stressful conditions. Patience is the ability to control aggression in whatever form. It is of relevance to work environments where bullying has been a particular concern. Fair-mindedness assesses fairness in judging the actions of others and is important in work environments that are beset with strife. Loyalty assesses the sense of obedience to company policy, and is useful in work situations that necessitate independent action by staff on the organization's behalf. Initiative assesses a sense of purpose and a forward-looking approach. It is of relevance to organizational settings about to undergo major change.

The results of the present study provide encouraging evidence for the validity of the Orpheus Minor scales. As well as the sound correlations found in Table 3, we can see in Table 4 that the data is relatively free from social desirability artifacts. Table 5 shows us that the Orpheus Minor scales have discriminant validity over and above any effects which can be accounted for by the `big five' traits assessed by the Orpheus Major Scales. The results have not been corrected for attenuation. However, had they been so then the correlations would have been somewhat higher.

REFERENCES

Camara, W.J. & Schneider, D.L. (1994). Integrity tests: Facts and unresolved issues. American Psychologist, 49, 112-119

Collins, J.M. & Schmidt, EL. (1993). Integrity, personality and white collar crime: A construct validity study. Personnel Psychology, 46, 295-311.

Conoley, J.J. & Kramer, J.C. (1989). (Eds) Tenth Mental Measurement Yearbook. Buros Institute of Mental Measurement, University of Nebraska Press; Lincoln NE.

Goldberg, L.R., Grenier, J.R., Guion, R.M., Sechrest, L.B. & Wing, H. (1991). Questionnaires used in the prediction of trustworthiness in pre-employment selection decisions. Washington DC; American Psychological Association.

O'Bannon, R.M., Goldinger, L.A. & Appleby, G.S. (1989). Honest and integrity testing. Atlanta, GA: Applied Information Resources.

Ones, D.S., Viswesvaran, C. & Schmidt, EL. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance [Monograph]. Journal of Applied Psychology, 78, 679-703.

Rust, J. (1996). Orpheus Handbook. The Psychological Corporation, London and San Antonio.

Rust, L. (1997). Giotto Handbook. The Psychological Corporation, London and San Antonio.

Sackett, P.R., Burris, L.R. dc Callahan, C. (1989). Integrity testing for personnel selection: An update. Personnel Psychology, 37, 491-529.

Sackett, RR. & Wanek, J.E. (1996). New developments in the use of measures of honesty, integrity, conscientiousness, dependability, trustworthiness and reliability for personnel selection. Personnel Psychology, 49(4), 787-830.

Schmidt, EL., Ones, D.S. & Hunter, J.E. (1992). Personnel selection. Annual Review of Psychology, 43, 627-670.

US Congress, Office of Technology Assessment (1990). The use of integrity tests for pre-employment screening (OTA-SET 442) US Government Printing Office; Washington DC.

JOHN RUST
Goldsmiths College, University of London, England