Validation

The Institute for Motivational Living worked with Larry Price Ph.D., Professor of Psychometrics & Statistics at Texas State University, to complete a cutting edge DISC validation and reliability study based on workplace behaviors, and Dr. William Marston's original work.

Dr. Price states, "Collectively, the results of the present study provide evidence that the IML DISC provides a robust measure of personality style. The reliability and validity evidence indicates that the IML may confidently use the DISC in their effort to better understand the personality style of individuals in the workplace environment."

Use the links below to find out more information on the testing and validation of PeopleKeys DISC assessments:

 

Establishing evidence for the validity of a particular psychological or educational instrument, scale or test is likely to be the most important aspect of instrument development. According to the American Psychological Association, the American Educational Research Association, and the National Council on Measurement in Education (1999), “validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests”, (p. 9). Moreover, in psychological measurement and assessment, the term “validity” represents a judgment or expression of how well an instrument or test measures what it purports to measure. It is a judgment based on evidence about the appropriateness of inferences draw from test scores. Therefore, validation, involves the process of developing a sound, scientifically-based argument that will support the intended use of the scores or classifications derived from using the instrument. The process of test or instrument validation involves conducting investigations and accumulating evidence to provide a sound framework for the proposed score or classification interpretations. The score interpretations are theoretically linked to the theoretical underpinnings or construct that the instrument purports to be measuring. In the present study, the theoretical framework that the IML DISC instrument incorporates was first proposed by William Marston in his text “Emotions of Normal People” published in 1928. Within the context of the present study, the underlying factor structure of the DISC is theoretically represented by four factors or latent constructs first proposed by William Marston (1928). These four factors (dimensions) are: 1) Dominance, 2) Influence, 3) Steadiness (Submission originally), and 4) Compliance. Therefore, the latent factor structure underlying the DISC instrument is multi-dimensional and consists of certain items within the total twenty-four items that reflect a particular piece of the multi-dimensional structure of the DISC theory. Finally, the validity of an intended interpretation of test scores or classification outcomes relies collectively on adequate score reliability, appropriate test administration, accurate scoring and scaling, and cultural fairness to examinees. The Institute for Motivational Living.

2015 IML DISC Instrument Validation Study Overview: DISC Profiles

Establishing evidence for the validity of a particular psychological or educational instrument, scale or test is likely to be the most important aspect of instrument development. Moreover, in psychological measurement and assessment, the term “validity” represents a judgment or expression of how well an instrument or test measures what it purports to measure. This validation study investigated the psychometric properties of the Institute for Motivational Living's (IML) DISC instrument within the context of the workplace environment. A purposive sampling strategy was employed with respondents obtained from the social service and counseling field, and ranging in job position from entry level administrative to National Executive Director. The final sample obtained was cross-sectional representing seven different regions across the United States in 26 separate locations (Northeast, Midwest, South, and Southwest). The ethnic composition of the sample consisted of 76% Caucasian, 12% African-American, 11% Hispanic, and 1% Other (Asian/Pacific Rim). Eighty-seven percent (87%) of the subjects were female, 13% male with an age range of between 21 — 85 years. All of the subjects had attained a minimum of 12 years of formal education with most having 16 years. Three hundred thirty one (331) survey instruments were completed and returned for analysis.

Reliability Evidence. The internal consistency reliability of the scores obtained on the DISC was calculated using coefficient alpha (a) for each of the four subscales (Dominance, Influence, Steadiness, Compliance) and overall for the total scale. Results of the reliability analyses were observed to be: 1) Dominance a = .84, 2) Influence a = .70, Steadiness a = .73, Compliance a = .84, and Overall a = .87. The percent agreement between the examinee's responses within a "primary" style or a close secondary "primary" style according to DISC theoretical expectations was high (91%). In order to further examine the proportion/percent of agreement between DISC theory and the observed examinee responses, the McNemar nonparametric test for two related (matched) dichotomous outcome variables was used. McNemar's test is particularly useful in detecting changes in patterns of examinee responses in "before-and-after" or "related subjects" designs. The results of the McNemar were observed to be statistically significant at the p < .05 level of significance. Interpretation of this finding reveals that the percent agreement was not only "good," but may also be stated to be "statistically significantly." Finally, the Kappa coefficient, an agreement statistic that statistically adjusts for capitalization due to chance was also calculated. In this study, coefficient Kappa was observed as r = .40. This level of congruence between the theory and actual observed responses is classified as "good" agreement beyond the level of chance (Landis & Koch, 1977a).

Validity Evidence. The underlying factor structure of the DISC is theoretically represented by four factors or latent constructs first proposed by William Marston (1928). These four factors (dimensions) are: 1) Dominance, 2) Influence, 3) Steadiness, and 4) Compliance. Therefore, the latent factor structure underlying the DISC instrument is multidimensional and consists of certain items within the total twenty-four items representing a particular piece of the multidimensional structure of the DISC theory. Content validity evidence refers specifically to themes, wording with current cultural issues in mind, item formats, tasks, and guidelines for administration and scoring. In the present study, evidence for appropriate content validity was established by qualified experts at the Institute for Motivational Living by compiling a series of statements and traits that are universally accepted as the basis for the four DISC styles. To extend the content validation process, a group of 12 certified behavior analysts were asked to rate themselves using this survey instrument as well as have their colleague review their choices to ensure congruence between how the analysts rated themselves using the DISC related to how others observed their behavior. Finally, since this validation study was conducted within the context of a workplace setting or environment, the Bair Foundation National Office began the study by taking 25 people from the National Office and had them take the 24 item DISC. This step served as baseline or pilot information for the study as well as provided an orientation to the instrument to the Bair Foundation who provided access to the overall sample. As previously discussed, the underlying structure of the DISC is multidimensional in nature and is composed of four subscales each representing a separate but correlated dimension of personality style. Therefore, an important goal of the present study was to rigorously investigate the factor structure of the IML DISC instrument in relation to Martsonís theoretical framework using two specific types of factor analytic approaches: 1) the Q — Technique, and 2) the R — Technique within a structural equation modeling framework using modified ipsative scores.

The Q — Technique factor analytic method is particularly appropriate for the study of classifying individuals based on preferences or systems of typology — such as in personality assessment. Specifically, the issue of interest in the Q — Technique is the notion of ìintra-individual differencesî (within subjects) rather than ìinter-individualî (between subjects) differences. The inherent nature of the forced-choice measurement approach make traditional factor analytic techniques (R — technique) often inappropriate due to the correlated nature of the item formats and also because of the distributional shapes of the data obtained from respondents. The Q — Technique factor analyses proceeded by randomly selecting 10 separate samples ( n = 20) from the total sample. Next, 10 separate factor analyses were conducted using both Principle Components Analysis (PCA) with Varimax rotation and then using Principle Axis Factor Analysis (PAF) with Promax Oblique rotation. In all 10 analyses, the results provided evidence of a four factor dimensional structure underlying the IML DISC instrument. Across all analyses, an average of 70% of the total variance was explained by the four factor DISC model. As a general rule, the guideline for the percentage of variance accounted for in a factor analytic model to be evaluated as ìacceptableî in the psychometric literature is 70%.

The inherent nature of the forced-choice measurement approach make traditional factor analytic techniques (R — technique) inappropriate due to the ìcorrelatedî nature of the item formats and also because of mathematical scaling problems. To account for this artifact, a modified scaling approach to the items was employed by the author thereby allowing for the exploratory (preliminary) investigation of the theoretical factor structure using structural equation modeling. Several fit indices were used as indicators for the goodness-of-fit of the DISC structural equation model. For the four-factor DISC model evaluated herein, the following fit statistics were observed: c2 (244, n=200)=272.65, p > .05, GFI = .90, AIC = 348.67, RMR = .08, RMSEA = .02.

Review: Results presented herein indicate that the psychometric properties of the IML DISC are internally consistent and reliable. Classical true score model estimates of internal consistency reliability were found to be consistent with previous reliability estimates for the IML DISC instrument. Furthermore, the development process that the IML organization adhered to for the item selection and refinement are closely aligned with the Standards for Educational and Psychological Testing (1999) extending support for the collective validation process. Collectively, the results of the present study provide evidence that the IML DISC provides a robust measure of personality style. The reliability and validity evidence indicates that the IML may confidently use the DISC in their effort to better understand the personality style of individuals in the workplace environment.

Larry Price, Ph.D.