Multi-trait Multi-method Assessment

 If you’re looking to move into Clinical Behavior Analysis, you need to have some understanding about how “psychology” talks about and tends to make judgments about the types of measurement they typically use. 

Measures in psychology generally are compared to each other to determine whether they have adequate “convergent”, “discriminant”, and “construct” validity. Convergent validity is determined by examining the correlation between measures that are expected to measure the same “thing” as the measure being evaluated. For example, if one is evaluating a measure of “generalized anxiety disorder” symptoms – you would expect the correlation between responses on this measure to correlate with the same sample’s responses on other measures of “generalized anxiety” symptoms. “Discriminant” validity is the degree to which the measure being evaluated correlates with measures with which we would not expect it to be highly correlated. For example, if one is developing a measure of “generalized anxiety” symptoms a very high correlation between your samples responses on this measure and say a measure of “psychosis” symptoms would suggest that your measure of generalized anxiety symptoms cannot “discriminate” between ‘psychosis’ and ‘generalized anxiety.’ Construct validity, oh such a dirty word for behavior analysts… is understood in “psychology” as a measure demonstrating both adequate convergence with expected measures and adequate divergence (discrimination) from measures that would not be expected to be highly related.

A common method of evaluating the construct validity of measures follows in a completely fictitious example below, this is the Multi-trait, Multi-method Assessment of Construct Validity. 

A psychologist at a mental health center develops a self-report inventory to assess generalized anxiety. Each new client at the mental health center is asked to complete this self-report inventory as well as a second questionnaire used to assess depression on two different occasions, separated by one month. In addition, two staff psychiatrists also conduct independent mental status exams with each new client and provide separate ratings of generalized anxiety and depression for each. Evaluate construct validity of the self-report inventory of generalized anxiety by interpreting all four sets of correlation coefficients within the multitrait-multimethod matrix that is obtained. Which, if any, of the four measures appears to possess adequate construct validity?

According to the multitrait-multimethod matrix only psychiatrist ratings of anxiety show adequate construct validity. When working through the multitrait-multimethod matrix it becomes apparent that the self-report measures of depression and psychiatrist ratings of depression show too low a correlation with each other and thus do not have adequate convergent validity. As the correlation is low, it would appear that these two methods of measurement do not measure “depression” in the same way or that factors related to measurement (rater biases, etc) may be interfering with measurement. Convergent validity between the self-report measures of anxiety and psychiatrist ratings of anxiety does appear to be sufficiently high though, indicating that the two methods of assessing “anxiety” appear to be measuring the same construct (or at least closely related constructs) through different methods.

When one examines the discriminant validity of these measures using the multitrait-multimethod matrix it appears that self-report measures of anxiety and depression are too highly correlated for adequate discriminant validity. Basically, these measures appear to be measuring constructs that are too closely related (probably because they are both influenced by a third ‘construct’ and are only topographically different). However, the psychiatrist ratings of anxiety and depression do appear to show adequate discriminant validity. As the correlation between these two is low, it would appear that they do not measure the same construct.

When examining the construct validity of the measures, only psychiatrist ratings of anxiety has both adequate convergent and discriminant validity. When the correlations for discriminant and convergent validity for this measure are compared it also appears that there is a substantial difference between them. This indicates that psychiatrist measures of anxiety tend to be reliably correlated with other measures of anxiety and not with measures of other related constructs. Thus, psychiatrist ratings of anxiety are considered to have good construct validity. None of the other measures meets all the requisite conditions for construct validity. However, despite this good evidence for the construct validity of psychiatrist ratings of anxiety we cannot be entirely sure of the construct validity of this measure; we are, however, much better justified in making such a leap.  

If you like learning about methods or topics that expand your ability to understand the field of “psychology” as it relates to behavior analysis – sign up for our blog or leave us a comment below.

For those interested, here another example worked out of a multi-trait, multi-method assessment matrix.