Studies of measurement challenges

Here, I present four studies that focus on measurement of externalizing symptoms, dissociative identity disorder (DID), personality, and dementia. I don’t often address measurement, but it’s obviously critical to those who test individuals. Rivers, Winston-Lindeboom, Ruan-Iu, Atte, Tien & Diamond (2022) published “Validation of a Parent Report on Externalizing Symptoms Scale: A downward extension of the behavioral health screen” in School Psychology. Here’s an edited version of the impact statement and abstract.

To address the challenges of behavioral health school-based screening, this project tested new measures assessing parent-reported attention-deficit hyperactivity disorder and oppositional defiant/conduct symptoms for elementary age children. This study finds that the new scales show initial promise and work well for a sample of children referred to behavioral health services. The scales are part of an assessment system which may help close gaps in the receipt of services for students with these concerns. Externalizing problems are common in children ages 6–14, can have lifelong consequences, and may pose a particular risk when combined with other risk factors and symptoms (like depression and anxiety). Schools are uniquely positioned to assess and address these types of behavioral health concerns, but many school-based assessments do not focus on mental health distress (partially because they often lack the infrastructure for identification, screening, and referral). To address this gap, the Behavioral Health Works program student mental health software system has integrated teacher training, psychometrically strong assessments, feedback, and referral tools. However, this self-report tool for adolescents needed to be adapted for younger children. Thus, a parent-report version was added as well as new scales for better assessing this age group. The present study examines the psychometric properties of the new parent-report attention-deficit hyperactivity disorder (ADHD) and oppositional defiant/conduct scales within a sample of 440 children referred for school-based assessments. Overall, the new scales demonstrated good structural validity, measurement invariance across most demographic groups, discrimination in item response theory analyses, and evidence of convergent validity and good classification accuracy in relation to a validation battery. These externalizing scales are distinct and precise and show promise for improving the effectiveness of school-based programs for identifying at-risk children. 

Here’s more about Behavioral Health Works (BHW) bhwcares.com. We help individuals with autism and other developmental disabilities reach their potential by working collaboratively with families, schools, and relevant professionals. By focusing on ways in which people respond to their environment (behavior), we provide opportunities for them to learn in specific and observable ways, leading to a positive outcome (health). There resources for parents presented through a blog on the website. What I like about their work is the emphasis on early assessments of externalizing symptoms.

The second study addresses personality. Bieldorn et al. (2022) published “Personality Stability and Change: A meta-analysis of longitudinal studies” in Psychological Bulletin. Here’s an edited abstract:

Past research syntheses provided evidence that personality traits are both stable and changeable throughout the life span. However, early meta-analytic estimates were constrained by a relatively small universe of longitudinal studies, many of which tracked personality traits in small samples over moderate time periods using measures that were only loosely related to contemporary trait models such as the Big Five. Since then, hundreds of new studies have emerged allowing for more precise estimates of personality trait stability and change across the life span. Here, we updated and extended previous research syntheses on personality trait development by synthesizing novel longitudinal data on rank-order stability (total k = 189, total N = 178,503) and mean-level change (total k = 276, N = 242,542) from studies published after January 1, 2005. Consistent with earlier meta-analytic findings, the rank-order stability of personality traits increased significantly throughout early life before reaching a plateau in young adulthood. These increases in stability coincide with mean-level changes in the direction of greater maturity. In contrast to previous findings, we found little evidence for increasing rank-order stabilities after Age 25. Moreover, cumulative mean-level trait changes across the life span were slightly smaller than previously estimated. Emotional stability, however, increased consistently and more substantially across the life span than previously found. Moderator analyses indicated that narrow facet-level and maladaptive trait measures were less stable than broader domain and adaptive trait measures. Overall, the present findings draw a more precise picture of the life span development of personality traits and highlight important gaps in the personality development literature. 

I have always questioned attempts to measure personality, in general, and the Big Five in particular. What I like about this research is it tells us, not at all surprisingly, that personality is far less stable in children and adolescents than in young adults, emotional stability increases over the lifespan, and broader domain and adaptive trait measures of personality are more stable than facet-level and maladaptive trait measures.

The third study looks at dissociative identity disorder (DID). Barth, Brand & Nester (2023) published “Distinguishing Clinical and Simulated Dissociative Identity Disorder using the Miller Forensic Assessment of Symptoms Test” in Psychological Trauma: Theory, Research, Practice, and Policy. Here’s the edited abstract and impact statement:

Individuals with dissociative identity disorder (DID) experience severe and broad-ranging symptoms which can be associated with elevations on measures designed to detect feigning and/or malingering. Research is needed to determine how to distinguish genuine DID from simulated DID on assessment measures and validity scales. This study examined whether the Miller Forensic Assessment of Symptoms Test (M-FAST), a screening measure of malingering, could differentiate between individuals with DID and DID simulators. Thirty-five individuals with clinical, validated DID were compared to 88 individuals attempting to simulate DID on the M-FAST. A MANCOVA compared the two groups on total M-FAST score and subscales. While the M-FAST performed well at classifying malingering among those attempting to simulate dissociative identity disorder (DID), it correctly classified approximately only 83% of individuals diagnosed with DID as not malingering, indicating that a cut-off score of 6 on the M-FAST performs poorly with this population. However, by increasing the cut-off score to 7, the M-FAST improved substantially such that most individuals with DID were classified correctly, while still maintaining high sensitivity. These results suggest that a cut-off score of 7 may be indicated for individuals with DID. 

I like this study because it provides very specific recommendations for differentiating clinical from simulated DID, which is often very important in forensic cases.

Finally, Gaudet, Castelluccio, Gold, McLaughlin & Correia (2022) published “Limitations of Performance Validity Tests in Dementia Evaluations: The role of base rates” in Psychological Assessment. 

Performance validity tests (PVTs) are frequently used to detect invalid performance on cognitive testing. The inclusion of PVTs in cognitive test batteries is commonplace irrespective of the condition of interest. However, base rates of invalid performance vary across clinical populations. Research accounting for base rates of invalid performance in varying clinical populations and PVT classification accuracy rates are not commonly synthesized. To address this gap, the present study examined the clinical utility of select PVTs used with older adults presenting for dementia evaluations. We computed posterior probabilities of invalid performance for the select PVTs using an estimated 5% base rate of invalid performance based on prior published studies. Posterior probabilities of invalid performance based on a PVT failure (i.e., invalid performance identified as invalid) ranged from 7.3% to 60.3% across PVTs; posterior probabilities of a false positive (i.e., valid performance identified as invalid) ranged from 39.7% to 92.7%. Conversely, posterior probabilities of a true negative (i.e., valid performance identified as valid) ranged from 95.7% to 99.3%; posterior probabilities of a false negative (i.e., invalid performance identified as valid) ranged from 0.7% to 4.3%. Results call into question the utility of PVTs in dementia evaluations. Consequently, the use of PVTs in dementia evaluations is likely to erroneously identify valid test data as invalid (i.e., false-positive error) at a frequency that exceeds the estimated 5% base rate of invalid performance. Further research examining correlates of invalid performance among older adults will clarify base rate estimates and potentially enhance the utility of PVTs. This study found that older adults presenting for dementia evaluations are at an increased risk for having their cognitive test data inaccurately deemed invalid when performance validity tests are administered. Moreover, documentation of invalid cognitive test performance, irrespective of it being a true positive or false positive, risks biasing providers’ clinical diagnostic and care decisions in a way that negatively impacts the patient. 

These data are critical for any clinician who uses performance validity tests in dementia evaluations.