Measurement in crisis? A critical evaluation of measurement validation practices in the psychological sciences

Research relies on valid measurement. However, serious concerns have been raised about validation practices in the psychological sciences. In this thesis, I take three steps toward improved psychological measurement. First, I critically evaluate the validity evidence for one of the most widely used measures of adult social cognitive ability, the Reading the Mind in the Eyes Test (Eyes Test). Second, using the Eyes Test to demonstrate the serious negative consequences that inadequate validation practices can have, I advocate for measurement-focused research reforms. Third, I propose practical strategies to improve the validation of psychological measurements. In Chapter 1, I provide background information on the history of the concept of validity and the current state of validation practices in the psychological sciences. In Chapter 2, I report a systematic scoping review of 1,461 empirical studies which demonstrates that studies frequently report little or no validity evidence for Eyes Test scores, and where reported, the evidence frequently indicates low levels of validity. In Chapter 3, I report findings from confirmatory and exploratory factor analyses in nine pre-existing non-clinical samples (n = 558 to 9,267) which demonstrate that Eyes Test scores have weak structural properties. In Chapter 4, I argue that the Eyes Test is unlikely to produce valid measurements of social cognitive ability. Moreover, to illustrate how inadequately validated measurements can undermine scientific research findings, I present two peer-reviewed critiques of recently published studies that reported findings based on Eyes Test scores. In Chapter 5, I argue for measurement-focused reforms. To support this argument, I present a commentary article in which I argue that unaddressed measurement issues undermine a proposed framework intended to improve the commensurability of research findings across studies. In Chapter 6, I present a commentary on an article that proposes a proxy failure framework to explain a phenomenon in which proxy measures appear to be inherently prone to exploitation and failure. In my commentary, I argue that the proxy failure framework can be fruitfully applied to the case of psychological measurement. In Chapter 7, I turn to the concept of validity, arguing that the broad definition endorsed by the American Psychological Association offers insufficient practical guidance for how to validate psychological measurements. As an alternative, I build on a narrower concept of validity focused on causal relationships that has been proposed in the literature, and I introduce “validity potential” as a concept focused on the development of theoretically supported measurement procedures. Finally, in Chapter 8, I summarise and expand on the aims of the thesis and suggest additional practical strategies to improve validation practices.<p></p>