On the relationship between maximal reliability and maximal validity of linear composites

Abstract

A linear combination of a set of measures is often sought as an overall score summarizing subject performance. The weights in this composite can be selected to maximize its reliability or to maximize its validity, and the optimal choice of weights is in general not the same for these two optimality criteria. We explore several relationships between the resulting reliability and validity estimates in different situations. Only in the case of congeneric tests are maximal reliability and maximal validity attained with the same weights, and a precise relationship between these two maximality concepts can be derived. A widely and readily applicable procedure for point and interval estimation of maximal validity is also outlined. Several inequalities are established for the case when the measures are not congeneric.