Comparative studies of social security systems have increasingly turned towards the use of replacement rates as measures of the level of benefits in different countries and therefore of the degree of social protection afforded by different welfare systems. The rationale for this is that replacement rates provide consistent measures of the relative generosity of payments and therefore indicate the ‘quality’ of social security systems. This paper reviews the use of replacement rates in comparisons of the generosity of retirement pensions and argues that they are not necessarily reliable as such measures. This reflects a number of factors, including incomplete measurement of benefit packages and differences in what must be bought out of disposable incomes. Most importantly, the paper suggests that the levels of earnings in different countries are not independent of the processes of redistribution. In particular, countries which rely on social security contributions from employers appear to provide more generous benefits than those which rely on income taxes or employee contributions. This is a consequence of the fact that employer contributions do not figure specifically in the calculation of replacement rates. The relative generosity of benefit systems is overstated in countries which rely on employer social security contributions to fund benefits. The paper concludes that a range of complementary indicators of social security systems should be used in future analysis of these issues.