Intraclass correlation – A discussion and demonstration of basic features

A re-analysis of intraclass correlation (ICC) theory is presented together with Monte Carlo simulations of ICC probability distributions. A partly revised and simplified theory of the single-score ICC is obtained, together with an alternative and simple recipe for its use in reliability studies. Our main, practical conclusion is that in the analysis of a reliability study it is neither necessary nor convenient to start from an initial choice of a specified statistical model. Rather, one may impartially use all three single-score ICC formulas. A near equality of the three ICC values indicates the absence of bias (systematic error), in which case the classical (one-way random) ICC may be used. A consistency ICC larger than absolute agreement ICC indicates the presence of non-negligible bias; if so, classical ICC is invalid and misleading. An F-test may be used to confirm whether biases are present. From the resulting model (without or with bias) variances and confidence intervals may then be calculated. In presence of bias, both absolute agreement ICC and consistency ICC should be reported, since they give different and complementary information about the reliability of the method. A clinical example with data from the literature is given.


Appendix 2 Derivation of the EMS (Expected Mean Square) relations
For this derivation we use a matrix with n = 3 rows (subjects) and k = 2 columns (measurements) as an illustration: As in Table 2 (main text), S 1 , S 2 , and S 3 are the mean values of the rows, M 1 and M 2 are the mean values of the columns and is the total mean value.
We will first assume Model 2, i.e. that each matrix element x ij may be regarded as the sum of four terms, where µ is a constant, r i is sampled from a normal distribution with standard deviation  r , c j is sampled from a normal distribution with standard deviation  c and v ij is sampled from a normal distribution with standard deviation  v . Note that Model 1 is obtained simply by putting all c j = 0.
Assuming that the model (A2-1) is used to generate the matrix x ij shown above, we will now estimate the resulting mean squares, i.e. MSBS, MSBM, MSWS, MSWM and MSE.
We observe that each matrix element x ij is, apart from the constant µ, the sum of three terms, each of which is sampled from a normal distribution. The total variance of x ij is therefore the sum of the three independent variances  r 2 ,  c 2 and  v 2 . It follows that each term in (A2-1) will give its own, independent contribution to each of the above five mean squares, for example to MSBS. In order to estimate these contributions, the simplest procedure is to study one term in (A2-1) at a time, assuming the others to be zero.
We may expect the constant µ to give zero contribution to variance and thus zero contribution to each mean square quantity (MS). This is easily verified. Putting each term in eq.(A2-1) except µ equal to zero, the matrix and its averages in will be reduced to the following: We use the formulas in Appendix 1, and get, for example, In a similar way we may easily confirm that MSBM = MSWS = MSWM = MSE = 0.
Here, is the mean value of the three r i . Again using Appendix 1, we get where the "" sign means "is an estimate of". The last member in both equations follows when we realize that We next put all terms in (A2-1) except the c j equal to zero. The resulting matrix is Here, is the mean value of the three c j . We need not do all the above calculations again, but merely observe that rows and columns have changed roles. Thus, we will obtain the desired formulas simply by replacing k by n,  r by where, again, the last member follows since we recognize the square of the standard deviation of the c j about their mean value in the next last member. The result, i.e. the contributions from the c j term, is therefore Here we may insert the derivation in the case that the c j terms are fixed, i.e. Model 3. The matrix looks precisely the same. However, in eq.(A2-6) we do not make the estimate (   c 2 ) in the last member, but simply use the fact that We now consider the last term in (A2-1), i.e. we put all terms equal to zero except v ij .The resulting matrix is given by c 1 c 2 c 1 c 2 c 1 c 2 c 1 c 2