Answers to 20 Questions About Interrater Reliability and Interrater Agreement
LeBreton & Senter (2008) · Moratorium on unjustified nulls
LeBreton and Senter set out, question by question, how researchers should justify averaging individual ratings into a group score, and argue that the most common shortcut, a uniform null for rwg, will often overstate agreement. They separate interrater reliability, the relative consistency with which judges rank targets, from interrater agreement, the absolute consensus in their scores. Composition models that treat a team mean as a team property rest on agreement, so pure reliability indices are rarely used in multilevel work.
rwg, arguably the most popular agreement index, compares the observed variance in judges' ratings with the variance expected if they responded at random, and that baseline carries the sharpest argument. Most researchers assume a uniform null, which yields the largest error variance and so the highest agreement. Because rating biases are pervasive, they say this will often inflate agreement, and they call for 'a moratorium on the unconditional (i.e., unjustified) use of any null distribution, especially the uniform null distribution', supplying expected variances for skewed, triangular and normal nulls.
They say the .70 cut point 'artificially dichotomizes agreement', offering graded bands from lack of agreement (.00 to .30) to very strong (.91 to 1.00), heuristics they call arbitrary. The level required should rest on the quality of the measures, the seriousness of the consequences and the composition model. ICC(1) is read as an effect size for how far ratings depend on group membership, ICC(K) as the reliability of the group mean, where researchers will probably want .70 to .85. In a hypothetical case, ICC(1) of .12 with five raters per organisation gives ICC(K) of .41, too low for group effects to be likely to appear. Low agreement in some groups may itself be worth modelling as dispersion.
It is a tutorial with no new data; the SPSS example data were 'specifically contrived' to be clean, published cut-offs for average deviation assume a uniform null, and the indices assume parallel judges that organisational samples may not supply.
The paper does not mention psychological safety; the bridge is ours. Team psychological safety scores are commonly built by averaging members' ratings, and this paper sets out how that step should be justified. Kostopoulos and Bozionelos (2011) report mean rwg .86, ICC(1) .36 and ICC(2) .92 before using team means. Eldor, Hodor and Cappelli (2023) cite this paper, and Tom's article on their study points to 'inconsistent ICC' values in its unit-level analysis and questions the decision to aggregate despite them. Treating dispersion as a variable sits beside Tom's argument that a team is only as safe as its least safe person.
Save this paper to a bibliography, keep notes on it, and export citations. Create a free account →
Related psychsafety.com articles (4)
Related papers (5)
- Higgins et al (2022): When Is Psychological Safety Helpful in Organizations? A Longitudinal Study
- Stachowski, Kaplan & Waller (2009): The Benefits of Flexible Team Interaction During Crises
- Eldor, Hodor & Cappelli (2023): The Limits of Psychological Safety: Nonlinear Relationships with Performance
- Uitdewilligen & Waller (2018): Information Sharing and Decision-Making in Multidisciplinary Crisis Management Teams
- Kostopoulos & Bozionelos (2011): Team Exploratory and Exploitative Learning: Psychological Safety, Task Conflict, and Team Performance
