intercoder reliability in qualitative research

Krippendorff (2004) suggests that Cohen’s Kappa is not qualified as a reliability measure in reliability analysis. On top of the aforementioned well-known paradoxes, Zhao (2011) describes twelve other paradoxes with kappa and suggests that Cohen’s kappa is not a general measure for inter-rater reliability but a measure of reliability that only holds under particular conditions, which are rarely met. While the two paradoxes are not mentioned in older textbooks (e.g., Agresti, 2002), they are fully introduced as the limitations of kappa in a recent graduate textbook (Oleckno, 2008). (2) Unbalanced marginal distributions produce higher values of kappa than more balanced marginal distributions. (1) A low kappa can occur at a high agreement In a series of two papers, Feinstein & Cicchetti (1990) and Cicchetti & Feinstein (1990) made the following two paradoxes with Cohen’s kappa well-known: Researchers started to raise issues with Cohen’s kappa more than three decades ago (Kraemer, 1979 Brennan & Prediger, 1981 Maclure & Willett, 1987 Zwick, 1988 Feinstein & Cicchetti, 1990 Cicchetti & Feinstein, 1990 Byrt, Bishop & Carlin, 1993). “It is quite puzzling why Cohen’s kappa has been so popular despite so much controversy with it. You will find that many researchers use Cohen’s kappa and that this measure is also recommended in many popular books even though there is plenty of literature pointing out the limitations of this measuring instrument Xie (2013) for instance, explains: You probably turn to the literature and search what other researchers have used or look at the textbooks in your field. If you decide that measuring inter-coder agreement is essential for your research, the next question is what measure you want to use. Selection of a suitable measuring instrument

Intercoder reliability in qualitative research