Fleiss's kappa

Fleiss's kappa is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items or classifying items. This contrasts with other kappas such as Cohen's kappa, which only work when assessing the agreement between two raters or the intra-rater reliability (for one appraiser versus themself). The measure calculates the degree of agreement in classification beyond what would be expected by chance. It is named after American biostatistician Joseph L. Fleiss, who introduced it in a 1971 article.

Fleiss's kappa can be used with binary or nominal-scale. It can also be applied to ordinal data (ranked data): the MiniTab online documentation gives an example. However, this document notes: "When you have ordinal ratings, such as defect severity ratings on a scale of 1–5, Kendall's coefficients, which account for ordering, are usually more appropriate statistics to determine association than kappa alone." Keep in mind that Kendall's rank coefficients are only appropriate for rank data.