A comprehensive guide to study the agreement and reliability of multi-observer ordinal data
Best practices
Abstract
A recent systematic review revealed issues in regard to performing and reporting agreement and reliability studies for ordinal scales, especially in the presence of more than two observers. This paper therefore aims to provide all necessary information in regard to the choice among the most meaningful and most used measures and the planning of agreement and reliability studies for ordinal outcomes. It considers the generalisation of the proportion of (dis)agreement, the mean absolute deviation, the mean squared deviation and weighted kappa coefficients to more than two observers in the presence of an ordinal outcome. It provides an interpretation of these measures, a way to construct confidence intervals and a method to make sample size calculations.
Links
Published paper
Citation
@article{vanbelle2024,
author = {Vanbelle, Sophie and Hernandez Englhart, Christina and Blix,
Ellen},
title = {A Comprehensive Guide to Study the Agreement and Reliability
of Multi-Observer Ordinal Data},
journal = {Statistics},
volume = {24},
number = {no},
pages = {310},
date = {2024},
url = {https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-024-02431-y},
doi = {10.1186/s12874-024-02431-y},
langid = {en}
}