What is inter-rater reliability?

• A. The stability of a measure's scores across time.
• B. The agreement between different observers or coders in their assessments.
• C. The accuracy of a measure in predicting outcomes.
• D. The consistency of a measure across time and items.

Answer: (B)

Inter-rater reliability is about how consistently different observers or coders rate the same phenomenon. When multiple evaluators independently assess the same behavior, artifact, or item, high agreement among them indicates that the measurement method yields stable, reproducible results rather than being driven by a single person's judgment. In practice, you assess this consistency with statistics like Cohen’s kappa for two raters on categorical judgments or the intraclass correlation coefficient for numeric ratings; with more raters, you’d use Fleiss’ kappa or a multi-rater ICC. These metrics account for agreement that could occur by chance and show whether the scoring system produces reliable ratings across different observers. The other options describe different concepts: stability across time is tested with test-retest reliability, predicting outcomes relates to predictive validity, and consistency across time and items points to internal consistency or general reliability rather than agreement among separate raters.