Understanding the Inter-Rater Consistency Report

Helping you interpret and use Inter-rater consistency insights effectively

Updated at July 24th, 2025

Table of Contents

Things to Know Before You Begin What the Report Shows Exporting Data

+ More

The Inter-Rater Consistency Report is a powerful tool designed to help administrative-level users analyze the consistency of subjective ratings across multiple evaluators. This report provides a visual overview of how well raters agree when scoring scenarios, evaluations, or courses.

Things to Know Before You Begin

This report is only available to administrative-level users.
It is designed to assess agreement among multiple raters who provide subjective evaluations.
You can export aggregated dataacross:
- Organizations
- Courses
- Scenarios
- Evaluations

For more detailed guidance, refer to the full Inter-Rater Consistency Report documentation.

What the Report Shows

Visual overview of rating consistency
Helps identify variability or bias in scoring
Supports quality assurance and training improvements for evaluators

Exporting Data

You can export the report data for further analysis or record-keeping. Export options typically include:

CSV or Excel formats
Filters by organization, course, scenario, or evaluation

inter-rater consistency report export data visual overview rating agreement multiple raters subjective ratings administrative users aggregated data organizations courses scenarios evaluations report access evaluator consistency rating variability scoring bias quality assurance evaluator training report export csv export excel export report filters data analysis report documentation