In the Skill area column of this report, n= shows how many evaluations the user completed that feed into their consistency score. An n of 2 is going to be much less reliable of an indicator than an n of 200.

The Inter-rater Consistency report can be found by clicking Reports in the global navigation bar. This report provides a visual to Users so that they can see the agreement between subjective ratings by multiple raters. 

Within this report, Users are able to select, within a date range of the Last 30 days, Last 90 days, Year to Date, and All Time, multiple organizations, courses, scenarios, or evaluations, and the report will aggregate the data which you can then export into an Excel spreadsheet by clicking Export Data > Inter-rater Consistency.

Note: The downloaded Excel sheet will show the Username formatted as Last Name, First Name, and Middle (if applicable).

IMPORTANT

• If only filtering by Organization, Course, or Scenario, we will include all Patient Evaluations, Monitor Evaluations, and Scoring Evaluations that fall into these filters.
• Participant Evaluations and Course Evaluations will not be included in this calculation as they are completed and apply to the same person.

Note: A message that reads Based on your filters, there are no evaluations or meaningful analysis will appear if your selections contain data that is not useful. 

Once the report is generated, the Excel document will include the following:

• Question categories included within the filter
• Standard Patient graders
• Z Score for each grader by Question Category
• Mean and Standard deviation of each Question Category
• The overall Z Score by Grader
• The mean score for each grader

Additionally, the export will be color-coded red or green to highlight a z-score as well as 1 or 2 standard deviations above or below the mean. The UI will show these colors as light and dark orange.

This information will be shown on two pages with the second page showing the filters that were applied during the export.

Page One

Page Two

Z-Scores

A Z-Score, which will be color coded in the export, tells you where the score lies on a normal distribution curve and describes a value's relationship to the mean of a group of values. These scores are important because they show a comparison between two scores that are not in the same normal distribution. 

Z-Score color codes and their meanings

The below color codes are used to bring attention to large gaps between the expected value and the actual value.

• Dark red - Will show a -2, which means the Z-score is 2 standard deviations below the mean.
• Light red - Will show a -1, which means the Z-score is 1 standard deviation below the mean.
• Dark green - Will show a 2, which means the Z-score is 2 standard deviations above the mean.
• Light green - Will show a 1, which means the Z-score is 1 standard deviation above the mean.

Z-Score calculations

There are two ways the Z-score is calculated when using this report:

• Evaluator and Question categories: For the intersection of the Evaluator and Question categories, the Z-score is calculated in this way: ((evaluator mean per QC) - (Mean of the QC)) / (Standard Deviation of the QC)
• Evaluator and Overall categories: For the intersection of the Evaluator and Overall categories, the Z-score is calculated in this way: ((evaluator mean across all their evals) - (Mean of ALL evals)) / (Standard Deviation of ALL evals)