Question analysis report

The Question Analysis report will show each evaluation and an analysis of how all participants who have answered performed on each individual question. The Question Analysis will identify how many users received No Credit, Partial Credit, or Full Credit. The Mean score will be presented for each question. Easy identification of questions that participants need remediation with is readily accessible by sorting each column.

Note: This report can be downloaded as an Excel or PDF report.

1. Evaluation – Choose the desired evaluation. If a scenario filter has been selected, only the selected scenario evaluations will appear in the dropdown.
2. Question Instruction and Question – The questions for the selected Evaluation will appear in this field.
3. Question Statistics – Number of no credit answers, partial credit answers, full credit answers, mean score and the point biserial for each question appear. Note: A graphic representation to the left of the statistics quickly identifies which questions need review.
4. Scenario Filters – Allow for the selection of specific scenarios to appear in the Evaluation selection. If no scenario is selected, all course evaluations will appear for selection.
5. Evaluator/Evaluatee Filters – Allow for the input of an evaluator or evaluatee's name.

Access and work with the Question Analysis report

There are two ways to access and work with the Question Analysis report.

Option one

1. Click Reports in the global navigation bar.
2. Click Course Results under the QUICKLINK column.
3. Select Question Analysis from the Report dropdown.

Option two

1. Click Courses in the global navigation bar.
2. Click the course you want to work in.
3. Click View reports in the Results pod.
4. Select Question Analysis from the Report dropdown.

Color coding

• Red - No credit given
• Light green - Partial credit given
• Green - Full credit given 

A new feature of the Question analysis report is the addition of each question's correlation with overall student performance. The purpose of this addition is to identify the good questions from the bad questions through the use of point-biserial calculations. Based on findings by Helena Chmura Kraemer, the calculation ((mean1 - mean0) / standard deviation) * Square root of (proportion1 - proportion 0) is the difference between the mean evaluation scores of the correctly answering students, minus the mean evaluation scores of the incorrectly answering students, divided by the standard deviation of evaluation scores - times the square root of the difference of proportion of correct vs incorrect answer counts.

We show the point-serial numbers in green and yellow. Green indicates the question has a high correlation, while yellow indicates the question has a low correlation. Questions that do not meet the criteria for calculating the correlation are shown as N/A.

Additionally, users have the ability to check a box labeled Count no answers as no credit which will show the point-biserial score if an incorrect answer is provided as opposed to when a question is not answered.

Answer analysis

Each question and answer from an evaluation may be displayed for analysis. This report is beneficial for surveys with Likert-style responses or knowledge-based multiple-choice evaluations. 

1. Evaluation – select the desired evaluation for viewing.
2. Answer Analysis Field – the Question Instruction, Question, and Individual Answers will display. After each answer, the following will display:

• • • Points available
    • Percent chosen (with graphical representation)
    • Count

      3. Scenario Filters – filter evaluation choices by specific course scenario.

Additionally, generated reports for Question and Answer analysis show the date the report was generated, if the report was for a Question analysis or Answer analysis and if a Scenario, Evaluator, or Evaluatee filter was applied.