Inter-Rater Reliability Reporting in Taskstream LAT by Watermark

If you have utilized the Multiple Evaluation or Outside Evaluation feature within any of your DRF Programs, the  Performance by DRF Category Report will be able to show you the full multiple evaluation history. It will also give  you the ability to compare specific evaluators. 

 

Reporting for Multiple Evaluations

To access the Performance by DRF Category Report, first click the TS Coordinator menu.

Screen_Shot_2022-09-21_at_3.19.06_PM.png

Click the link for the Performance by DRF Category Report.

Screen_Shot_2022-09-21_at_3.19.15_PM.png

Select whether to run the report on a single DRF template or on one OR multiple DRF programs.

 

Screen_Shot_2022-09-21_at_3.19.22_PM.png

Select on whom you wish to report.

 

Screen_Shot_2022-09-21_at_3.19.31_PM.png

All Authors in one or more programs using selected DRF Template or Program(s) 

This option provides a view of the names of the Programs where the selected DRF Template was used. Select the  Programs you want to run reports on. 

Random sample of Authors in one or more Programs 

This option prompts you to select the sample size of the group you want to return. You can select to include  anywhere from 1% of the Authors enrolled in the program to 50% of the Authors enrolled in the Program. You will  need to choose from which Programs you want to pull the sample from. 

All Authors evaluated by a particular Evaluator 

This option provides a list of Evaluators in each Program using the selected DRF Template. Select the Evaluators  you wish to include in your report. 

All Authors grouped with a particular Evaluator 

This option provides a list of Evaluators in each Program using the selected DRF Template with whom Authors are  grouped. Select the Evaluators you wish to include in your report. 

A single Author  

Type a name in the search box provided. 

Advanced search  

This option offers search filter by demographics collected by your organization.

Screen_Shot_2022-09-21_at_3.21.15_PM.png

Authors that are eligible for an outside evaluation (enabled by special permission) 

These evaluations are used as a basis for comparison to the original evaluation (e.g., for measuring inter-rater  reliability). If you select this option, the next screen will prompt you to select which programs on which to run a  report to compare outside evaluation with final evaluations. You will not be able to filter by date. 

In the Filter by evaluation date area, you can choose whether you want to: 

  • Include all evaluations 
  • Include only items evaluated between certain dates 

Please note: When there are multiple evaluations for a single item, the system ONLY includes the latest  evaluation. 

Screen_Shot_2022-09-21_at_3.21.24_PM.png

Click Continue to generate the report. 

From the main results page, click the link (or magnifying glass icon) for a category or requirement in the DRF to  view the details for that area.

Screen_Shot_2022-09-21_at_3.21.33_PM.png

The category/requirement report shows the performance of the Authors who have access to the DRF area  selected.

Screen_Shot_2022-09-21_at_3.21.44_PM.png

From the Change View drop down menu, choose Multiple Evaluation & Reconciliation History.

Screen_Shot_2022-09-21_at_3.21.54_PM.png

The initial view will show you how many evaluations each Author’s submission had as well as results if multiple  evaluations were performed. It is recommended to click the boxes in the Show menu to display additional details.

Screen_Shot_2022-09-21_at_3.22.03_PM.png

You can also compare results between two specific evaluators using the Compare menus.

Screen_Shot_2022-09-21_at_3.22.13_PM.png

If there is enough data to compare between the two evaluators, the system will calculate the percent agreement  as well as the Pearson Correlation Coefficient.

Screen_Shot_2022-09-21_at_3.22.22_PM.png

If you use the Compare menus to select to compare two (2) Evaluators: 

  • One Evaluator’s scores are being compared against another Evaluator’s scores. 
  • For each Author evaluated by the selected Evaluators, you can see a column for the individual  evaluations completed by each Evaluator and a column for the range between those scores. 

What is the Percent Agreement? 

It is the number of evaluations where both Evaluators are within a certain score range, out of the number of  evaluations where both Evaluators have entered a score (the number of score pairs). 

To change the score range: 

  1. Enter the score range you want to use. 
  2. To display those evaluations completed by both selected Evaluators that fall within your indicated range,  click Change.  

What is the Pearson Correlation Coefficient? 

This is one of several measurements that can be used for measuring inter-rater reliability.

Screen_Shot_2022-09-21_at_3.22.50_PM.png

 

The formula for Pearson Correlation Coefficient is as follows:

Screen_Shot_2022-09-21_at_3.22.59_PM.png

Where: 
r = Pearson Correlation Coefficient 
X = Eval 1 
Y = Eval 2 
N = the number of score pairs 

If a rubric was used for evaluation, you also have the ability to view the comparison of criterion scores that were  given between the selected Evaluators. The Summary Statistics display the Percent Agreement and Pearson  Correlation Coefficient between the individual rubric criterion scores for all valid pairs.

 

Reporting for Outside Evaluations 

To enable Outside Evaluations on a requirement in your DRF Program, access the TS Coordinator menu.

 

Screen_Shot_2022-09-21_at_3.23.15_PM.png

 

Click the link for the Performance by DRF Category Report.

Screen_Shot_2022-09-21_at_3.23.40_PM.png

 

Select whether to run the report on a single DRF template or on one OR multiple DRF programs.

 

Screen_Shot_2022-09-21_at_3.23.48_PM.png

 

In the Select on whom you wish to report section, select Authors that are eligible for Outside Evaluation.

 

Screen_Shot_2022-09-21_at_3.23.59_PM.png

 

In the Filter by evaluation date area, you can choose whether you want to: 

  • Include all evaluations 
  • Include only items evaluated between certain dates 

Please note: When there are multiple evaluations for a single item, the system ONLY includes the latest  evaluation.

 

Screen_Shot_2022-09-21_at_3.24.08_PM.png

Click Continue to generate the report. 

In the main results, you will see a summary of results for completed outside evaluations.

 

Screen_Shot_2022-09-21_at_3.24.16_PM.png

 

Click the link or magnifying glass for a specific requirement to drill into the detailed results.

 

Screen_Shot_2022-09-21_at_3.25.00_PM.png

 

From the detailed results you will be able to compare the original final evaluation that was released to the student  to the outcome of the outside evaluation. The percent agreement and Pearson Correlation Coefficient are also calculated here. For an explanation of these calculations, view the equation under "What is the Pearson Correlation Coefficient?"

 

Screen_Shot_2022-09-21_at_3.25.08_PM.png


Click the link below to compare the Taskstream features of Multiple Evaluations versus Outside Evaluation:
Multiple Evaluation vs. Outside Evaluation

Was this article helpful?
0 out of 0 found this helpful

Articles in this section

See more
How to Contact Support
Click here to submit a ticket for the Support Team.
Watermark Academy
Click to access the Watermark Academy for free webinars, workshops, certifications, and free on-demand training