There are two features available in LAT to assist with measuring inter-rater reliability.
The Multiple Evaluation capability requires multiple evaluations for a single submission. In this case, at least one individual must be given the Reconciler permission. This permission enables you to reconcile multiple evaluations, suspend evaluation and call for additional evaluations of any work submission. By default, the Author can access only the final reconciled evaluation.
The Outside Evaluation capability provides the opportunity for an additional Evaluator to evaluate the submission. The Outside Evaluation is compared to the final evaluation within the Final Evaluation versus Outside Evaluation Report. This outside evaluation is not released to the Author and is kept completely separate from the final evaluation.
Advantages
Considerations
Multiple Evaluation Multiple evaluations take place prior to the outcome being released to the student. The results of these evaluations count toward the Author’s final score. Once the required individual evaluations are completed, an Evaluation Manager will need to reconcile them into a final evaluation to be released back to the student.
Allows control over which submissions evaluators have access to (manual grouping available).
Random sampling option not available.
Scores will have to be reconciled once the individual evaluations are completed.
The reconciliation has to be completed before the score can be released to the student.
Outside Evaluation Outside evaluations take place after evaluations of work have been marked as final. The results of this evaluations do not count toward the Author’s final score on the evaluated work and are never presented to Authors. Instead, they are used as a basis for comparison to the original evaluation (e.g., for measuring inter-rater reliability). The evaluation method is the same as the method that was used for the original evaluation.
Outside evaluations can be configured to apply to a random percentage sample of evaluated work, or to all work submitted for a particular area.
Random sampling of submissions is available.
No reconciliation necessary to finalize the score being released to the student.
Outside evaluations can occur after the original evaluation has been released to the student.
Does not allow control over which submissions evaluators have access to. (No grouping settings available).
Note: You can decide who has Outside Evaluator access.
Click below for a guide for Inter-Rater Reliability: