If you have utilized the Multiple Evaluation or Outside Evaluation feature within any of your DRF Programs, the Performance by DRF Category Report will be able to show you the full multiple evaluation history. It will also give you the ability to compare specific evaluators.
Reporting for Multiple Evaluations
To access the Performance by DRF Category Report, first click the TS Coordinator menu.
Click the link for the Performance by DRF Category Report.
Select whether to run the report on a single DRF template or on one OR multiple DRF programs.
Select on whom you wish to report.
All Authors in one or more programs using selected DRF Template or Program(s)
This option provides a view of the names of the Programs where the selected DRF Template was used. Select the Programs you want to run reports on.
Random sample of Authors in one or more Programs
This option prompts you to select the sample size of the group you want to return. You can select to include anywhere from 1% of the Authors enrolled in the program to 50% of the Authors enrolled in the Program. You will need to choose from which Programs you want to pull the sample from.
All Authors evaluated by a particular Evaluator
This option provides a list of Evaluators in each Program using the selected DRF Template. Select the Evaluators you wish to include in your report.
All Authors grouped with a particular Evaluator
This option provides a list of Evaluators in each Program using the selected DRF Template with whom Authors are grouped. Select the Evaluators you wish to include in your report.
A single Author
Type a name in the search box provided.
Advanced search
This option offers search filter by demographics collected by your organization.
Authors that are eligible for an outside evaluation (enabled by special permission)
These evaluations are used as a basis for comparison to the original evaluation (e.g., for measuring inter-rater reliability). If you select this option, the next screen will prompt you to select which programs on which to run a report to compare outside evaluation with final evaluations. You will not be able to filter by date.
In the Filter by evaluation date area, you can choose whether you want to:
- Include all evaluations
- Include only items evaluated between certain dates
Please note: When there are multiple evaluations for a single item, the system ONLY includes the latest evaluation.
Click Continue to generate the report.
From the main results page, click the link (or magnifying glass icon) for a category or requirement in the DRF to view the details for that area.
The category/requirement report shows the performance of the Authors who have access to the DRF area selected.
From the Change View drop down menu, choose Multiple Evaluation & Reconciliation History.
The initial view will show you how many evaluations each Author’s submission had as well as results if multiple evaluations were performed. It is recommended to click the boxes in the Show menu to display additional details.
You can also compare results between two specific evaluators using the Compare menus.
If there is enough data to compare between the two evaluators, the system will calculate the percent agreement as well as the Pearson Correlation Coefficient.
If you use the Compare menus to select to compare two (2) Evaluators:
- One Evaluator’s scores are being compared against another Evaluator’s scores.
- For each Author evaluated by the selected Evaluators, you can see a column for the individual evaluations completed by each Evaluator and a column for the range between those scores.
What is the Percent Agreement?
It is the number of evaluations where both Evaluators are within a certain score range, out of the number of evaluations where both Evaluators have entered a score (the number of score pairs).
To change the score range:
- Enter the score range you want to use.
- To display those evaluations completed by both selected Evaluators that fall within your indicated range, click Change.
What is the Pearson Correlation Coefficient?
This is one of several measurements that can be used for measuring inter-rater reliability.
The formula for Pearson Correlation Coefficient is as follows:
Where:
r = Pearson Correlation Coefficient
X = Eval 1
Y = Eval 2
N = the number of score pairs
If a rubric was used for evaluation, you also have the ability to view the comparison of criterion scores that were given between the selected Evaluators. The Summary Statistics display the Percent Agreement and Pearson Correlation Coefficient between the individual rubric criterion scores for all valid pairs.
Reporting for Outside Evaluations
To enable Outside Evaluations on a requirement in your DRF Program, access the TS Coordinator menu.
Click the link for the Performance by DRF Category Report.
Select whether to run the report on a single DRF template or on one OR multiple DRF programs.
In the Select on whom you wish to report section, select Authors that are eligible for Outside Evaluation.
In the Filter by evaluation date area, you can choose whether you want to:
- Include all evaluations
- Include only items evaluated between certain dates
Please note: When there are multiple evaluations for a single item, the system ONLY includes the latest evaluation.
Click Continue to generate the report.
In the main results, you will see a summary of results for completed outside evaluations.
Click the link or magnifying glass for a specific requirement to drill into the detailed results.
From the detailed results you will be able to compare the original final evaluation that was released to the student to the outcome of the outside evaluation. The percent agreement and Pearson Correlation Coefficient are also calculated here. For an explanation of these calculations, view the equation under "What is the Pearson Correlation Coefficient?"
Click the link below to compare the Taskstream features of Multiple Evaluations versus Outside Evaluation:
Multiple Evaluation vs. Outside Evaluation