Outcome Performance Reports allow Assessment Coordinators to see performance based on an outcome within an Assessment Project.
Within the Outcome Performance Report, you’ll be able to see:
• The average score for the outcome selected
• The average score of each criterion in the outcome from all completed evaluations to date
• The number of scores from completed evaluations that factored into the calculation
• Which scored submissions are included in the calculations
From this report, you can filter by:
• Assignment
• Course
• Demographic Data
• Rubric Criteria
When Projects allow for more than one score for some or all submissions, OAP results include data from evaluations as they are completed; results are then updated as any subsequent evaluations are completed.
The average scores for every criterion included in an outcome are expressed in a bar chart, with the average for the learning outcome appearing as a line through the criteria. The distribution of scores for each criterion by quartile is expressed in a boxplot.
Outcome Performance Report results can be saved as a PDF.
Within the Outcome Performance Report, you also have the option of exporting individual rubric criteria results and comments to Microsoft Excel for manipulation outside of the system. These exports contain information regarding the:
• Assignment information such as the name, course, course section, term, and year.
• Student demographic information such as their name, student ID, and any additional demographics collected by the institution.
• Individual rubric criterion information and scores for students including N/A Usage and Score Count
• Outcome averages
• Comments provided by the Evaluator
To run an Outcome Performance Report, you must have Assessment Coordinator access. Once logged in, select your project. From the About This Project page select the View Results button from the navigation bar on the left-hand side.
After selecting View Results, choose which learning outcome you’d like to report on and then click Create Report.
The top of the next page will display averages by criterion in a bar chart. Hovering your mouse over a bar will display the average score, number of submission, and number of scores.
The bottom of the page will display the score distribution by criterion in a box plot. Hovering your mouse over a bar will display the maximum, upper quartile, median, lower quartile, minimum, and number of submissions.
Since each submission may have a different number of completed scores, both the number of submissions and the number of completed scores for each criterion is shown. Additionally, because it is possible for some criteria to be excluded from an Assignment, or marked N/A and excluded from averaging, the number of submissions and scores for each criterion may vary.
Because a criterion may be scored multiple times in one submission while only once in another, each criterion average in the Project is calculated from the average score of that criterion in each submission, in order for each submission to have equal weight in the Project’s criterion averages.
The same is true for calculating score distributions in that the min, max, median and quartiles are based on the average scores for the criterion from each submission, rather than averaging all scores from every Evaluator for each criterion.
To see the Average by Performance for a single criterion, select the drop-down menu next to View By. This drop-down menu will allow you to select a single rubric criterion.
Selecting a single criterion hides the other criteria in the bar chart and adds a histogram of average score frequencies so you may analyze the distribution in that criterion more closely.
If the option to choose N/A for criteria when scoring is available to Evaluators in a Project, the histogram includes the option to see how frequently N/A was either selected by an Evaluator for a submission scored only once or was selected by every Evaluator who completed the submission.
When a Project allows Evaluators to select N/A for a criterion when scoring a submission, that selection is not included in the averaging of scores for that criterion in that submission.
• If one Evaluator chooses N/A while two others choose 2 and 3 respectively, the average score for the criterion in that submission factoring into these results is 2.5 rather than 1.67 and that submission would not be included in the N/A frequency in that criterion’s histogram.
• If every Evaluator who scored a single submission selected N/A for that criterion, then the submission itself would not be included in the submission count for the criterion in these results and the submission would be counted in the N/A frequency in that criterion’s histogram.
From the Outcome Performance page, you may filter the data by Assignment, course and the demographic data that may have accompanied the student to see results for different segments of the scored submissions. To apply a filter, select the Filters button towards the right-hand side of the page.
Selecting the Filters button will open a panel on the left to show you which attributes are available for narrowing the scope of the report and its calculations.
• Click on a filter name to view and select values
• Use Search field to find item(s) when a large number are found for the selected filter
• As filters are added, the results are updated
• Select the check box next to the items you’d like to filter by
• Click the X to hide the filters panel again
A summary of applied filters then appears at the top of the report.
If no submissions with completed scores can be found for the outcome or criterion selected and based on the filters applied, the following message is displayed.
If this message seems in error, please note that reports are filtered to display score data for only submissions in courses that fall within the user’s organization level.
An Evaluator in a Project (in addition to being an assessment Coordinator or Faculty), may score work that is outside his or her organization level. In this case, scores for work the Evaluator scored that is attached to a course outside his or her organization will not be included in the results.
You may also generate a PDF of the results on screen for any outcome or specific criterion.
INDIVIDUAL SUBMISSION RESULTS
After selecting an outcome in the Outcome Performance Report, you may also generate a detailed report containing all the submissions from students in your organization with one or more completed scores in that outcome.
Submission details in results can not only be downloaded, but also viewed online in a Browse Scored Submissions area.
To download the individual results by criteria, select the Reports link at the upper right hand corner of the screen.
Select a CSV layout and save and open file in a spreadsheet or other application for review.
Two CSV options appear but they each contain very similar information. Both files include every submission for which at least one completed score can be found for the outcome selected. The single or average scores for each scored criterion is also included, along with the count of completed scores and, if allowed by Project settings, the instance of N/A selections and the Faculty intention for each of those criteria.
• CSV By Criteria: This includes a row for every criterion in every submission that has been scored at least once. It includes a column for the criterion name and the single or average score for that criterion in that submission. A submission in which 5 criteria were scored or marked N/A will occupy 5 rows in this file. The CSV also contains the year and terms in which the work was submitted and any demographic information collected by the institution.
• CSV By Submission with Comments: This includes a row for every submission with columns for every criterion in the outcome. A submission in which some or all criteria in the Project have been scored at least once will occupy a single row. This file also includes any comments on the outcome added by an Evaluator who scored the submission. The CSV also contains the year and terms in which the work was submitted and any demographic information collected by the institution.
BROWSING SCORED SUBMISSIONS
Browsing by Scored Submissions will present a list of all submissions in the Project from students belonging to your organization that have at least one score for the outcome selected. The single or average scores for each scored criterion is also included, along with the average of those scores shown as the Learning Outcome average.
To run this report, select the Reports button at the upper right hand corner of the Outcome Performance Report page and click Browse Scored Submissions.
The subsequent list is initially sorted by the Learning Outcome average, but can be sorted by any of the criterion scores or submission IDs by clicking the column heading. To sort in the opposite direction, click the heading again.
If N/A was selected by the Evaluator or all Evaluators who scored the submission, N/A will appear.
Selecting a single criterion from the criteria filter hides any submissions where that criterion was not scored due to Faculty intention and Project settings.
Leaving all criteria selected and selecting a single score range from the score range filter hides any submissions where the learning outcome average does not fall in that score range.
Choosing a single criterion and a score range will hide all submissions where the score or average score for that criterion does not fall in that range. It is still possible to sort the filtered list by another column.
To browse submissions with scores for another outcome in the Project, simply change the outcome at the top.
Selecting the arrowhead to the left of any submission will allow you to view the course, Assignment and student details.
If artifacts of student work were required, any supplied with the submission appear also. Nothing is listed if student artifacts were not required and if also no Evaluator added evidence when scoring the submission.
To view any of the artifacts or evidence, select the artifact name. This will open the file in OAP. If more than one file is included with the submission, arrows will appear on the left and/or right to view the next or previous file.
Running Outcome Performance Reports
Watermark Support