Assessment Sampling Guide

Assessment Sampling

LiveText’s Assessment Sampling feature allows a domain administrator to capture pools of student learning artifacts from course assignment submissions so that evaluators assess or re-assess them outside the confines of a course. This sampling tool can help programs and institutions gather more robust assessment data around student learning component areas and core learning objectives.

Before administrators can take advantage of this feature, the LiveText Administrator should

contact our Support Team by clicking Submit A Request and request to have the institution’s LiveText domain enabled for Assessment Sampling.

Once enabled, the Assessment Sampling feature is available in the Sampling Administrator account. Additionally, faculty accounts can be assigned as Sampling assessors.

Create a new Collection

1. Log into an account which has been enabled with Sampling Manager privileges.
2. Click on the Admin tab.
3. Click on the Sampling Collections link, which is located under Administration along the right side of the page.
4. Click on the + New button.

5. In the General Information area, enter a unique collection name in the Collection Name textbox.
6. Add an optional description in the Description textbox.

7. In the Assignment Filters area, select a term from the Term drop down menu.
8. For the Active option, you may select the Yes radio button to set the collection as active or select the No radio button to inactivate the collection. Inactivating a collection removes that collection from the Dashboard of each assessor.
9. To search for a course, you may enter a course code, course name, course description, or instructor name into the textbox to filter within the selected term. The list will be automatically updated after entering the search filter.

10. Select the checkbox next to any course code to include that course for artifact collection.
11. Click on the Show Assignments button to select specific assignments which will be used for sampling.

12. To search for an assignment, you may enter an assignment name, or assignment description into the text box to filter within the selected courses. The list will be automatically updated after entering the search filter.
13. To view an assignment’s description, click on the assignment’s Show Description link. The Show Description link will only appear if a description has been entered for the assignment. To hide a displayed assignment description, click on the corresponding Hide Description link.
14. Select the checkbox next to any assignment to include its submissions for artifact collection. Note: The Submission Count column displays the number of actual submissions followed by the number of expected submissions for each assignment. The number of actual submissions will continue to update until the collection is distributed.
15. In the Adjusted number of submissions by percentage text box, enter a percentage of the total number of actual submissions that will be included in the collection or enter the total number of submissions in the Total number of submissions to be included in collection text box. Note: Entering a number into one of these fields will change the content of the other field to retain the submission percentage to submission count ratio.

16. In the in the Rubrics area, click on the Browse button.

17. Select the checkbox next to the title of any LiveText assessment documents which are to be associated with the collection. Multiple assessment documents may be selected.

18. Click the + Insert Documents button.
19. In the Assessors area, click on the Search button.

20. In the Search for Assessors window, enter a first and last name, or a username for the account you would like to add as an assessor. Alternatively, you may enter the name of a group.
21. If the system recognizes what you’ve typed, one or more names and usernames that match your entry should appear below the text box. Click the name and username associated with the LiveText account that you'd like to add as an assessor. Repeat steps 20 and 21 to add another assessor or group of assessors.

22. Click on the + Add Assessors button, to add the selected assessor(s). Note: A minimum of three assessors must be added in order to use the Assessment Validation option.

23. You may select the Display assignment description to assessors checkbox to allow assessors to view the assignment description for each assigned submission.

24. By default, each submission in the collection is assigned to only one assessor. However, enabling the Assessment Validation option will assign each submission to two assessors.
If the scores entered by the assigned assessors differ by a value greater than the value entered by the Sampling Administrator, then the submission will be assessed by a third assessor.

To enable the Assessment Validation option,

1. Select the checkbox next to Assign one submission to two assessors.
2. In the Rubric element point discrepancy value textbox, enter the value by which the assigned assessors must differ in order to have a third assessor assigned.

25. In the Select Score Breakers section, choose the faculty that will be Primary Assessors or Score Breakers. If no Score Breakers are chosen, Primary Assessors will be randomly selected to do the assessment reconciliation when the discrepancy value threshold has been exceeded.

Note: The Score Breaker is only assigned rubric elements where a discrepancy exists, based on the set value.

26. You may click the Save button to save the collection for later editing.
27. You may click the Distribute button to distribute the collection to selected assessors.

Edit a Distributed Collection

Once a Sampling Collection has been distributed, the Sampling Administrator’s ability to edit that collection becomes limited. The name of the collection, the term, the courses and assignment submissions associated with the collection and the Assessment Validation option can no longer be changed.

1. Log into an account which has been enabled with Sampling Manager privileges.
2. Click on the Admin tab.
3. Click on the Sampling Collections link, which is located under Administration along the right side of the page.
4. In the Collection Name column, click on the name of the Sampling Collection you would like to edit.

5. Click the Edit button which is located across from General Information.

6. In the General Information area, you may add or edit the collection’s description.

7. In the Assignment Filters area, for the Active option, you may select the Yes radio button to set the collection as active or select the No radio button to inactivate the collection. Inactivating a collection removes that collection from the Dashboard of each assessor.

8. To view an assignment’s description, click on the Show Description link. The Show Description link will only be displayed if a description was entered for the assignment. To hide a displayed assignment description, click on the corresponding Hide Description link.

9. The Selected Submissions column displays the exact number of submissions which has been taken from each selected assignment.
10. In the Rubrics area, click on a Remove button to remove the corresponding rubric. Note: If any assessor has completed an assessment for the collection, rubrics may not be added to or removed from that collection.

11. In the Assessors area, you may remove any assessor who has not completed an assessment and does not have submissions assigned to them. You may need to reassign Sampling submissions before you can remove an assessor. Click a Remove button to remove the corresponding assessor. You may also add additional assessors to a collection at any time.

12. You may select the Display assignment description to assessors checkbox to allow assessors to view the assignment description each assigned submission.
13. Click the Save button to save your changes to the collection.

Reassign Sampling Submissions

1. Log into an account which has been enabled with Sampling Manager privileges.
2. Click on the Admin tab.
3. Click on the Sampling Collections link, which is located under Administration along the right side of the page.
4. From the list of collections, click on the progress bar associated with the Sampling Collection which has submissions to be reassigned.

5. Click the icon next to the name of the assessor whose artifacts will be reassigned.

6. In the Reassign Submissions box, select the name of the assessor who will receive the reassigned submissions from the dropdown menu.
7. Enter the percentage of artifacts the assessor should receive. For example, Training Faculty 2 could receive 20% of Training Faculty 1’s artifacts.
8. To reassign submissions to additional assessors, click on the +Add Assessor link. Repeat steps 6 – 8 as needed.

9. Click on the Reassign button. Note: When the Assessment Validation option has been enabled, the same submissions are assigned to multiple assessors associated with the collection. If any submissions which are being reassigned are already owned by the selected assessor, then the Sample Administrator will receive the following alert message:

All submissions except those owned by the selected assessor will be reassigned. The Sample Administrator may select another assessor to reassign more submissions. Submissions with submitted assessments are not able to be reassigned at any time.

Complete a Sampling Assessment

1. Log into an account which has been assigned as a Sampling assessor.
2. Click on the progress bar associated with a collection.

3. Click on a student in the Awaiting Assessment column. Note: The name associated with a student’s submission is hidden from the assessor.

4. If the Sampling Administrator has selected the Display assignment description to assessors checkbox then the assignment description will be displayed to the Sampling assessor. Click on any links under Artifacts to view the student submitted work. LiveText documents with open directly in the browser window. We recommend downloading external file attachments and then opening them using the appropriate software on your device.

5. Click the Assessment Rubrics button to view the rubric(s) that can be used in completing an assessment for the submission.

6. If there are multiple rubrics listed within the rubric window, you must click the title of a particular rubric to view that rubric.

7. Click the cells within the rubric to assess each element.

8. Click the blue balloon on the far left side of a row in the rubric and a comment box will appear in which you may enter a comment.

9. Once all scoring and comments have been entered on the rubrics, close the rubric window.
10. You may enter a grade in the Grade box, if applicable.
11. You may enter a comment in the Comment box, if applicable.

12. Once all parts of the assessment are completed, the assessor may click the Save button to save the assessment for later editing or click the Submit Assessment button to save and submit the assessment. Note: Completed Sampling assessments will not be viewable to the assessed student or the instructor for the course.

Run a new Sampling Assessment Report

1. Log into an account which has been enabled with Sampling Manager privileges.
2. Click on the Admin tab.
3. Click on the Sampling Collections link, which is located under Administration along the right side of the page.
4. Click on the Reports tab.
5. Click on the + New button.

6. You may enter a title and a description for the report. Note: A title is required to save the report.

7. Select the appropriate Collection, Assessment Rubric, Assessor and Inter-Rater Summary reporting filters.

8. Click on the View Report button.
9. In the Refine Results area, you may refine your results by selecting specific courses from the Course Section filter or assignments from the Assignment Information filter. Then click on the Refresh button to update the report.

10. You may click the Save button to save the Sampling assessment report.
11. You may click the Export to PDF button to export the report to a PDF.
12. You may click on the Export to CSV button located below each rubric report to export the data to a spreadsheet.

Export Sampling Data

1. Log into an account which has been enabled with Sampling Manager privileges.
2. Click on the Admin tab.
3. Click on the Sampling Collections link, which is located under Administration along the right side of the page.
4. Click on the Reports tab.

5. Click on the Export Sampling Data button.
6. Select a collection from the Collection Name dropdown menu.

7. You may optionally provide your own filename for the export or you may use the filename which was generated for you.
8. Click on the Yes button.

Interpreting the Sampling Data Export

The exported data will be contained in an Excel workbook; this workbook consists of two spreadsheets. The first spreadsheet is labeled Collection data. Each row is associated with a single Assignment Id (column I) which is not repeated on this spreadsheet. Each assignment on the Collection data spreadsheet will be associated with the following data fields in the export.

The second spreadsheet is labeled Submission data. Each row in this spreadsheet is associated with a single assessment completed by a Sampling assessor. Each Assignment Id field (column C), on the Submission data spreadsheet, will correspond to the Assignment Id field for a single row on the Collection data spreadsheet. You may use this field to correlate the assessment information on the Submission data spreadsheet with the collection information on the Collection data spreadsheet. Each assessment on the Submission data spreadsheet will be associated with the following data fields in the export.

In addition to these columns, each row on the Submission data spreadsheet contains columns marked with RL, RS and a number. This is the representation of the collected assessment data.

RL is an abbreviation for “Row Level” and represents the selected performance level for each row. RL1 would be the first row in the rubric, RL2 the second and so on. The number in this field is the position of the selected column, not the score or title of the column. A “1” in this field would indicate that the assessor selected the left-most column for that particular element; a “2” would indicate the next column to the right was selected and so on. These numbers do not reflect any weighting applied to each column, or any point value for each performance level; it is only a report of what position was selected.

See the following sample line from the export:

These values are visually represented in the following diagram:

The number of columns marked RL will depend on the length of the longest rubric in the ExhibitCenter™. As a result, some of the RL fields will be empty if the rubric is short.

RS is an abbreviation for “Row Score” and represents the weighted score the assessee received based on the point value associated with each performance level and the weighted value of each row in the rubric. It is calculated by multiplying the point value of the selected performance level by the weighted value of that row. A sample of the same line in the report, which was used in the previous example, is displayed below.

In this example, the rows have weights of 2, 4, 2, 1 and 1 respectively. The columns have a score of 3, 2, 1, and 0. Multiplying the selected column by the weight value of that particular row (not the percentage value) produces the scores that appear in the table above. The diagram below shows how this relates to the original rubric.

Similar to the columns marked RL, the number of columns marked RS will depend on the length of the longest rubric in the ExhibitCenter™. As a result, some of the RS fields will be empty if the rubric is short.

Frequently Asked Questions

1. What artifacts can be included in a collection?
Any and all artifacts attached to an assignment submission are included in sampling. This includes LiveText Documents, such as the e-Portfolio.
2. Are video and movie files included in sampling?
While streaming is not available in Sampling, video files can be included in collections.
3. Does the student’s assignment need to be assessed in order to be included in a Sampling Collection?
Students’ submissions do not need to be assessed. Any student submission Awaiting Assessment (Yellow Category on the Progress Bar) will be included in Sampling.

4. Can assessors be added to a collection after it has been distributed?
Assessors can be added to the collection at any time, even if artifacts have been assessed in the collection.
5. Can assessors attach files to an assessment?
Files cannot be attached to an assessment.
6. How many assessors can assess a submission?
In a collection, a submission can only be evaluated by one assessor, unless the Assessment Validation has been enabled.
7. Can assessors undo assessments?
Yes, assessors can undo assessments. However, this option is not available for collections with assessment validation enabled.