Assigning DRF Template Evaluation Methods

About Evaluation Methods

In order for the various artifacts students are submitting in the DRF to be assessed, the appropriate evaluation  instrument (rubric, form, etc.) must be assigned to each category/requirement. These assignments are made in the  Step 3: Evaluations tab in the DRF Template Builder.

Consistency wherever possible is recommended when assigning evaluation methods in DRF Templates to make data  aggregation/reporting easier.

Selecting the Level of Evaluation

After you have saved the structure of your new template, you are navigated to the Evaluation tab to choose the level  at which submissions to this DRF will be evaluated. You can evaluate by requirement (Each requirement evaluated  individually) OR category (Requirements in each category evaluated simultaneously).

To set the evaluation level for your DRF Template:

Select either requirement- or category-level evaluation and click Save & Continue.

(Optional) Check to include an overall folio evaluation ("OFE"). Authors will not submit the OFE, but if a "holistic" score is required for the complete body of work, you may wish to include this option. Evaluators will add a score for the OFE that is separate from the scores recorded for the individual categories or requirements. The option to include an OFE appears at both levels. The OFE selection checkbox also appears on the Evaluation tab.

You are navigated to the Step 3: Evaluation tab to select a specific evaluation method for each level Change Evaluation Level

You can return to this option by clicking the Change Level of Evaluation button in the upper left of the Step 3: Evaluation home page.

You CANNOT change the level of evaluation for a DRF if any of the following are true:

• Authors have already submitted work in any requirement using this template.
• The DRF contains any "no-submission" requirements
• Any Field Placement database (active or not) is associated with a DRF program that uses this template.

In this tab, DRF areas are displayed in the same order as they appear in the template structure. On this page, you decide:

• which evaluation method occurs at each level
• if you want DRF weighting
• how to set the point values
• whether evaluation methods are visible to Authors

If you choose to make Evaluation methods visible to Authors, they will see how the final score will be calculated (as well as a blank version of the rubric or form, when included). Note that this is SEPARATE from allowing Authors to see their actual SCORES.

An option to include an overall folio evaluation ("OFE") appears at the top of the list. Whether or not this box is checked is based on the choice you made when you selected the level of evaluation for this template. 
Note: Once the overall folio evaluation has been started for one Author, you cannot remove an OFE from the related template.

Types of Evaluation Methods

Basic Evaluation Methods

Three "basic" evaluation methods are available for scoring work submitted to a DRF.

Advanced Evaluation Methods

Assigning Basic Evaluation

Click the Assign Evaluation Method button to the right of a specific requirement/category.

If you have selected to evaluate at the category level, the method you select is assigned to the category as a whole;  otherwise each requirement is individually assigned a method.

The Select from Basic and Advanced Evaluation Methods page loads. Click Select for one of the available Basic  evaluation methods.

You will then be brought to the evaluation method details page. For basic evaluation methods, you can enter optional notes to the evaluator. If you’ve chosen Write-in Score, you’ll also have to specify a maximum point value for that  requirement/category. Click Apply Changes when finished.

Assigning Advanced Evaluation Methods

Rubric

Rubric evaluation can help you evaluate a section according to how well the work aligns with the pre-set assessment  criteria contained in the rubric. When you select Rubric as your evaluation method for a DRF area, you can either create a  new rubric or use an existing rubric. You are required to:

• create a rubric to use for evaluating this requirement;
• assign a rubric scale;
• (Optional) assign weights to rubric criteria, to more heavily consider some criteria over others in  calculating average values;
• (Optional) set rubric display preferences to indicate how rubrics and scores should be displayed to  Evaluators and Authors.

Click the Assign Evaluation Method button to the right of a specific requirement/category.

If you have selected to evaluate at the category level, the method you select is assigned to the category as a whole;  otherwise each requirement is individually assigned a method.

The Select from Basic and Advanced Evaluation Methods page loads. Click Select for Rubric.

1. Select a rubric from the pull-down list of existing rubrics.

   Assigning DRF Template Evaluation Methods 

   1. Optional Click Preview to view the selected rubric in a pop-up window and confirm that it is appropriate  for this DRF area.

      

      This option will not appear if there are no rubrics available for you to

      select. In that case, it will be necessary for you to create a new rubric.

2. Use a radio button to designate which kind of numeric values can be entered by Evaluators. Choose between  allowing the full range of scores OR only the exact values specified in the scale.

   

3. (Optional) Check to enable either or both rubric comment areas and enter a label for each comment area you  enable. You can choose to include:
   1. Comment area for each criterion - used for review comments that are sent back to the Author with the rubric scores.

   2. Comment area for administrative use only - only visible by Program Managers with the Generate Reports  permission.

      

      Evaluators will not be forced to enter comments for either of these areas.

      Note: Your choice does not affect the overall comment area that is available for all evaluations.

4. Select the method to be used to indicate a final mark or score. You can choose only one final scoring method for this DRF area.

   

   It is recommended to use consistency whenever possible in choosing the final scoring method for each requirement.

   Select rubric visibility to determine who should have access to the completed evaluation rubric. You can choose to:

   1. Include the rubric with the Authors’ evaluation report as well as in the administrative record, OR
   2. Only make the rubric available for the administrative record. In this case, ONLY Managers of the Program can view the rubric. Authors view only the final mark/score and overall comments (if any). 

   

   This does not apply to the Evaluation Method visibility setting with which the Author sees a BLANK copy of the rubric.

5. (Optional) Enter a note to the Evaluator. For example, a note can clarify how the final scoring will be performed.  These directions are visible to the Evaluator during the evaluation process.
   1. Note has a maximum of 8,000 characters. A character count automatically displays as you write. • To enhance the appearance of your note, use the HTML toolbar in this area.
6. (Optional) To quit without saving this custom rubric evaluation for this area, click Cancel.
7. Click Apply Changes. You will then be navigated to a screen to assign values to the rubric scale.

In order for Evaluators to use the rubric method for this DRF area, you must create an evaluation scale. Assign a point  value to each column (level of performance) by typing a number in the corresponding Enter Value box.

Points can be sequenced (e.g. 0, 1, 2, 3) or can be gapped (e.g. 5, 10, 15, 20). Regardless of the numbers used, the  scale should align with performance so that the highest performance level corresponds with the highest numeric  score, while the lowest score equates to the worst level of performance (e.g. on a scale of 1 - 10, 1 is the worst  performance and 10 is the best), even if the columns are reversed to list the best performance in the left-most  column.

Reports that you generate later aggregate data based on the assumption that the best mark that someone  receives is also given the highest score.

When an Evaluator uses the rubric to assess a submission, the raw score will be determined according to this range.  The final scoring method selected for this rubric will determine how this raw score is interpreted.

The effectiveness of the point scale for evaluation will also be impacted by whether you have chosen (in Step 2 of the  "develop a custom rubric evaluation" procedure, described earlier) to allow evaluators to use the full range of scores  or only the values specified in this scale.

If you want to allow Evaluators to score certain rows of the rubric as Not Applicable, check Enable 'N/A' as a Scoring Option for those rows.

• Values of 'N/A' will not count toward the average or cumulative score.
• Weighted rubrics are not allowed to use 'N/A' as a scoring option for ANY criteria
• Evaluators are NOT permitted to select 'N/A' for ALL the criteria. AT LEAST ONE raw score must be  available to drive the average and cumulative value, even if the final mark for the DRF area will be  indicated as a numeric score, pass/fail or meets/does not meet requirements.

Once you've assigned the rubric scale, you have 3 options:

• Set Rubric Weighting
• Set Rubric Display
• Save and Return

Rubric Weighting

If you wish to assign weights to each rubric criterion, click Save Draft followed by Set Weighting Preference.

• Rubric weighting begins with the premise that the rubric as a whole (all criteria) has a combined value of  100%. Weighting values, established as percentages of this whole, enable you to indicate how much each  criterion is worth relative to other criteria in the same rubric. For example, in a rubric with four, evenly weighted criteria, each criterion is worth 25%.
• Weighting values apply only to the rubric as it relates to evaluating the CURRENT category or  requirement. They do not save to the rubric itself for reuse as an evaluation method for any other area.  • These weighting values are completely separate from DRF weighting that might be selected to apply  ACROSS categories/requirements.
• By default, no weighting is set for a rubric and each criterion will have the same weight when calculating  average scores across criteria.
• Rubric weighting only appears as an option if the final scoring method for the rubric (in Step 4 of the  "develop a custom rubric evaluation" procedure, described earlier) has been set to have the system  compute the average score or total cumulative points.

To continue assigning weights to your rubric, select Weighting Preferred as your preference and click Apply Changes.

You are navigated to a display of the rubric that includes a Weight Alignment column. Once you select this preference,  you MUST assign percentage values to all rubric criteria or Authors will NOT be able to submit work.

For each criterion (row), enter a percentage that indicates what this criterion is worth relative to the rubric as a  whole. The sum of all percentage weights must equal 100. The system flags rubrics as incomplete if the weighted  scores do not equal 100.

To save your work and remain on the Assign Values screen for this rubric, click Save Draft

OR

To return to the DRF Evaluation home page after saving, click Save and Return.

Rubric Display

To display Rubric Scoring Preferences (e.g. how scores should be displayed to Authors) click Save Draft followed by  Set Rubric Display.

You can control how rubric headings and scores should be displayed to Evaluators. Choose to display for the rubric  and scores as:

1. Performance Level Labels equivalent to numeric values (e.g. Exemplary). Label text is pulled from column  headings
2. Numeric Values plus Performance Level Labels (e.g. 3.0 Exemplary)

3. You can control how rubric scores should be displayed to Authors. Choose to display the scores for each row of the  rubric as:

   1. Numeric Values (e.g. 3.0)
   2. Performance Level Labels equivalent to numeric values (e.g. Exemplary). Label text is pulled from column  headings
   3. Numeric Values plus Performance Level Labels (e.g. 3.0 Exemplary).
   4. Just Rubric Comments -- neither numeric values nor labels will display. This preference is only available if  the author-facing comment area was enabled in Step 3 of the "develop a custom rubric evaluation"  procedure, (described earlier).

   If Evaluators assign a score that falls between two performance level labels, both performance level labels are  included as the final score (e.g., Acceptable/Target).

   To continue setting the rubric display:

   1. Select your preference.
   2. (Optional) To quit your action, click Cancel.
   3. Click Apply Changes.

To save your work and remain on the Assign Values screen for this rubric, click Save Draft;

OR

To return to the DRF Evaluation home page after saving, click Save and Return

Save and Return

To return to the DRF Evaluation home page, click Save and Return. You will be asked to confirm that the scale aligns  properly with levels of achievement.

Custom Evaluation Form

A Custom Evaluation Form enables you to evaluate an area of a DRF against criteria defined in an evaluation form  designed by you or someone from your organization. Forms are a flexible medium, enabling assessment criteria to be  presented in a variety of ways including ratings scales, multiple selection checkboxes, and single response radio  buttons. By including questions to be answered by free-form text of varying length, you can extend the scope of  evaluation to provide focused commentary and/or author comments.

When you select Custom Evaluation Form as your evaluation method, you can either create a new form or use an  existing form. Regardless of the form you choose, you must also select a method for Evaluators to assign an overall  final score for the related requirement.

To customize a form evaluation for a DRF area:

Select a form from a pull-down list of existing forms. You can select a form that you have created, or one that was  created by another member of your learning community.

(Optional) Click Preview to view the form in a pop-up window and confirm that it is appropriate for this DR area.

Use a radio button to Select final scoring method for this area. Choose between three basic evaluation methods:

• Numeric Score ("write in")
• Pass/Fail
• Meets/Does Not Meet Requirement

If you choose Numeric Score, you must indicate the maximum number of points that an Evaluator can assign.

Select form visibility to determine who should have access to the completed evaluation form. You can choose to:

• Include the form with the Authors' evaluation report as well as in the administrative record •
• Only make the form available for the administrative record. In this case, ONLY Managers of the DRF  Program can view the assessment form. Authors view only the final mark/score.

(Optional) Enter a note to the Evaluator. These directions are visible to the Evaluator during the evaluation process  and has a maximum of 8,000 characters. For example, a note can clarify how the final scoring will be performed. To  enhance the appearance of your note, use the standard HTML toolbar in this area.

(Optional) To quit without saving this form customization for this area, click Cancel

Click Apply Changes.

Combined Evaluation Method (Form + Rubric)

This evaluation method combines the scaled assessment of a rubric with the flexibility and potential for focused  commentary provided by a custom evaluation form. When you select Combined Evaluation Method (Form +Rubric) as your  evaluation method, you are required to select both a form and a rubric. You can either select these from existing artifacts,  or create new ones.

Because this method combines two artifacts, your setup includes parameters for the use of each artifact as well as criteria  for the final (combined) scoring.

To customize a combined (form + rubric) evaluation for a DRF area:

1. Select a form from a pull-down list of existing forms.

   (Optional) Click Preview to view the selected form in a pop-up window and confirm that it is appropriate for this DRF area.

   

2. Select a rubric from a pull-down list of existing rubrics.

   (Optional) Click Preview to view the selected rubric in a pop-up window and confirm that it is appropriate for this DRF area.

3. The radio button you select for first evaluation method to display will determine which evaluation method appears  first on the page viewed by Evaluators (by default, Form appears first).

   

4. Customize rubric parameters:
   1. Use a radio button to designate which kind of numeric values can be entered by Evaluators. Choose  between allowing the full range of scores OR only the exact values specified in the scale.
   2. (Optional) Check to enable either or both rubric criterion comment areas and enter a label for each  comment area.

   3. Select the method to be used to indicate a final mark or score. You can choose only one final scoring  method for this DRF area.

      

5. Use radio buttons to individually select form and rubric visibility.

   

6. (Optional) Enter a note to the Evaluator. For example, a note can clarify how the final scoring will be  performed. These directions are visible to the Evaluator during the evaluation process.
   1. Note has a maximum of 8,000 characters. A character count automatically displays as you write.
   2. To enhance the appearance of your note, use the standard HTML toolbar in this area.
7. (Optional) To quit without saving this combined evaluation method for this area, click Cancel.
8. Click Apply Changes.

You are navigated to a screen to the Rubric Scale page. When you have completed development of your selected rubric,  click Save and Return.

Import Parameters (Test or Exam and GPA requirements)

For Test or Exam and GPA requirements, part of the assignment of an evaluation method is the definition of the import  parameters. This evaluation option enables Evaluators to:

• Enter scores for each Author individually
• Import scores from a comma separated values (.CSV) file for a group of Authors

To define import parameters for a Text/Exam/GPA requirement:

In the Structure/Requirements area, create your Test or GPA requirement

In the Evaluation tab of the DRF Template Builder, click Define Import Parameters for the Test/GPA requirement.

• Enter the maximum number of points that can be entered for the requirement. (The maximum must be a  positive whole number.)
• (Optional) Add an ID or Description.
• Click Apply Changes.

Copying Evaluation Methods from one requirement or category  to another

You can save time and ensure consistency by copying evaluation methods from one requirement or category to  another. The copy will include any details you may have specified for the initial DRF area with which it is associated.

To copy an evaluation method from another area:

1. Select a requirement from the Copy method from another requirement pull-down list.
2. (Optional) Click Preview Method to see a mockup of how the DRF area will appear to an Evaluator and  determine whether this Evaluation method and details is appropriate for this area.
3. Click Copy Method. You are asked to confirm that you want to continue.