The Assessment Insight System™ (AIS) allows institutions and programs to:
- Define and use Standards/Outcomes
- Design pathways for program and student success
- Be thoughtful in their planning and re-design efforts
- Document assessment findings and action plans
- Access data for analysis
- Showcase assessment plans, reports and findings to internal and external stakeholders
The information contained within the following guide is designed to assist institutions in effectively implementing the Assessment Insight System from the role of the Assessment Insight System Administrator account.
If you have any questions, please email support@watermarkinsights.com
AIS Account Setup (Authors/Contributors & Reviewers)
Registering Users
If your institution is already using LiveText and faculty already have accounts, they will not need to re-register an account to use the Assessment Insight System. If faculty do not have LiveText accounts, access can be easily provided by constructing a group and using an in-app mail merge.
The following steps outline how to provide access to faculty and other plan contributors that do not have active LiveText accounts and are not the instructor of record:
1. Use the attached "module_AIS" file to construct a group.
2. Complete columns H, I, and K with first name, last name, and email address.
3. Save as .csv, .xls, .xlsx, .ods, or .sxc file.
4. In the Admin account, click on the Course Admin tab and then "Course Catalog Import"
5. Upload your "module_AIS" file
6. Once the file has finished uploading ("Completed" will be displayed on the page), click "Validate Course Catalog"
7. If there are no errors, click "Commit Course Catalog." If there are errors, please correct them in your file and then repeat steps 5 and 6
Inviting Users to AIS
LiveText includes an application called Keycode Management, which allows institutions to customize an email message and invite users to access AIS. Once the import process outlined above is completed (if necessary), you can follow these steps to send a custom invitation:
1. Log into the Keycode Management admin account and click on the Admin tab.
2. Click the Customize registration email message.
3. For Keycode Type, choose student or faculty. Institutions can customize the email for assessment teams, faculty and students.
4. Compose your message in the Email Message field.
NOTE: Any text that is in the {} area must not be edited and remain as it is displayed. All other text can be edited.
5. Click Preview/Save to view the completed customization.
6. Click Yes to continue or No to return and make changes.
7. Click Distribute and manage keycodes.
8. Click the Distribute link at the end of the row containing the appropriate batch of keycodes.
NOTE: Please be cautious to select the CORRECT batch you wish to distribute.
9. From the dropdown menu, choose the applicable Term for distribution of keycodes.
10. Select the checkbox in the far right column for the course(s) with instructor(s) that need to be invited.
NOTE: All of the course modules and instructors of record will display for the selected term. This includes instructors who have registered, have been invited but not yet registered, and those that have not been invited.
11. Click Send Invites.
12. Click Ok to close the message screen.
13. The system will then display a notification message that the emails will be sent out in 30 minutes.
Once the invitee receives the email invite, the user will click on the provided URL and follow the registration prompts.
Collection and Display Unit Types
Within AIS there are three collection units: Curriculum Map, Assessment Map, and Assessment List. The components of each will be explained within this guide. Below is a chart summarizing the attributes of each.
| Curriculum Map | Assessment Map | Assessment List | Assessment Map/List Findings | |
| LiveText Outcomes | X | X | X | |
| Free Text Outcomes/Goals | X | |||
| Legends | X | X | X | |
| Courses/Events | X | X | ||
| Assessment Measures | X | X | ||
| Improvement Types | X | |||
| Criterion | X | X |
Defining Curriculum Mapping Legend
During the creation of curriculum maps, institutions align courses to standards/outcomes and indicate whether they are, for example, (I) Introduced, (R) Reinforced, or (A) Assessed. The curriculum map can be used to support an assessment map or assessment list. LiveText gives you the freedom to develop a custom legend for institutional purposes to meet this requirement. AIS administrators can also create custom legends for each program of study and at the institutional level.
Legends
1. Click the AIS tab.
2. Click the Legends link.
3. Click New.
4. Add the abbreviation of the milestone, example: “I“ for introduced.
5. Add the full Title in the description box.
6. Choose Yes or No to indicate whether or not evidence has been collected.
7. Click Save.
NOTE: Unused Legends can be deleted. Used Legends cannot be deleted but can be edited.
Defining Assessment Measures
Once a curriculum map is developed and all standards/outcomes have been aligned to courses, programs will define HOW they intend to successfully measure the outcome. Assessment Measures examine measures/activities that are being observed within the curriculum map. All of the milestones created in the legend can be defined further with Assessment Measures and Criterion for success. The Administrator will pre-populate "Assessment Measures" options. Options might include: Grade, Quiz, Standardized test, Rubric, Survey, Other...
Assessment Measures
1. Click the AIS tab.
2. Click the Assessment Measures link.
3. Click New.
4. Add the Name of the Assessment Measures.
5. Choose whether the Assessment Measures is based off of a Direct or Indirect measure.
6. Add a description of the Assessment Measures.
7. Click Save.
NOTE: Unused Assessment Measures can be deleted. Used Assessment Measures cannot be deleted but can be edited.
Defining Improvement Types
An Improvement Type is an optional feature that can be added to Assessment Map Findings or Assessment List Findings. These are customizable listings that can be used by the Author to represent the action plans regarding the representative evidence as displayed in either the Assessment Map Findings or the Assessment List Findings. When an Improvement Type is by the Program/Units it will populate the Improvement Type Summary (optional) table for both Met or Not Met measures.
1. Click the AIS tab.
2. Click the Improvement Types link.
3. Click New.
4. Add the Name of the Improvement Type.
5. Add a description of the Improvement Type.
6. Click Save.
NOTE: Unused Improvement Types can be deleted. Used Improvement Types cannot be deleted but can be edited.
Importing Courses/Events
The Courses/Events feature can be used for both academic and non-academic alignment of outcomes to courses, or outcomes to educational experiences. AIS administrators have the ability to import or manually add Courses/Events. Once courses or events are imported, they can be used for either Curriculum Mapping, Program Review or other events designated by the AIS administrator.
Course/Events Batch Import
1. Click the AIS tab.
2. Click the Course/Events link.
3. Click on Import Courses/Events link.
4. For a new sample file, click Download a Sample Import File.
5. Click Select File after the file has been completed and saved.
6. Click Commit after to finalize the import.
Manual Creation of Courses/Events
1. Click the AIS tab.
2. Click the Course/Events link.
3. Click Add and enter the name.
4. Click Save.
NOTE: Unused Courses/Events can be deleted. Used Courses/Events cannot be deleted but can be edited.
Programs/Units
Program/Unit Coordinators are typically the primary contact responsible for documenting and submitting an assessment plan, but they can also add collaborators (Authors) as well as add Courses/Events. Authors will be able to add content to the plan.
Please define the "Program" or "Unit" which will be responsible for documenting and submitting plans to the Administrator. Coordinators can be added to the Programs/Units by either a manual or batch import process.
Adding Program/Units- Manual
1. Click the AIS tab.
2. Click the Program/Units link.
3. Click New.
4. Add the name of the Program/Unit, example: School of Education, Registrar Office.
5. Add a description.
6. Search for the person you want to make a Program/Unit Coordinator (Search by first and/or last name, or by LiveText username).
7. Search for the person you want to make an Author (Search by first and/or last name, or by LiveText username).
8. Once users are selected, click Save.
Adding Program/Units- Batch Import
1. Click the AIS tab.
2. Click the Program/Units link.
3. Click on the Import Programs/Units.
4. Click on Download a sample import file.
5. Populate all required fields (NAME, DESCRIPTION, USERTYPE (Coordinator or Author), FIRSTNAME, LASTNAME, EMAIL). USERDOB is optional
6. Once completed, Upload the file and the Commit. (Acceptable file types are *.xls, *.xlsx, *.ods, *.sxc, *.csv)
Note: The batch import function is for adding new Programs/Units and does not edit or remove existing Programs/Units. Unused Programs/Units can be deleted. Used Programs/Units cannot be deleted but can be edited.
AIS Plan Reviewers
In addition to the AIS Administrator collecting and reviewing plans, other Reviewer groups can be created and assigned.
To create new Reviewers, following these steps:
1. Click on the Reviewers link.
2. Click New.
3. Enter the Name for the Evaluator Group.
4. Enter a Description.
5. Search for the person you want to make Evaluators (Search by first and/or last name, or by LiveText username).
6. After all members have been added, click Save.
NOTE: Unused Reviewer Groups can be deleted. Used Reviewer Groups cannot be deleted but can be edited.
Template (Plan) Creation
AIS has been developed to provide an area within LiveText for both educational and non-education programs to design and complete their program plans and reports. Once a template is completed, Evaluators can review the document and use assessment tools, such as rubrics, to collect data.
Templates consist of sections. A section can consist of survey style fields that include: text boxes, number boxes, radio fields, checkboxes, dropdown menus and attachment fields for each of the sections. Depending on the purpose of the template, a section may also be designed for a curriculum map, assessment map, assessment findings, or other purposes. The following demonstrates the process for creating templates and building template sections.
It is suggested that you first finalize your template outside of LiveText. Once you are satisfied, it can be added to the AIS for distribution.
Create a Template
1. Click the AIS tab.
2. Click the Templates link.
3. Click New.
4. Click in the Title box to add a unique name/title to the template.
5. Hover over the Section Title, and then click on the pencil/edit icon.
6. Name the Section and click Save.
Template Basics
Basic items include the Title and Label links for each section of the Template. Titles can be thought of as headers and Labels as sub-headers.
1. Click the Title icon in Basic.
2. Click in the text box or the pencil/edit icon.
3. Add content to the Title.
4. Click Label icon under Basic.
5. Click in the text box or the pencil/edit icon.
6. Add content to the Label.
Template Components
Template components are the various tools and resources that can be included in a section. They include survey style fields, curriculum map, assessment map, assessment map findings, assessment list, and assessment list findings. The following demonstrates how to create each of these components.
Adding a Text component
1. Click the Text icon component.
2. Click pencil/edit icon.
3. Add content in the Label box.
4. Add a Reminder Message (optional).
5. Click Save.
Adding a Rich Text component
1. Click the RichText icon component.
2. Click pencil/edit icon.
3. Add content in the Label box.
4. Add a Reminder Message (optional).
5. Click Save.
NOTE: The following items will not be displayed in the Word output of the Plan Content Report if used within a Rich Text component by the Author: color, color highlight, underline, image, font size, and text alignment. If the intended submitted item is an image and it needs to be accessible in the output report, the Template Manager must supply an Attachment component where the authors can upload the image as an attachment.
Tables can be used but cannot contain spanning columns or rows. Each cell cannot be adjacent to more than four other cells.
Also, Rich Text does not export to the Excel output of the Plan Content Report.
Adding a Number Component
1. Click the Number icon component.
2. Click pencil/edit icon.
3. Add content in the Label box.
4. Add a Reminder Message (optional).
5. Click Save.
Adding a Checkbox Component
1. Click the Checkbox icon component.
2. Click pencil/edit icon.
3. Add content to the Label. For example, this might be a multiple choice question.
4. Add a Reminder Message (optional).
5. Edit the default Option 1 to the caption for the first checkbox.
6. Click on Click to add option to create another option field.
7. Edit the default Option 2 to the caption for the second checkbox. Repeat this process for the desired number of checkboxes.
8. Click Save.
Adding a Radio Button Component
1. Click the Radio icon component.
2. Click the pencil/edit icon.
3. Add content to the Label. For example, this might be a single choice question.
4. Add a Reminder Message (optional).
5. Edit the default Option 1 to the caption for the first radio button.
6. Select Click to add option to create another option field.
7. Edit the default Option 2 to the caption for the second radio button. Repeat this process for the desired number of checkboxes.
8. Click Save.
Adding a Dropdown Component
1. Click the Dropdown icon component.
2. Click the pencil/edit icon.
3. Add content to the Label. For example, this might be a single choice question
4. Add a Reminder Message (optional).
5. Edit the default Option 1 to the caption for the first choice in the dropdown menu.
6. Select Click to add option to create another option field.
7. Edit the default Option 2 to the caption for the second choice in the dropdown menu. Repeat this process for the desired number of checkboxes.
8. Click Save.
Adding an Attachment Component
1. Click the Attachment icon component.
2. Click the pencil/edit icon.
3. Add content to the Label. For example, this might be instructions for which document to attach.
4. Add a Reminder Message (optional).
5. Click Save.
NOTE: Any attachments provided by an Author can be exported to a zip file by the Administrator in the reporting platform.
Adding an Outcome Component
1. Click on the Outcome icon component.
2. Click the pencil/edit icon.
3. Click in the Title box to rename the outcomes part of the section.
4. From the Set Library dropdown menu, select a standards library from which to search. Choices include All Sets, LiveText Sets, Institution, or My Favorite Sets.
5. Once a Set Library is selected, that entire group can be searched.
6. From the Set dropdown menu, select a standards set. Sets can be located by scrolling through the list or typing the first few characters of the Standard name (Ex: “HI” for Hawaii).
7. To search for a standard set by using the free text Search box, type in a word or phrase and then click Search. This field allows users to search by Identifier, Keyword, or Level.
8. Once the desired Standard Set has been located, check the box next to the Identifier. This will push the identifier to the right side of the screen and place it within the Selected column.
a. Or click the checkbox next to Select All to choose all of the standard sets displayed in the view plane.
9. Click Save.
Other Sections
Adding Curriculum Maps, Assessment Map, Assessment Findings, Assessment List and Assessment List Findings components are optional. Each one requires a separate section.
Curriculum Map
NOTE: A Legend and an outcome component are required to add a Curriculum Map.
1. Click New at the top of the template.
2. Click Curriculum Map component .
3. Click the pencil/edit icon.
4. Choose which items from the Legend apply.
5. Click Save.
Assessment Map
The Curriculum Map and Assessment Measures components are required for an Author(s) to populate an Assessment Map.
1. Click New at the top of the template.
2. Edit Title of section
3. Click Assessment Map component .
4. Click the pencil/edit icon.
5. Choose which items from the Assessment Measures apply.
6. Select “Template Authors have the ability to define their own assessment measures” if you would like the Authors to write in their own measures if it is not included in the list provided by the administrator. For more information see the Author Guide, section Assessment Map or Assessment List.
7. Click Save.
Assessment Findings
NOTE: The Course/Event and an Assessment Map components are required for an Author(s) to populate an Assessment Findings section.
1. Click New at the top of the template.
2. Edit Title of section.
3. Click Assessment Findings component .
4. Click the pencil/edit icon.
5. Choose which items from the Improvement Types apply (Optional).
6. Click Save.
Assessment List
The Legend and an Assessment Measures components are required for an Author(s) to populate an Assessment List section.
1. Click New at the top of the template.
2. Click Assessment List component .
3. Click the pencil/edit icon.
7. Click the Select Outcomes button.
8. From the Set Library dropdown menu, select a standards library from which to search. Choices include All Sets, LiveText Sets, Institution, or My Favorite Sets
9. Once a Set Library is selected that entire group can be searched.
10. From the Set dropdown menu, select a standards set. Sets can be located by scrolling through the list or typing the first few characters of the Standard name (Ex: “HI” for Hawaii).
11. To search for a standard set by using the free text Search box, type in a word or phrase and then click Search. This field allows users to search by Identifier, Keyword, or Level.
10. Once the desired Standard Set has been located, check the box next to the Identifier. This will push the identifier to the right side of the screen and place it within the Selected column.
11. Click OK.
12. Choose which items from the Legend apply.
13. Choose which items from the Assessment Measures apply.
14. Select “Template Authors have the ability to define their own assessment measures” if you would like the Authors to write in their own measures if it is not included in the list provided by the administrator. For more information see the Author Guide, section Assessment Map or Assessment List.
15. Click Save.
Assessment List Findings
The Course/Event and an Assessment List components are required for an Author(s) to populate an Assessment List Findings section.
1. Click New at the top of the template.
2. Edit Title of section.
3. Click Assessment List Findings component.
4. Selected the desired Level for the Assessment Findings to be addressed (Assessment Measure, Standards/Outcomes, or Goal/Objective). At least one assessment finding level should be selected.
5. Select any Improvement Types for the Author’s use.
6. Click Save.
Improvement Type Summary
1. Click New at the top of the template.
2. Click Improvement Type Summary component .
3. Choose which items from the Improvement Types apply.
4. Click Save.
Adding Assessments/Rubrics
The Template Manager is responsible for adding assessment documents/rubrics to the template. Rubrics must be in the Template Manager’s account within the Documents tab. Rubrics can be added to the entire template for an overall review and/or to individual sections of the template. Rubrics are optional to the review process.
Attaching a rubric
1. Click in the search box at the bottom of the template for either the Template or an individual Section.
2. Click the title of rubric/assessment document to attach.
3. Optional: Multiple rubrics can be attached.
Template Distribution
1. Click on the AIS tab.
2. Click the Templates Link.
3. Click on the Distribute icon in the Distribute column of the template to disperse.
4. Choose the Programs/Units that will receive the template.
5. Choose an optional Due Date.
6. Choose a Reviewer Group.
7. Click Distribute.
Template Retraction
If a template has been distributed in error or is no longer needed to be populated by a specific Program/Unit, it can be retracted.
1. Click on the AIS tab.
2. Click the Templates Link.
3. Click on the Distribution Changes icon in the Distribute column of the template that has been dispersed.
4. The warning message will appear; “Retracting a template from a Program/Unit will delete any data entered by the Program Coordinator or Author of this template.” Click Ok to continue with retraction.
5. Deselect the Program/Unit that needs to be removed from having access to the template and click Ok.
Changing the Reviewer Group of Distributed Template
If a template has been distributed and the Reviewer Group selected was in error or is no longer needed to be part of the review process, it can be changed.
1. Click on the AIS tab.
2. Click the Templates link.
3. Click on the Distribution Changes icon in the Distribute column of the template that has been dispersed.
4. Make the desired adjustments for the listed Reviewer Group
5. Do not de-select the Program/Units as this will retract the template from the selected groups.
Template Clone (Cycle Continuity)
The Template Manager can clone a template. This will save time for the Programs/Units by importing data from the previous cycle. This will also present the Program/Units with a summary of their action plans from the previous cycle and give them the opportunity to make edits and revisions to the new plan.
Once the template is cloned, the Template Manager can make edits and additions to the plan. However, removal of any data field components will prevent the Program/Units from importing the previous cycle’s data for that specific field.
1. Click on the AIS tab.
2. Click the Templates Link.
3. Click on the Clone icon in the Action column of the template that is to be cloned.
a. Only Templates that have been distributed can be cloned.
4. Rename the cloned Template and click Ok.
5. The Clone Template will be added to the Template list as a Draft
6. The Template Manager can make edits or revisions to the cloned Template prior to distribution.
Template Copy
Copying a template means you are creating a new instance of a template based on the original. This copy will not be related to the original and will not allow Programs/Units to import information authored in a previous version.
1. Click on the AIS tab.
2. Click the Templates Link.
3. Click on the Copy icon in the Action column of the template that is to be copied.
4. Rename the copied Template and click Ok.
Notifications
After a template has been distributed to Programs/Units, the Administrator may make modifications to the distributed template to require additional content or information from the Programs/Units. A notification will appear for the Programs/Units that the template has been modified. Program/Units will also be notified when rubric scores and comments have been added or changed.
If a Template Manager attempts to distribute a template to the same Program/Units again they will receive the following warning message:
“You already distributed this template to program(s) College of Education. Distributing the template to these programs again will restrict reporting to the new instance of the template. Programs will not be able to copy over any previously authored content from the originally distributed version. Are you sure you want to proceed?”
AIS Manager Dashboard
Once the template/plan is distributed, the AIS Manager can utilize the Dashboard to track the progress of its completion.
Dashboard
1. Click on the AIS tab.
2. Click on Dashboard.
3. Select the desired Template from the dropdown menu.
The top section of the Dashboard will display the Template and Section Status of the plan with the corresponding key located at the bottom.
Template Status: The pie chart will display how much of the template has been Not Started, In Progress, Submitted, and Accepted.
Section Status: This bar graph will display how much of the template has been Not Started, In Progress, Submitted, and Accepted.
The bottom section of the Dashboard will give the AIS Manager visibility to the status of the individual sections of the plan.
Each program that the plan has been distributed to will display along with the Overall Status and the section status. The abbreviated key will reflect the current status.
Additionally, the Due Date column will allow the AIS Manager to email the Program/Unit a reminder of a potential upcoming due date.
Complete Review Status
Upon completing the review of an Author’s submission, the Reviewer will select Complete Review. This will prevent the Author from making any changes to the template. The AIS Administrator can reset this status which will allow Authors to make edits to the submitted work.
Changing the Complete Review Status
1. Click on the AIS tab.
2. Click on the identified Template and or Template Section.
3. Click on Unlock Review.
4. A confirmation message will appear. Click Update to approve action.
5. This will reset the Template Section status back to Submitted. This will not alter any of the information submitted by the Author(s) for the selected Template section.
Reports
Plan Content Report
The Template Manager has the ability to report on the plan content data. The Plan Content Report can be exported to both Microsoft Word and Excel. Each reporting type provides the Template Manager several filters in order to modify the data results. Below are the steps to follow to create Plan Content Reports in both Word and Excel formats.
Users that have been designated as Program Coordinators can also run the Plan Content Report for the Program/Units to which they have been assigned.
To run the Plan Content Report, follow these steps:
Word
1. Click on the AIS tab.
2. Click the Reports Link.
3. Click on New.
4. Select Plan Content Report as the Type.
5. Select Word as the Format.
6. Select the appropriate Template (more than one Template can be incorperated into the report).
7. Select either Program or Section in the Group By section (this will act by the primary grouping component of the report).
8. Select All or specific Programs.
9. Select All or specific Sections.
Note: For Programs and Section selections, the information will appear in the report in the order in which they were selected.
10. Create a Title of the report.
11. Add report Description (optional).
12. Select Save, Save & Export, or Cancel.
a) Save will retain the selected filters and return the user to the main Reports tab.
b) Save & Export will retain the selected filters and export the data out in the elected format.
c) Cancel will abandon the reporting activity.
13. When Save & Export is selected the downloaded report will be available directly above the action buttons.
Excel
1. Click on the AIS tab.
2. Click the Reports Link.
3. Click on New.
4. Select Plan Content Report as the Type.
5. Select Excel as the Format.
6. Select the appriopriate Template.
7. Select All or specific Programs.
8. Select All or specific Sections.
9. Select All or specific Mapping Component.
10. Select All or specific Custom Components (fields created by Template Manager within the Template).
Note: For Programs and Section selections, the information will appear in the report in the order in which they were selected.
11. Create a Title of the report.
12. Add report Description (optional).
13. Select Save, Save & Export, or Cancel.
a) Save will retain the selected filters and return the user to main Reports tab.
b) Save & Export will retain the selected filters and export the data out in the elected format.
c) Cancel will abandon the reporting activity.
d) When Save & Export is selected the downloaded report will be available directly above the action buttons.
NOTE: Attachments provided by an Author can be exported to a zip file.
Outcome Coverage Report
The Outcome Coverage Report showcases whether or not sets of outcomes are being covered by programs. The report will allow an Administrator to determine the number of times an outcome is being covered (aligned or associated with a goal or assessment) and also whether there are gaps in coverage.
To run the Outcome Coverage Report, follow these steps:
1. Click on the AIS tab.
2. Click the Reports Link.
3. Click on New.
4. Select Outcomes Coverage Report as the Type.
5. Select the appriopriate Template.
6. Select All or specific Outcomes.
7. Select All or specific Programs/Units.
8. Create a Title of the report.
9. Add report Description (optional).
10. Select Save, Save & View, or Cancel.
a) Save will retain the selected filters and return the user to main Reports tab.
b) Save & View will retain the selected filters and display the data on screen.
c) Cancel will abandon the reporting activity.
11. When the Save & View option is selected, a table displaying the results will be displayed.
12. After the report has been generated, the Export option will become active. This will allow the data table to be exported out into Excel.
Rubric Assessment Data Export
The Rubric Assessment Data Export (Excel) will provide the AIS Administrator the rubric scores and the comments as entered by the Reviewers. The export is only accessible by accounts designated as AIS Administrators and will output to Excel only.
To run the Rubric Assessment Data Export, follow these steps:
1. Click on the AIS tab.
2. Click the Reports Link.
3. Click on Export Rubric Assessment Data
4. Select the appropriate Template from the dropdown menu
5. Update the filename for the exported excel file (optional)
6. Click on Yes to retrieve the export
