Predict Troubleshooting

The following instructions provide the initial steps for troubleshooting missing and/or incorrect risk level indicators displayed in Student Success & Engagement from Predict.

Overview

By design, the risk indicators displayed in SS&E are only associated with certain students/sub-populations.

For each student included in the Predict Analytics Student population, the Persistence risk level displays on the left, and the Course Completion risk level displays on the right.

Each risk level displays either high (red), medium (yellow), or low (green).

• The student sub-population suppressions are usually defined during Predict implementation and are very rarely changed.
• Each institution defines its own scope. This includes defining the student population and dates used for Predict risk models.
• By default, the scope is included in each institutions' Predict Executive Summary.
• If the SS&E data associated with the risk level looks incorrect:
  1. Identify the proper data feed from the troubleshooting steps below.
  2. From SS&E Administration -SIS Integration - Datafeeds, search the Imported Data View in order to identify the source system id associated with the "incorrect" data displayed on the UI.
  3. Download the data extract from the Datafeeds screen. For instructions on how to find/view a data extract Json file, click here.
  4. Search the data extract by the source system id from step 2 to check if the "incorrect" data is included in the data imported from the SIS. 
     • • If the data extract includes the correct data and it is not displayed in SS&E, check for import errors/is the data feed out of date? If not, open a ticket with Watermark SS&E Support.
       • If the data extract does not include the correct data, this must be fixed in the SIS/Data Extract definitions.

Troubleshooting Steps

1. Before troubleshooting begins, the scope of the Predict risk models must be identified.
   • Users should reach out to their SS&E Administrator and/or open a ticket with Watermark SS&E Support requesting the institution's Predict Suppressed Student Sub-Populations if it is not readily on-hand.
2. Once the suppressed students are identified, the next steps are to find the students that are missing analytics in SS&E and check if they are included in the suppressed students.
3. For each student that is missing risk indicators or displaying incorrect risk indicators in SS&E, review the following areas to check if the student is/was recently included in the suppressed population.
4. In addition, students will not have risk scores displayed in Student Success & Engagement if a staff role is present on their Person record along with a student role, or if the student was identified as a Graduate student which we have never included in the scope of the predictive models. 
    
   Note: Students not included in scoring include NDS (non degree seeking) or students with a missing program in addition to students identified as Masters level students. 

Course Completion Risk Troubleshooting 

1. 1. 1. On the Student Profile tab:
         • • • Are there any tags that identify a suppressed population?
               • • Student tags are either assigned by an automated tag filter or the SIS via the Person Tag data feed (Data-backed tags).
                 • If the student tags look incorrect, check the tag filter or the data feed extract.
                 • If the student tag is incorrect, the SIS data-backed tag source data or the student data included in the automated tag filter definition must be fixed.
             • Is the student’s Campus associated with a suppressed population?
               • • The student's campus is imported on the Person data feed as the Location Id. (Primary campus a student is affiliated with, must match Location.srcSystemId).
                 • If the campus seems incorrect, download the Person data extract from the Datafeeds menu then search by the person Id to verify the student's person record/campus location imported from the SIS.
                 • If the data extract is incorrect, make sure the data feeds are current and no import errors exist, then fix the SIS source data and/or data extract definitions so that the correct campus data updates in SS&E
             • What are the student’s program, degree, and intent; is the student degree-seeking or otherwise excluded?
               • • The program/degree data displayed on the Student Profile tab imports from the SIS on the Transcript data feed.
                 • If the data looks incorrect, download the Transcript data extract and search by the Person Id to see the data imported from the SIS.
                 • If the data extract is incorrect, make sure the data feeds are current and no import errors exist, then fix the SIS source data and/or data extract definitions so that the correct transcript data updates in SS&E.
      2. On the student Courses tab - is the student registered in the current term? Was the student associated with the current term in the past, i.e. is the current term dropped?
         • • • By design, course risk indicators only display when the student has registrations in current and/or future terms.
             • If the student is not currently registered in a current or future term, their course completion risk indicator will either display a question mark or nothing.
             • The student's course data imports from the SIS on the Transcript Course data feed.
             • If the data looks incorrect, download the Transcript Course data extract and search by the Person Id to see the data imported from the SIS.
             • If the data extract is incorrect, make sure the data feeds are current and no import errors exist, then fix the SIS source data and/or data extract definitions so that the correct Student enrollment course data updates in SS&E.
      3. On the student Programs tab - did the student change their degree/program from an excluded population?
         • • • If yes, this may indicate why the student analytics were "skipped" and it must escalate to our L2 Data Science team to check why the analytics data is incorrect.
             • The student's program data imports from the SIS on the Person Degree Program data feed.
             • If the data looks incorrect, download the Person Degree Program data extract and search by the Person Id to see the data imported from the SIS.
             • If the data extract is incorrect, make sure the data feeds are current and no import errors exist, then fix the SIS source data and/or data extract definitions so that the correct Program data updates in SS&E.
      4. On the student Transcript tab - scroll to check if the student was recently a transfer student?
         • • • If yes, this may indicate why the student analytics were "missed" and it must escalate to our Data Science team to check why the analytics data is incorrect.
             • The transcript data displayed on the Transcript tab imports from the SIS on the Transcript data feed.
             • If the data looks incorrect, download the Transcript data extract and search by the Person Id to see the data imported from the SIS.
             • If the data extract is incorrect, make sure the data feeds are current and no import errors exist, then fix the SIS source data and/or data extract definitions so that the correct transcript data updates in SS&E.
      5. On the Student Filter - you can quickly check the scope of an issue by running a filter. From the Student Information section, create a new filter scoped to “everyone” for the current Registration Term and all Course Risk Levels (Low/Med/High) to check if most students are missing analytics. If yes, this could indicate a wide spread issue.
         • • • If many results are missing from the filter results, there may be a delay/communication issue. If the issue persists, this should escalate to our Data Science team to check why the analytics data isn't updating in a timely manner.
             • Running a filter will show the scope of the issue. To see which students are assigned or missing a risk level, you can run the Enrollment Management -> Enrollment report for more details.
      6. Is there a timing issue between the Predict data processing and the SIS Import job?
         • • • Predict data processing begins once a day at 4 pm.
             • The first step pulls data from SS&E, and the data processing takes up to 12 hours to complete. When completed, Predict Analytics import into SS&E. 
             • The Import job shows when Predict data was last updated in SS&E.
             • If SIS source data was not present in SS&E when data was pulled into Predict, the associated analytics will not display in SS&E until the following day.
             • The “Import Analytics File Job” can be monitored from SS&E Administration - Advanced - Batch Processing, found here. If the job is in stopped or stopping status, open a ticket with Watermark SS&E Support.

Persistence Risk Troubleshooting

Persistence Risk Levels only display for the most recent registered academic term.

• By design, past persistence scores expire at the end of a completed academic term so that the student does not display a persistence risk from a long time ago.

The most common question asked about Persistence risk indicators is why a question mark is displayed? 

• A questions mark displays on the Persistence Risk indicator when the academic calendar term ends, and the student is not registered in a future academic term.
  • • If the current academic term has not yet ended, and we are within 14 days of the next academic term, then the persistence risk changes to display for the next academic calendar term.
• If the student is not registered in the next academic calendar term, will the question mark display on the student record 14 days before the next academic term start date, or will it only display after the current term is completed and "past" persistence scores are removed/expired?

Next Steps

If you are unable to identify why students are missing their risk model analytics, please reach out to Watermark SS&E Support including all findings from the above troubleshooting steps.

For the next steps, the provided information will be verified and then escalated to the Data Science team for a resolution.