SIS Integration Overview and Requirements

Overview

Data is extracted from the SIS and transmitted to the Student Success & Engagement application using several data feeds. Student Success & Engagement accepts the data using one of two methods: via CSV files using Apache Drill, or via a Direct Database Connection (using queries within SS&E Accelerate or SQL Views). See the Connect Overview and Import Process articles for requirements and additional information.

The individual data feeds must follow the format requirements outlined in the SIS Import Definitions. Data feeds may be configured and customized using the Accelerate Configuration in order to follow the import definitions. Some data feeds support data deletion via the allIds feature. See the Data Feed Actions for a list of feeds that support data deletion, and SIS Data Deletion for more detailed information.

SIS Integration and Data Feed Implementation Process

Before beginning the SIS integration process, the Connect Server setup must be completed by the institution. The Connect Overview lists the server requirements.

The SIS Import Definitions article defines the import requirements for each data feed.

1. Decide on Connect Method: CSV or Direct DB based on SIS
2. Confirm data feeds to import in Data Sheet
3. Identify Sample Students
4. Build Priority 1-4 Data Feeds (required to move to Phase 2: Design & Build)
5. Build Priority 5 Optional Data Feeds
6. Data Verification 1
7. Data Verification 2

Building Data Feeds

In preparation for building the data feeds, the Implementation Project Manager will share a Data Feed Status workbook to keep the project team aligned on the progress of the data feeds. A Data Integration ticket will also be created for communication about data integration, questions, notes, etc.

Institution Admin and Data Lead:

• Using Data Feed
  • Verify which of the optional feeds the institution plans to implement.
  • Update "Using Data Feed" column. Enter "N" for data feeds that won't be used. Note: Some feeds are marked "N" by default as they are only used for specific modules or features. Change to "Y" if necessary.

Data Lead:

• Query/CSV Built  
  • Create SQL Queries/Views or CSV files following SIS import definitions. 
    Note: Sample queries/reports may be provided by Watermark (if available for the institution's SIS). These samples should be used as a starting point or reference, but as each institution is different, each data feed must be customized by the institution to accurately pull the necessary data. 
  • Note questions/differences in Data Feed Status sheet and Data Integration ticket for review.
  • Update “Query/CSV Built” column with “Y” once the View/file is ready to be reviewed.
  • Update Data Integration ticket with questions or when progress has been made and feeds are ready for Watermark to review.

Watermark Technical Staff:

• Accelerate/Connect 2 Runs
  • If applicable, verify that the data in the CSV file is ready for Accelerate.
  • For Connect Method 1: Create query in Accelerate config (if available, use parent or copy sample query) and make adjustments based on data in CSV file.
  • Verify that Connect/Accelerate has run successfully: there is data in the JSON extract (SS&E Admin > SIS Integration > Datafeeds > Download Extract) after Connect has run and no failures/major errors have occurred in log (SS&E Admin > SIS Integration > Datafeeds > View Logs OR SS&E Admin > Advanced > External Logging).
  • Update “Runs” column with “Y” once the query is running.
• Sanity Check
  • Verify that the data in the JSON extract (SS&E Admin > SIS Integration > Datafeeds > Download Extract) looks reasonable at a glance: each field appears to have the right kind of data in the correct format, the amount of data seems reasonable, etc..
  • Update “Sanity Check” column with “Y” once the data in the JSON extract looks reasonable.
• Importing into SS&E Production
  • Enable import for data feed (switch Enabled flag to yes in SS&E Admin > SIS Integration > Datafeeds OR add to list of restricted streams in SS&E Admin > SIS Integration > Manual API Calls).
  • Verify that the feed has imported successfully into the SS&E DB (SS&E Admin > SIS Integration > Datafeeds > Imported Data > View). There are no major import errors (SS&E Admin > SIS Integration > Import Errors).
  • Update “In Production” column with “Y” once data is importing without major errors.
  • Update Data Integration ticket with information as needed.

Data Verification

After all data feeds are in production, Watermark staff will provide the institution with two Data Verification worksheets. This is a worksheet that is pulled from the Student Success & Engagement database using the list of sample students and advisors provided by the institution. It contains many data points which can be used to identify patterns of missing data, inconsistencies, or other errors.

Institution Admin and Data Lead:

• Identify sample students and advisors for verification.

Watermark Technical Staff:

• Provide data verification sheet.

Institution Admin and Data Lead:

• Data Verified
  • For each sample student, verify that the data imported into SS&E is accurate and complete by comparing the data verification sheet with the source data in the SIS and LMS.
    • Note: the verification is static and accurate as of the day it was sent, but will not reflect future changes made in the SIS/LMS or in SS&E. If something seems out of date, first verify that the live data in SS&E matches the SIS/LMS. Allow 24h for new changes to flow from the SIS/LMS into SS&E.
  • For each sample student and sample advisor, review all profile tabs in SS&E to confirm accuracy.
  • Work together to identify and resolve any inaccuracies in the queries/reports. It is recommended to focus on patterns of missing/inconsistent data. 
    • Specifically, take note if any one field is empty or wrong for all records. If the institution has the data in the SIS, it should be included in the data feeds, even if not explicitly required for import. There may be specific exceptions to this, but as a rule institutions should include all available data, even optional fields. This ensures that all functionality and features are available to users.
  • If the institution is unable to find a resolution to a data issue or inconsistency, they should reach out to Watermark staff for support via Data Integration ticket. Changes may be made to the Accelerate configuration or other solutions may be explored together.
  • Update “Data Verified” column with “Y” once data is correct in SS&E.

Roles and Responsibilities

The institution's Data Lead is primarily responsible for the creation and maintenance of all data feeds. 

Watermark Technical Staff may provide support, however the data is ultimately owned by the institution and can often only be changed by the institution's IT team. The institution should first try to solve any data feed inconsistencies internally between the SS&E Admin and the Data Lead, and reach out to Watermark staff for additional support if needed.