SIS Data Import Process and Infrastructure Summary

Import Process Overview

There are two methods for implementing Student Success & Engagement SIS Data Integration on the institution's Connect server.

  1. Connect Method 1 uses CSV files (often exported from the SIS via Informer or another custom export method) as the data source.
  2. Connect Method 2 is a direct database connection using SQL Views or Queries.

No matter which of the above methods is used by the institution to extract data from the SIS, the Connect2 (Accelerate) process SFTPs data from the institution's Connect server to the SS&E AWS S3 data store, optionally transforming the data to match the import requirements.

The SIS data is then imported from AWS S3 into the SS&E database via the SIS Import Batch Job so that the imported data becomes available on the UI and visible from the Datafeed Views.

The details outlined in this guide are subject to change as Watermark processes and requirements change over time.

Connect Method 1: CSV Files

CSV.png

Note: see downloadable image at end of article.

Connect Method 2: Direct Database Connection

Direct DB.png

Note: see downloadable image at end of article.

Infrastructure Summary

sse-infrastructure-summary.png

Additional Information

Schedule (for all Connect methods):

The Connect2 scheduled task is scheduled to run four times a day, so that SIS data can import up to four times a day, depending on how long it takes each import job to complete.

  • The Connect2 (Accelerate) job runs at 2 AM, 8 AM, 2 PM, 8 PM by default, but may be customized by the institution.
  • The SIS Import Job runs at 4:30 AM, 10:30 AM, 4:30 PM, 10:30 PM. This schedule may not be changed.

Accelerate XML Files

If anything needs to be changed, this is entered into the Child XML which is run first.

If code is not found in the Child XML, then the Parent XML (default Accelerate code) is used to process the data feed.

SIS Import Timing Issue

The Connect job will upload the SIS data to AWS S3 (in the cloud) from where it is imported into SS&E.

  • There can be a timing issue if the most recent SIS Import job ran before data was uploaded to the cloud. However, the next time the SIS Import job runs, since we verified that new data is included in the most recent data extract json file on S3, as long as there are no import errors/issues, the new data will soon appear in SS&E as expected.
  • If data was delayed, and did not display within SS&E as quickly as expected, the way to check on this is to check the different parts of the SIS Data Import Process to try and figure out what caused the data delay. For example:
        1. First, start by identifying the data feed that imported the delayed data.
        2. Ask the institution's IT Staff to check the Connect Server to make sure there were no recent infrastructure or network issues that may have caused a delay for the source data availablity via Informer/CSV or Direct DB SQL/Query.
        3. Search for the data feed in External Logging to verify that the Connect Job ran on schedule and generated the data extract on the cloud/S3. Check for any recent issues by viewing the Log Level/Log Dates/Log Messages. For more information about External Logging, click here.
        4. Verify that the SIS Import Batch Job ran and completed on schedule in Batch Processing. Drill down into recent jobs to search for the data feed and view the read/write record counts. If the count is zero that means that there was no new data found in the data extract imported from the cloud/S3 into SS&E.

Connect2 Job Logging

For information about Connect Job logging displayed in SS&E Administration Advanced External Logging, click here.

Import Logging

Import Errors can be found in SS&E Administration SIS Integration Import Errors, and Import logs can be found in SS&E Administration Advanced Batch Processing

Advanced Import Caching 

SSE stores a hash key in our cloud to detect differences in incoming data. If there are no differences in a record since the last time we encountered it, it will not reimport

Clearing cache(s) via SSE administration will cause a full data import to take place

Colleague/Informer Implementations

Informer schedules may be set at implementation, but each institution can customize/update the informer report schedules to export as often as they would like to fit their needs.

Articles in this section

See more
How to Contact Support
There are many ways to reach out! Click here for our support options.
Watermark Academy
Click to access the Watermark Academy for consultation, training, and implementation companion courses.
Customer Community
Can’t find the answer? Ask fellow users how they’re making the most of Watermark in our Community!