Data Stream Out of Date Troubleshooting

Updated June 13, 2024 18:31

Watermark Support

Overview

Who is notified?

When SS&E detects that a data extract(s) is out of date, the system will notify SS&E Administrators by displaying a warning in the SS&E Administration and notifying the SIS Health Notification email address set in Institution Administration.

What does this mean?

When a data feed(s) is out of date, it means that data is delayed and can explain why data that you expect to see in SS&E is missing or incorrect.

When data feed(s) are out of date, for some reason, the Connect job was unable to successfully generate a corresponding data extract/.Json file to import new data into SS&E so the data in SS&E is most likely getting "stale" as it has not been recently refreshed.

When a data stream is out of date, this is discovered during the Connect Job that runs by default four times a day (at 2/8 AM/PM) as a scheduled task on the Connect server before the SIS Import Job runs and imports the SIS data into SS&E.
For this reason, all errors related to a data feed that is out of date will appear in External Logging each time the Connect job runs and tries to process the data feed. This usually results in the same error displaying on every data feed that is out of date four times a day starting when the underlying issue began until it is resolved.
Whether discovered in SS&E Administration or by receiving an email, when a data stream is out of date it means that the corresponding data extract has not been processed successfully by the Connect job (and delivered to the cloud) in the past 18 hours.
When the SIS data extract delivery is delayed, this means that Student Success & Engagement has not been updated recently and is likely becoming less effective.

Who can help resolve data feeds out of date?

The following article includes troubleshooting steps to help resolve data feeds out of date.

When all data feeds are out of date, this is most likely related to something that happened on the Connect server that is preventing the Connect job from running and/or completing successfully.

If only one/some data feeds are out of date, this is most likely related to something that happened in the source data that is preventing the Connect job from successfully processing the data feed(s).

The first place to look for root cause why one/some/all data feeds are out of date is in External Logging, accessed from SS&E Administration > Advanced > External Logging.
Filtering on Log Level "Error" and clicking into the error message details will most likely show why the data feed(s) is out of date.
Once the underlying issue is resolved, the data feed will stop showing as out of date and the data feed errors will stop displaying in External Logging.

Before reaching out to SS&E Support, it is recommended to ask your local data lead/IT person familiar with the SS&E SIS Data Integration Connect Job to check if they can resolve the errors displayed in External Logging.

Depending on the error message details, local IT staff can most likely help check for issues related to the Connect server/disk space/Connect job/ scheduled task properties/source data file/formatting etc. and fix the underlying issue(s) without further assistance.

For example, if the Connect job is not running, IT can manually start the Connect job task and monitor the job to verify that it is completing or check/fix why it is not running properly.
If data feed(s) out of date cannot be resolved by the institutional data lead/IT, please include your data lead on the support ticket to help check the Connect job with guidance from SS&E Support.

For more information about the SIS Data Import process and infrastructure, click here.

Are all data feeds out of date or just one/some?

If all data feeds are out of date - this usually indicates a network /infrastructure /service /user or permissions related issue that occurred on the Connect server. Once the underlying issue is identified and resolved, this will most likely fix all the data feeds that are out of date.
If only one or some data feeds are out of date - this can be caused by many different reasons including a missing file, "bad data", "bad format", Parse Error <EOF>, incorrect syntax, low disk space. a memory issue, or other issues found in the data extract definitions, CSV file or Connect job.

Troubleshooting Steps

Filter External Logging on Log Level "Error" and the "Connect" source to check for recent errors.

From SS&E Administration > Advanced > External Logging.
Select Log Level "Error"
Select Source "Connect2"
After selecting Log Level "Error" and Source "Connect", check for errors.
Click on any error records to view the error message details.
In most cases, resolving an error will fix the data feeds out of date.
- To check when the error started, scroll to the first time the data stream error appears or add a filter condition for the data stream associated with the error, then click into earlier error record(s) to verify when the issue described in the error details first appeared.
- Most likely, the error will need to be resolved in the data extract definitions/on the Connect server.

Once the underlying issue is resolved, verify that the error stops displaying in External Logging and the data feeds are no longer out of date.

If there are no errors in External Logging, and there are data feed(s) out of date, please check some additional items and open a Support ticket for further assistance if needed.

If you know which data feed is causing a problem, filter on log level "Error" and the "Connect" source, along with the data feed name that is out of date/causing a problem during the Connect job.
Most likely, clicking into the error records will allow you to verify when the issue started and show error message details that will help figure out why the Connect job is unable to process a specific data feed(s).
Once the underlying issue is resolved, the External Logging error will stop, which will fix the data feed so that it's no longer out of date and data can import properly into SS&E.

External Logging Search Tips

Always verify in SS&E Administration that the data feed(s) are still out date, as this may have been the result of a temporary issue that has already been resolved.
Once confirmed that data feed(s) are still out of date, the first place to check for information about the underlying reason causing data streams to be out of date is in SS&E Administration > Advanced > External Logging further explained below.

Filter on Log Level "Error" to check for error messages ocurring on data feed(s) during the Connect job. Click into the error record to view the error message details.
Filter on Source "Connect2" to only view error messages from the Connect job.
Search for "Running SIS import" to see when the Connect job has recently started.
Search for "Finished SIS import" to see when the Connect job has recently completed.
- If the Connect 2 job is not running on schedule, go to Advanced Troubleshooting - Step 5.
Sort on the Log Date to view the most recent entry.
- If the Connect 2 job is not running on schedule, go to Advanced Troubleshooting - Step 5.
For more information about the Connect job processing, download the Connect 2 Full Log from the top right corner.
- Look at the beginning and the end of the file to view the most recent log dates.
- If the job is not running on schedule, go to Advanced Troubleshooting - Step 5.
- Search the log for additional information displaying immediately before/after the error message details found in step 1.

Common Errors seen in External Logging

Null Stream

If the Connect Job cannot generate a data extract (.json) file for a select data feed, either because data does not exist in the source system or the file received from the SIS is empty, then an error or a warning message will usually display in External Logging for a "Null Stream".

Searching by "Null stream" in External Logging will show all data feeds that are empty.
To resolve this, verify if the steam is expected to be empty and not in use. Once verified, the data feed should be disabled so that the system does not generate a warning message each time the SIS Import job runs.
If there should be data importing from the SIS for this data feed, then for some reason something changed on the Connect server indicating that the source data is currently missing. To resolve this, this needs to be investigated on the Connect server by the institutional IT staff to check what changed and why the data is suddenly missing.

Failed to Fetch Status: The specified key does not exist

In the Administration warning drop down and email notification, a message may include "Failed to fetch status: The specified key does not exist." when the data is missing.

To resolve this message, one of two things must happen:

If the specific data feed is not in use, it can be disabled in Datafeeds Administration.
- This will remove the data feed out-of-date warning since the feed is not being used to import data from the source system and there is no equivalent data importing from the SIS.
If there should be data importing from the SIS for this data feed, then for some reason something changed on the Connect server indicating that the source data is currently missing. To resolve this, this needs to be investigated on the Connect server by the institutional IT staff to check what changed and why the data is suddenly missing.

SYSTEM ERROR: IO Exception: There is not enough space on the disk

This error indicates an input/output memory issue on the Connect server.

To resolve this, institutional IT can usually check the Connect server to resolve this message by checking/clearing up disk space.
Once the underlying issue is resolved, the data should import successfully.

Drill Related Messages

For Drill service troubleshooting steps, click here.

Administration

To view/edit your institution's SIS Health Notification Email Address, click here.

To view/edit the frequency of SIS Health Status Check notifications, click here.
- - By default, SIS Health Status Notifications are sent every 24 hours, i.e. sent out once a day, yet this can be configured to send at a different frequency.

sisHealthCheckFrequencyHours	24	How many hours to wait between resending health check emails

Additional Information - External Logging

From Advanced Administration, search for error messages that occurred during the Connect Job by filtering on Log Level "ERROR", Source "Connect2" in External Logging.

1. Click into error results to view the error message details.
2. Click into past errors to see exactly when the current error/issue started.
3. In most cases, External Logging will display error message details that help explain why the Connect job is not running properly/why data feed(s) are out of date, and how to fix this.
4. If there are no relevant errors, filter on log level "WARN" to search for Null data feeds that are out of date.
  - A Null data stream indicates that the Connect Job cannot generate a .Json file for the data feed, either because data does not exist in the source system or the file received from the source system is empty.
  - Null streams may cause data feeds to appear out of date, however once this is verified, these out of date data streams can be ignored or the data feed can be disabled when not in use.
5. For more information about External Logging, click here.

When a data extract is out of date on S3, the SIS Import Job will still "import" data into SS&E every time the SIS Import Job runs unless there is an SIS Integration Import Error(s) preventing the most recent SIS data "on-file" from importing into SS&E.
SS&E does not have a mechanism that will check the actual "contents" of the data extract/ json file importing into SS&E. Therefore, when a "Data Stream is Out of Date" warning appears, this indicates that the SIS Import Job is importing an "older" data extract on file. However, this does not actually check what the data extract file contents are.
By default, the SIS Import Job runs every six hours.
If a new SIS data extract/json file is uploading successfully every time the Connect job runs as a scheduled task on the Connect server, yet the contents generated from the SIS/Informer/Data Extract Definitions are "not current", this will not appear as a data feed out of date.
- - - If the data received from the SIS is not up to date and the data extract is not out of date, then the data extract file contents will match the data displayed in SS&E, however, this will not match the data displayed in the SIS. To fix this, the SIS/Informer reports/Views/CSV files must include "current" data that subsequently upload successfully to S3 and import successfully to SS&E.

Data Stream Out of Date Validation

Before troubleshooting why data streams are out of date, we recommend verifying that this is still the case.

When a data feed(s) out of date is transient and brief, and a notification email was already delivered, this may cause confusion.
In Administration, check for a warning symbol on the top right. Click to see which data streams are out of date.
If there is no red warning then it was resolved and no further action is necessary.
For data feeds that sync data deletion with the SIS, if the data feed was out of date and the data deletion job has not yet run overnight, the data will still display incorrectly in SS&E. Since the data deletion job only runs once a day at 2 am, wait for the next morning to check if SS&E is correct.

Advanced Troubleshooting

When a data stream is out of date, the first step is to identify the root cause for the delay in generating the data extract (Json file(s) on S3).

Following are some of the most likely reasons why a data stream is out of date/delayed:

1. Is there a data stream that is out of date and not in use?
  - If a data stream is not in use yet is marked as enabled in SS&E, then it should be disabled.
  - Once disabled, SS&E will stop importing this data feed and will no longer display warning messages for this specific data feed as being out-of-date.
  - To disable a data stream, click here to find the relevant data stream from Administration -> SIS Integration -> Datafeeds, then click on the data feed name and uncheck "Enabled".
2. Are only one/some data feeds out of date? "An error has occurred on one or more of the data streams"
  - A common indicator for a data format extraction issue is when only one or a few data streams are out of date while other/most data extracts are successfully importing current data.
  - When the data feeds are updating inconsistently, a rare occurrence is when Java "clutter" on the Connect server causes the Connect job to freeze at random points during the "export/upload" process. In most cases, once identified, this can be resolved by installing a new JDK and increasing the max java memory available for the process. For more details, please see the Troubleshooting Tools - Java Update section below.
  - Something that may cause one data extract to be out of date is if Informer is crashing on the data report with a memory error, this can cause a specific data feed to not update with current data. To fix this, increasing the memory on the network/infrastructure will allow the Informer report to complete.
  - In most cases, when only some (i.e. not all) data feeds are out of date, there are SIS/CSV data/format or infrastructure changes that may result in an invalid query. For more details, please check the Troubleshooting Tools - External Logging steps listed below.
  - For example, if the error message found in External Logging and/or the Connect 2 Full Log is an ORA error or a SQL Server error code, this indicates an error in the SIS database. The associated error message details will usually describe what exactly is triggering the error and why the data extract is out of date. If the error message details found in External Logging aren't clear, then internet searching on the error will usually help find additional details.
  - Data Extract errors can also be investigated by manually running the data extract queries on the SIS data and then evaluating the results. This is further explained in the Troubleshooting Tools - Manual Querying section below.
  - The data extract error may reference an error related to the tempdb used by the extract query. This type of error is usually self-explanatory. For example, "Could not allocate space for object xxx in database 'tempdb' because the 'PRIMARY' filegroup is full" sounds like it would be resolved by creating disk space/deleting unneeded files/dropping objects/adding additional files/enabling autogrowth to existing files in the filegroup. The next time the Connect job runs it is easy to verify if whatever was done resolved the error/it went away.
3. Has there been an infrastructure change?
  - If the SIS Import jobs are completing successfully yet all data streams are out of date, this may indicate an infrastructure/network change causing the Connect job to "fail/break".
  - To confirm the status of the SIS Import Jobs, click here.
  - For a quick resolution attempt, when all data streams are out of date, rebooting the Connect server may fix everything since a server reboot can help "reset" an underlying issue.
  - If all data feeds are out of date on S3 and the Connect Job "Last Run Time" is current, i.e. the Connect job is running on schedule without any issues on the Connect server, then most likely there is something preventing the communication between the Connect server and Amazon AWS (S3).
    - - Check Event Viewer on the Connect server. Are there any messages in the server logs?
      - Is there is a firewall blocking the host resolution/connection?
      - Is there an issue with the DNS resolution?
      - Click here for a complete list of the Connect2 API URL endpoints.
4. The data extract query runs successfully yet the resulting data set is empty -
  - When the data extract query completes successfully and results in no data, there is usually an error message related to missing data ("No data for stream") displayed in External Logging and/or Import Errors.
  - For example, if the SIS/Informer/View/CSV file used to generate the data extract file on S3 is missing all data, then the reason the data extract Json file is out of date is either that the file is empty (has no data) or that the file is missing entirely.
  - To resolve this, the reason why there is no data/the file is empty or missing must be identified.
    - - In some cases, an empty or missing file may be the "expected" result, and the data feed out-of-date message is not relevant.
      - If we expect data, then the first place to check is the extract queries to search for a join/table where the CSV is not matching up as expected. For example, if person ids are missing a leading 0, this will unintentionally cause all data to be filtered out resulting in an empty data set, generating an empty data set/ "missing data" data extract.
  - Manually running the extract queries (explained below) will usually find what is happening since evaluating the results will show exactly where there is no data being returned.
5. Is the Scheduled Task running correctly?
  - From External Logging, make sure that the Connect job is running on schedule by checking the most recent log date(s) associated with Log Level - Info and the Application Log.
    - - When date-times are current, it means that the Connect Job is running on schedule.
      - When date-times displayed in External Logging and the Application Log are old, it indicates that the Connect Job/Scheduled Task is not running properly.
      - Scroll down for more details about using External Logging.
  - Only clients on Accelerate/Connect 2 can view the most recent Application Log generated on the Connect server by clicking on "Connect 2 Full Log" which downloads this to your local pc.
  - If the most recent Log Date isn't current, log on to the Connect server and open Windows Task Scheduler to check the Connect Job Scheduled Task (Connect and/or Connect 2).
  - From Windows Start, search for the Task Scheduler application, then find the Connect Job and check the task status, triggers, and last run time. Click on the task name to check all tabs for anything that stands out or errors.
  - "Task Scheduler: the operator or administrator has refused the request" can be permissions related or if there is another job running/stuck. To fix this stop the running task or let it finish. To prevent future occurrences, check the following task setting in the Task Properties "If the task is already running then the following rule applies - Stop the existing instance".
  - Check the Connect Job History tab and Windows Event Viewer logs to search for error messages that identify why the Connect task is no longer running on schedule.
    - - Search for log messages that show maintenance or permissions-related issues.
      - For example, "Task Scheduler failed to start " or "Connect task for user xyz" indicates a permissions-related issue for the user that is assigned to run the scheduled task.
        
        Is the password for the SS&E (Aviso) user running the Connect Job expired? To fix this, we recommend setting the Connect task user account password to never expire or changing the user account to a service account on the Connect server, and validating all the task security options so that this does not reoccur.
        
        Does the Connect task user have the proper permissions and group policies to run the scheduled task? To fix this, check the Connect task user, the user/group policy, and scheduled task properties. Verify that the proper user is assigned to the Connect job. When applicable, check the "Log on as service" policy to make sure it is not being revoked from the SS&E task user.

- - - - Something else that may prevent the Connect job from running in the Windows task scheduler is when the security settings in the AD group policy are updated to prevent PowerShell from running. When this is the case, there is a message stating "This program is blocked by group policy" and the job will run manually (by right-clicking on the Connect job and selecting "Run"), yet will not run from the task scheduler.
        
        To fix this, PowerShell must be allowed to run on the Connect server so that the Connect job process will run on schedule and update SS&E properly.

- - - - Once resolved, the scheduled task will run on schedule and generate the data extract/json files on S3 each time it completes.
      - For data to stay current in SS&E, the scheduled task/Connect job must run and complete successfully since it is what generates the data extract files on S3 that are used by the SIS Import Job when importing data into SS&E.

6. For CSV + Accelerate(Connect2) sites only -

All data feeds are out of date and every data stream has an error "A dependent service is unavailable" or "Failure in connecting to Drill"

1. 1. Check External Logging - Filter on Log level "Error" and Source "Connect2".
    - When there is an issue related to Drill, every data feed will have an error message that includes "Failure connecting to Drill..", "...Connection Refused..", or "a dependent service is unavailable" in the error message details.
    - The Drill service runs on the Connect server and is used for CSV and Accelerate customers when reading the CSV files via a SQL interface.
    - If the Drill service is unavailable, then most likely ALL data extract queries are failing.
  2. If there is a drill service related error message in SS&E Administration > Advanced > External Logging, check for more details about this on the Connect server.
    1. After logging into the Connect server, check the Windows Services and/or the Windows Event Viewer to check if the Drill service is/isn't running successfully and look for errors.
    2. Check the Task Manger Processes to see if there is an issue with the Connect server CPU resources.
    3. Check the C: drive disk space. If the hard drive is full/almost full, the drill service may not be able to function correctly. Check the C:\tmp\drill\profiles folder which contains files created by queries run in Apache Drill that can be removed.
    4. If there is an issue with the Connect scheduled task, it will most likely display a message that references why the Drill service is not running or "connection refused".
      - To test the drill service console, after logging on to the Connect server, navigate to http://localhost:8047. If this does not load, it indicates that the Drill service is currently not running.
    5. To fix the drill service, check the service user permissions and verify that the drill user permissions are configured correctly (verify the correct username/password set to never expire) and then restart the service.
    6. To fix the connect server resources, check for errors on other services or applications with high CPU usage that may be causing drill to drop connections.
  3. For more detailed steps and information about troubleshooting Drill, click here.

Troubleshooting Tools

Connector Job Troubleshooting

For more information about SFTP Connector Troubleshooting, click here.

External Logging

Click here to open External Logging in your SS&E application.
If the screen is blank, then you most likely have an older installation of Connect which does not include this feature. For more information about adding External Logging and upgrading Connect, click here.
When enabled, Administration - Advanced - External Logging displays messages encountered when the Connect Job/Scheduled Task is trying to upload the SIS data to S3.
- When External Logging displays current dates, this means that the Connect job is running and uploading data extract (.Json) files to S3.
- Filtering by Log Level "Error" will usually return results for the data feed(s) that are out of date. For example:
  - - If some/all data feeds are out of date and there are error messages stating that "SYSTEM ERROR: IOException: There is not enough space on the disk", it means that the Connect server is out of space.

- - - - When this message displays, it means that until the disk space issue is resolved, the SIS data displayed in SS&E will not be refreshed with current data since current files are not processing.
        
        Something to check is the Connect 2 logs stored in C:\Aviso\Connects\Logs. if this folder is taking up a lot of space, older logs may be removed from this folder.

- - - If all data feeds are Out of Date and External Logging shows the following error on every data feed: Error Message Details VALIDATION ERROR: Schema [[dfs, root]] is not valid with respect to either root schema or current default schema.
      - Googling this error message, it appears related to something that changed on the Connect server and is causing the drill service to to be unable to write files to S3.
        
        For additional resolution tips, click here.

- All Error Messages must be resolved in order to fix data feeds that are out-of-date.
Clients on Connect 2/Accelerate can also download the Application Log from the most recent Connect job that ran by clicking on "Connect 2 Full Log" from the top right. This button provides direct access to the SS&E "Aviso Application log" on the Connect server. and it includes all steps from the most recent Connect job that ran on the client Connect server.
- The application log will also display the last date-time the Connect job ran on the Connect server, so it should be current. If the application log dates are "old", this indicates that the Connect job is not running on schedule. Refer to step 5 for more details.
- When there is an error displayed in External Logging, or if nothing displays yet there is a problem, checking the application log may help shed additional information.
Modern versions of Connect that include External Logging also include an auto-updater task that prevents the need for SS&E staff to manually connect to the Connect server and update the Connector when there is a change.
For more information about External Logging and updating Connect, please click here.

Manual Querying - Accelerate Clients only

How to view and manually run the data extract queries displayed in SS&E:

For clients on Accelerate/Connect2, the Data Extract Queries that are being executed can be viewed in Administration from the data feeds screen found here.
Select "View" from the Extract Queries column to view the data extract queries.
Copy the extract queries and execute them in a database client on the SIS data.
- When using Accelerate with CSV files, a service called Drill is used to extract data from the SIS.
- To run queries against a CSV file, navigate to http://localhost:8047 and use the query window.
- To determine why the query isn't returning the expected data, run the query and then evaluate the source of the results.

Java Update

The following steps may be used by local IT staff to update Java. From experience, this is sometimes helpful when the data feeds are inconsistently and randomly out of date:

Uninstall all the Oracle Java JDKs and JREs that are currently installed on the Connect server.
Install the JDK that is used for new SS&E implementations which can be found at the following link*: https://corretto.aws/downloads/latest/amazon-corretto-8-x64-windows-jdk.msi

*This link should always be the most current, use the Windows x64 JDK MSI.