Overview

The SIS Data Deletion configuration is an integral part of the SIS Import Job for data streams that are set to synchronize data deletions from the SIS with SS&E.

Click here for a complete list of data feeds.

• "Delete = Yes" means that the data feed supports SIS Data Deletion integration.
• A few data feeds integrate data deletion by a different method. These data feeds are displayed on the Data Feed Overview with an Asterisk symbol (Delete=No*).
• This feature requires that the All Ids stream are first implemented and configured properly.
• Data removed via All Ids displays with a Status Id of Purged By Institution.

For each data feed that supports data deletion and was implemented to integrate data deletion, the SIS data deletion integration process:

1. Retrieves all of the known/active SIS Ids from the SIS (via All Ids).
2. Compares them to the source system ids associated with the corresponding data displayed in SS&E, e.g. active records found on the corresponding data feed view/in the SS&E database.
3. Removes from the SS&E User Interface all records that no longer exist in the SIS.

Attributes

• Only some data feeds support synchronized deletions in the SIS with SS&E. 
• The SIS Data Deletion job runs by default once every 24 hours at 2 am.
• Since the data deletion job runs only once a day, if there were data feeds out of date that sync deletion with the SIS, and they recently updated, verify that the most recent All Ids file is correct. If yes, wait for the next morning to check the SS&E UI. 
  • • If the most recent All Ids on file is correct (includes source system ids for data that should display in SS&E, does not include source system ids for data that should be removed from the UI) there may be a timing issue where the "correct/up to date" All Ids file was not used for the last data deletion process. Since the data deletion job syncs data deletion between SS&E and the SIS overnight, if the All Ids file is correct, everything should display correctly in SS&E the following morning.
• The SIS Data Deletion synchronization process is as follows:
  1. The system reads the All Ids file to gather a list of all the current/active source system ids that exist in the SIS.
  2. The system compares the list of All Ids with the active/current source system ids that exist in the SS&E database/display on the UI. 
  3. Active current ids that are currently active in SS&E yet are not included in the list of All Ids are marked as deleted in SS&E which removes them from the UI. 
• Most streams that support deletion will still show the deleted records on the Imported Data View with the last column displaying deleted=true. This value can also be filtered upon. For an example, see here.
• If the data feed view shows a record as deleted, and it currently exists in both the Extract file and the All Ids file, then most likely what happened is that in the past, the All Ids file did not contain the record source system id and therefore it was marked as deleted and removed from the UI. If the record reappears in the All Ids file, then it should no longer be deleted. If the data deletion job already ran and it is still marked as "deleted" in SS&E, this usually means that the reason it is not being reimported is that the system is reading from the data feed cache.
  • • This should happen very rarely, yet if it does, open a ticket with SS&E Support to clear the cache for the specific data stream including all details of the data not updating correctly.
    • Most likely, after the data feed cache is reset then the next time the SIS Import job runs it will bring in fresh data.

Note: Clearing the cache on a large data stream will cause the next SIS Import job to run longer.

• • • Alternatively, if only a few records are impacted, you can "delete/undelete" them directly from the data screen view. As long as the All Ids file contains the "correct" data/the same results, manual changes will not be reversed.
• To clarify, if a source system id is not included in All Ids, the record will be marked as deleted in SS&E. Once deleted=true, the record must be reimported from the SIS on the data feed, and the associated All Ids file must include the matching source system id so that the record is marked as deleted=false and displays in SS&E.
• Some data feeds do not allow past data deletion integration with the SIS. For example, Transcript Course and Transcript Course Attendance data can only be deleted if current or in the future.
  • • Both SIS Data Deletion Integration via All Ids and Manual Data Deletion via the Imported Data Feed View will not apply on "past" data. Removing past data from the UI will require an engagement with Professional Services to remove the data directly from the SS&E database.

Scheduled Job

The SIS Data Deletion job runs as a scheduled job once every 24 hours at 02:00:00 AM.

To view when the job last ran, search here for sisDeletionBatchJob.

Manual Data Deletion

Data streams that support deletion also support manual record deletion. Navigate to the data viewer via the Datafeeds screen. If the stream supports deletion, a "Delete" button will appear in the secondary navigation at the top of the screen. Select the records that you'd like to delete and click Delete.

Note: Manually deleting a record in SS&E Data Feeds Administration deletes/flags the record. However, it must also be removed from the data feed extract otherwise it will be reimported the next time that record is processed. Do not attempt to manually delete a record that still imports on the SIS data feed since it may reappear.

For more information about Manual Data Deletion, click here.

Troubleshooting:

To troubleshoot missing data associated with data feeds that integrate data deletion with the SIS, use the Administration - SIS Integration - Datafeeds screen:

1. If the imported data does not display in SS&E as you expect, check the Imported Data View found here. Search for the missing data and then check the Deleted column/are the results displayed in red. If yes, this means that the record is deleted=true, i.e. the record source system id is not included in the All Ids feed.
2. To verify that the data is missing/deleted=true because the source system id is missing from the All Ids file, download the All Ids data extract from the "Download Extract" column found here.

1. • Something important to verify is that the source system id is formatted properly and that all Person IDs are formatted properly in the source definitions.
     • • For example, in person_phone_address_ids.csv and person_phone_personal_ids.csv, the first column of both files is the Person ID number. If any/all Person ID number(s) are missing leading zeros, then the file will be empty/records will be missing. To correct this, including the leading zeros on the first column (Person ID) should take care of the issue.
          
   • To clarify, each Id listed in the All Ids data extract must be an exact match with the src_system_id displayed in SS&E on the associated data feed so that the record is not deleted. Any type of discrepancy will cause the Source System Ids not to match, resulting in the corresponding record(s) being removed from the UI.
2. If the record does not display in SS&E yet appears in the data view as not deleted (deleted=false/ the record is not red), verify if the linked data records are deleted, causing the data not to display.
   • In most cases, a "linked record" will appear in the data view with the linked data feed name and the src_system_id.
   • For example, in the Transcript Course data view, there is a field for the course_section_src_system_id which links a Transcript Course record to the corresponding Course Section data feed record. When a course section is deleted on the course section feed/all ids, the transcript course/student enrollment record will not display on the student-courses tab even though the transcript course record itself is deleted=false.
3. To summarize, if the data is missing in SS&E because the record is marked as deleted=true, this means that the matching source_system_id is not included in the corresponding All Ids data extract/Json file. In order to fix this, the "missing" source system ids must be included in the All Ids data extract definitions. Once included, the missing data should display in the UI.

• If the data feed integrates data deletion with the SIS and the data is missing in SS&E, the reason is most likely that the source system id associated with the missing data is missing from the corresponding All Ids feed.

• If the "missing" data is included in the data feed view yet the source system id records continue to be missing from the All Ids file, this will result in the data being re-imported as the import runs 4x/day and re-deleted once every 24 hours, as the SIS Data Deletion job runs once a day.

• Undeleting the records manually in SS&E and/or reimporting the data on the associated data feed will have little effect as this would only reverse the data to display in SS&E temporarily.

  • • In addition, if for any reason, the source system id format does not match between the data feed and the corresponding All Id file, these records will also be marked as deleted every time the deletion job runs, causing data to appear inconsistent and display intermittently.
    • For this reason, it is very important to make sure that your Accelerate configuration is defined correctly for the All Ids stream.
• For the All Ids deletion stream to be processed, the enableSisAllIdsPullFromConnector property in the Administrative Application console must be set to true. Navigate here to verify this property.
• The All Ids file only matches to active source system id records in SS&E and does not look at the contents of the data feed data extract/Json file.
• On select data feeds, the All Ids file only matches with "current" active records in SS&E.
  • • For example, Transcript Course All Ids and Course Section All Ids will only check against source system ids associated with course sections in a current or future term.
    • Therefore, the SIS data deletion integration will not remove a course section or transcript course record from a past term.
• Most imported data in SS&E links to other imported data points. For example, Transcript Course data (student enrollments) links to the Course Section data. When a Course Section is deleted from the SIS, it will not be listed in the Course Section All Ids file, and any Transcript Course records that are associated with it will also disappear from the student record. Therefore, if any data does not display as you expect, check the data view. If the record appears in the data view as not deleted (deleted=false), verify if the linked data records are deleted. If you find that data is missing in SS&E because the linked data record was deleted, this can be fixed by finding and updating the source of the linked data deletion, either in the SIS or the extract definition.

All Ids is Missing

• If the All Ids feed is missing for a data feed that integrates data deletion with the SIS, this means that data deletion integration is currently not implemented for the specific data feed.
  • • Something important to note is that IF the All Ids feed was imported in the past yet is now missing, nothing will be deleted from SS&E the next time the SIS Data Deletion job runs.
• If the All Ids file exists yet there are no contents and the file is empty, check if this is being used. If not being used then the All Ids feed should be disabled.
• If the All Ids field contents are empty and it should be used, the most common reason for this is a formatting issue. Verify that the Source ID's and Person IDs are displayed correctly and include all leading zeroes in the source data/csv files. If leading zeroes are missing, that can explain why the All Ids data extract file is empty, and once this is corrected then the All Ids file contents should reappear.

Manual Data Deletion

If the data feed can integrate SIS data deletion, yet data deletion integration is not implemented on the specific data feed that allows data deletion, or if BOTH the data feed data extract AND the All Ids data feed data extract are no longer importing the data that should be removed, you can manually delete the record from the data feed's Imported Data View so that it is removed from the UI.

On data feeds that can delete data via SIS Data Deletion, the secondary nav bar will include the following action items on the Imported Data View.

To manually delete a record:

1. Check here to confirm that the data feed integrates date deletion (Delete = Yes).
2. From SIS Integration Datafeed Administration found here, scroll to the data feed, then open the Imported Data View.
3. Find the records you wish to delete.
4. Make sure the data view filters are only displaying the records that you wish to delete.
5. Using the checkbox in the first column, check all records that are going to be deleted. 
6. Select "Delete selected records".
7. If you make a mistake, you can always re-filter and select "Undelete selected filters".

When deleting a record, SS&E does not actually remove the record from the database, rather it is  marked as deleted=true, which will remove it from displaying on the UI. On the Imported Data View, "Here records can be manually deleted, or undeleted if they are mistakenly marked as deleted."

• As long as the records are also removed from the data extract/All Ids/source data so that it does not re-import, this will not be reversed.
  • • However, if the "incorrect/bad/old" data is still in the source data, then any manual action will be reversed, since the next time the SIS Import job, the data will reimport unless the data feed is disabled and no data is being imported from the SIS.
    • In addition, there are some data feeds with special conditions. For example, Transcript Course records can only be manually deleted if the course section is current, future, or 'Transfer' and the person isn't a test record.

In addition, there are some places where data can be modified or removed within SS&E. For example:

• In People Administration, via the bulk action Add/Remove menus.
• In Course Section Administration, the Status dropdown can manually update the status of a course section if not updated through the SIS Import Course Section All Ids.
  • • Valid course section status values are: Active, Cancelled, Deleted.
    • If you manually change the course section status, you should also check the SIS Overridden box to prevent the SIS Import Process from updating this field in the future.