Connect SFTP Auto-Updates Deployment Guide

Updated
Watermark Support

In December of 2020, improvements were made to the Connect product utilized during the SIS data import process.

  • These improvements converted the Connector from a web service to a scheduled job running on the Connect server.
  • Removing dependencies on Apache Tomcat and HTTP was a main motivation for these changes.

Prior to this change, issues with Tomcat crashes and the general unreliability of the network were common. By removing the always-on Tomcat server, and utilizing SFTP instead of HTTP for data transfer, the reliability of the SIS data import process has greatly increased.

  • The method used for data extraction from the institution and data processing before the data transfer has not changed.
  • The only change is the data transfer method.
  • With the new data transfer method, instead of receiving and fulfilling web requests from the SS&E web application, the Connector job uploads (i.e transfers) data from each client to Amazon S3 via SFTP. From S3, SS&E reads the data and imports the SIS data into SS&E.

Courier

The Courier product was discontinued on 2/29/24.

SFTP Security

While utilizing SFTP, we do not have dedicated IPs on our SFTP endpoints.

If security is required for outbound connections made from the Connect Server, clients can DNS whitelist s-7f32074b0a444c36b.server.transfer.us-east-1.amazonaws.com.

Otherwise, this would require whitelisting all AWS IP addresses for the US-EAST-1 region found here: AWS IP address ranges - Amazon Virtual Private Cloud

Connect Job - Auto Update

Since the initial deployment of the new SFTP Connector, additional improvements have been made to the Connect product, including an auto-update mechanism.

  • Once the auto-updating mechanism is deployed, there should be no reason for SS&E staff to remote into the Connect machine to update Connect unless there is an urgent issue that requires manual maintenance.
  • The steps below include walk-through step-by-step instructions on how to deploy the auto-update mechanism.

Connect Job - External Logging

Another improvement to the new Connector is external logging.

Once implemented, the Connect 2 Full Log and Connect 2 scheduled task log messages will display in SS&E Administration under Advanced > External Logging every time the Connect job runs as a Scheduled Task in the Windows Task Scheduler on the client Connect server.

  • This new feature of External Logging is set up by adding a configuration setting specifying the API token found in SS&E Administration, further described in the step-by-step instructions below.

Note: As of January 1, 2021, Connect 1 has been deprecated in favor of Accelerate/Connect2 and there will be no non-security enhancements to the Connect1 product. For example, query changes, extract updates and logic changes will no longer be implemented or worked on for Connect1.

How to Verify that External Logging is Configured Properly

In order to verify that External Logging is setup and should be working, go to your Connect server and open C:\Aviso\Connect2\config\application.properties to confirm that the api.token and logging file are set.

  1. To find your API token, go to SS&E Administration > Application > apiToken.
    • The line in application.properties should show that the API token saved on the Connect server matches what displays in SS&E.

    • Also, make sure it's named api.token and not api-token. For example:

      api.token=123456789-abcdefg

  2. Confirm that the following line is also included in the application.properties file on the Connect server:

     logging.file=C:/Aviso/Connect2/logs/application.log

If External Logging is not configured properly, follow the instructions below for Connect 1 or Connect 2 to update the Connect Job to include auto updates and external logging.

The following deployment instructions are for updating Connect 1; scroll further down for instructions on updating Connect 2:

Connect 1

  1. Delete the existing Connect Windows Scheduled task.

    The Task Scheduler window showing the Task Scheduler Library tree and a highlighted scheduled task in the Name column marked Ready. The Actions pane shows options such as Create Basic Task, Create Task, Import Task, and Delete. The Delete button appears highlighted under Selected Item.

  2. Add a PowerShell script to automate most of the setup of the new tasks by creating the following file C:\Aviso\software\connect1TaskSetup.ps1.

    Note: If the file already exists, delete it and create a new file.

    To create this file, either:

    • Copy and paste the file from another computer
    • Copy and paste the content from another computer
    • Download* the newest version from connect1 Task Setup

    *If you directly download the script, open it in a text editor, add a new empty line at the end, then save in order to prevent unsigned script errors.

    The Connect2 folder window showing subfolders for config, logs, and files including connect2TaskSetup.ps1 and connector‑standalone.jar. The connect2TaskSetup.ps1 file appears highlighted. (screenshot/example is from Connect2)

  3. Run the following script in a Command Prompt to add the new scheduled tasks.

    Run as administrator, and enter credentials when prompted:

    powershell.exe -noexit "& 'C:\Aviso\software\connect1TaskSetup.ps1'"

    The Search window showing results for PowerShell and a context menu that includes Unpin from Start, Unpin from Taskbar, Uninstall, Open in new window, Run as administrator, and Open file location. The Run as administrator option appears highlighted. The PowerShell window showing a command entered to run connect2TaskSetup.ps1 and the resulting status messages. A credential prompt dialog appears, containing fields for User name and Password. The User name field appears highlighted.

    IMPORTANT NOTE: If the password changes for the user that is associated with these new tasks, it must be updated in Windows Task Scheduler in order for the tasks to run.

  4. Edit the existing config.groovy file (usually at C:\avconfig.groovy or C:\Aviso\config.groovy) by adding the following line at the end of the file. This will enable external logging. (Copy the token in between single quotes)

    api.token='xxxxxxxxxxxxxxxxxxxxxxxx'

    Use the API token from the Application menu in the Administrative view at Administration Application apiToken. Copy the entire value including any commas or special characters.

  5. Test the two new scheduled tasks by manually running them from Task Scheduler and verifying that they are running correctly:

    The Task Scheduler window showing the Task Scheduler Library list with scheduled tasks. One task named Aviso Connect2 appears highlighted in the Name column, and the Triggers column shows its start time and repeat schedule.

    • Verify the auto-update task by running it and checking that there is an entry in the auto_update.log at C:\Aviso\software\update\.
      The Notepad window showing the auto_update.log file with repeated log entries that note timestamps and indicate 'New JAR downloaded: False' followed by date ranges.
    • Verify the Connect task by running it and checking that there is data populating in external logging at Administration Advanced External Logging. The Log Messages page showing a table of log entries with columns for ID, Message, Log Level, Log Date, and Source. The entries display various INFO and ERROR messages from Connect2 along with their timestamps.
  6. If the Connect job does not run from the Task Scheduler, use the following command to check if it runs from a command prompt (Run as Administrator):

    java -Xms1G -Xmx8G -jar avisoConnector.jar port=8880

  7. If the new Connector job runs from a command prompt yet does not run from the Windows Task Scheduler, this is most likely a permissions-related issue that displays in the Windows logs. Search the Windows logs to find/fix the error and then try again.

Connect 2

  1. Delete the existing Connect Windows Scheduled task.

    The Task Scheduler window showing the Task Scheduler Library list with multiple tasks, including one highlighted entry named Aviso Connect2 marked Ready in the Status column. The Actions pane includes options such as Create Basic Task, Create Task, Import Task, Display All Running Tasks, and Delete, with the Delete button highlighted under Selected Item.

  2. Add a PowerShell script to automate most of the setup of the new tasks by creating the following file C:\Aviso\Connect2\connect2TaskSetup.ps1. Note: If the file already exists, delete it and create a new file.

    To create this file, either:

    • Copy and paste the file from another computer
    • Copy and paste the content from another computer
    • Download* the newest version from connect2 Task Setup

    *If you directly download the script, open it in a text editor, add a new empty line at the end, then save in order to prevent unsigned script errors.

    The Connect2 folder window showing subfolders config and logs along with files connect2TaskSetup.ps1 and connector‑standalone.jar. The connect2TaskSetup.ps1 file appears highlighted.

  3. Run the following script in a Command Prompt to add the new scheduled tasks. Run as administrator, and enter credentials when prompted:

    powershell.exe -noexit "& 'C:\Aviso\Connect2\connect2TaskSetup.ps1'"

    The Search window showing results for PowerShell along with a context menu containing options such as Unpin from Start, Unpin from Taskbar, Uninstall, Open in new window, Run as administrator, and Open file location. The Run as administrator option appears highlighted. The PowerShell window showing a command entered to run connect2TaskSetup.ps1 along with status messages. A credential prompt dialog appears, containing fields for User name and Password. The User name field appears highlighted.

    IMPORTANT NOTE: If the password changes for the user that is associated with these new tasks, it must be updated in Windows Task Scheduler for the tasks to run.

  4. Edit the existing application.properties file (usually at C:\Aviso\Connect2\config\application.properties) by adding the following line at the end of the file. This will enable external logging.

    api.token=xxxxxxxxxxxxxxxxxxxxxxxx

    Use the API token from the Application menu in the Administrative view at Administration Application apiToken. Copy the entire value including any commas or special characters.

  5. Test the two new scheduled tasks by manually running them from Task Scheduler and verifying they are running correctly:

    The Task Scheduler window showing the Task Scheduler Library list with scheduled tasks. The entry named Aviso Connect2 appears highlighted in the Name column, and the Triggers column shows its start time and repeat schedule.

    • Verify the auto-update task by running it and checking that there is an entry in the auto_update.log at C:\Aviso\Connect2\update\.
      The Notepad window showing the auto_update.log file with repeated log entries that record timestamps and state 'New JAR downloaded: False' followed by matching date ranges.
    • Verify the Connect task by running it and checking that there is data populating in external logging at Administration Advanced External Logging. The Log Messages page showing a table of log entries with columns for ID, Message, Log Level, Log Date, and Source, listing multiple INFO and ERROR messages from Connect2 along with their timestamps.
  6. If the Connect job does not run from the Task Scheduler, use the following command to check if it runs from a command prompt (Run as Administrator):

    java -Xms1G -Xmx8G -jar connector-standalone.jar --spring.config.additional-location=C:/Aviso/Connect2/config/application.properties

  7. If the new Connector job runs from a command prompt yet does not run from the Windows Task Scheduler, this is most likely a permissions-related issue that displays in the Windows logs. Search the Windows logs to find/fix the error and then try again.
Was this article helpful?
0 out of 0 found this helpful

Articles in this section

See more
How to Contact Support
There are many ways to reach out! Click the icon above for our support options.
Watermark Academy
Click the icon above to access the Watermark Academy for consultation, training, and implementation companion courses.
Customer Community
Can’t find the answer? Ask fellow users how they’re making the most of Watermark in our Community!