Creating a Flow using SFTP as the Source Connection

In this article

• Overview
• Core Concept of Using The SFTP Launcher in Flow Configurations
• Setting Up your Host or Integration Connection
  • Setting Up API Client Host System For Send Flow Types
  • Setting up HTTPs/API for Receive Flow Types.
• Setting Up a Flow
  • Setting Up Send Flow Types
  • Setting Up Receive Flow Types
• Setting Up SFTP Jobs and Using The SFTP Launcher
  • General
  • Configuration Settings
  • Integration Endpoint Settings
  • Advanced Settings

Overview

Since the introduction of our SFTP Launcher feature, we have updated the standard practice for creating a Flow that uses an SFTP as the Source System/Connection. The steps documented through this SOP will provide guidance on how to set up a Flow that utilizes this SFTP Launcher.

By using the SFTP Launcher feature, Chain.io is able to enforce single file processing when multiple files are found in the same SFTP directory. By enforcing single file processing, you get clearer and more concise execution messages, better visibility in the status of your transactions, and the ability to automate the resubmission process of SFTP executions.

Core Concept of Using The SFTP Launcher in Flow Configurations

There are three components that need to be configured when using an SFTP System/Connection as the source system.

1. API Client Host System or Receive-HTTPs/API Connection
2. Flow Configuration Settings
3. SFTP Launcher Job

As explained in the Flow support article, a Host System and Integration Connection must be set up before creating a Flow. If you are setting up a Send Flow, then an API Client Host System must be created. If you are setting up a Receive Flow, then a Receive-HTTPs/API Connection under that integration must be set up.

The purpose of the Flow Configuration is to receive a file, transform the data, and output the newly transformed file into its final destination.

The purpose of the SFTP Launcher is to pick up files from the SFTP system and feed that file into the Flow that uses the API Client Host System or HTTPs/API Connection.

Setting Up your Host or Integration Connection

Setting Up API Client Host System For Send Flow Types

Navigate to your Workspace Settings:

Select Host Systems:

Click the Add Host System button and then select API Client and then click Next

Name your API Client Host system and click Save

Once this is complete you will now be able to select this Host System when adding a Source System to a Flow Configuration.

Setting up HTTPs/API for Receive Flow Types.

Navigate to the Integration where you would like to add this Connection and select Settings from the options in the main navigation.

In the Connections section at the lower left of the screen, click New.

On the following screen, click the dropdown and choose HTTPs/API under the Receive Data Into Host Systems grouping. 

Give your Connection a name so you can easily find it in Flow Configuration selection panels when assigning a Destination Connection. Click the Save button when complete.

Now that you have set up this Integration Connection you will be able to select it as a Source Connection when setting up Receive Flow Types.

Setting Up a Flow

For more information about how to setup a Flow you can read our support documents that cover Flows.

Setting Up Send Flow Types

This process is the same for every Send Flow configuration. So, without going through which Integration or Send Flow type you chose we can start at the Flow Configuration screen and move on from that point.

Within the Flow configuration screen under Source System, click the Select Host System button to bring up the Host Selection Panel and choose the API Client Host System you created in the previous step. 

Next add a Sub Path and API key to complete the configuration. These values are used to link this Flow with the SFTP Launcher Job we will be setting up in the next step. Essentially this is making the Flow an API endpoint for us to use when sending files to the Flow from an SFTP.

• The Sub Path can be and alphanumeric set with no spaces. This is a value you can choose and does not need to be specific. It is recommended that it is named something associated with the Flow or the SFTP Job use case.
• The Authentication Type should be set to API Key can be any combination of ten or more alphanumeric characters, without spaces. This is used as an identifier for this Flow when setting up the SFTP Launcher Jobs associated with this Flow.

NOTE: Use the copy buttons at the end of each field to keep the URL and API by pasting them into a temporary text file. These values will be needed in the next step when configuring the SFTP Launcher. 

Setting Up Receive Flow Types

This process is the same for every Receive Flow configuration. So, without going through which Integration or Receive Flow type you choose, we can start at the Flow Configuration screen and move on from that point.

Within the Flow configuration screen under Source Connection, click the Select Connection button to bring up the Connection Selection Panel and choose the API Client Host System you created in the previous step. 

Next add a Sub Path and API key to complete the configuration. These values are used to link this Flow with the SFTP Launcher Job we will be setting up in the next step. Essentially this is making the Flow an API endpoint for us to use when sending files to the Flow from an SFTP.

• The Sub Path can be any alphanumeric set with no spaces. This is a value you can choose and does not need to be specific. It is recommended that it is named something associated with the Flow or the SFTP Job use case, like the Flow name or Integration Name.
• The Authentication Type should be API Key can be any combination of ten or more alphanumeric characters, without spaces. This is used as an identifier for this Flow when setting up the SFTP Launcher Jobs associated with this Flow.

NOTE: Use the copy buttons at the end of each field to keep the URL and API by pasting them into a temporary text file. These values will be needed in the next step when configuring the SFTP Launcher. 

Also, be aware that the Authentication Type must be API Key for this to successfully work with the SFTP Launcher

Setting Up SFTP Jobs and Using The SFTP Launcher

In this section we will be setting up the SFTP Launcher Jobs and associating them with the Flow Configurations we created in the previous step. The SFTP Launcher allows you to set up SFTP jobs across all Integrations in your Workspace. This step will show how we to create a SFTP Job under the SFTP Launcher so that it can pick up files from an SFTP location/server and redirect those files to the Flow that you created in the previous step.

Navigate to your Workspace Settings

On the right, in the Setups section, select SFTP Launcher

You will now see the SFTP Launcher Settings landing screen. The SFTP Launcher is where you set up SFTP Jobs, view information about those SFTP Jobs and review events triggered by SFTP Jobs associated with Flows across your Workspace. If this is the first time you are setting up an SFTP Job in the Launcher you will see a mostly blank screen like the one below.

To add a new SFTP Job and link it to your Flow, click the Add Job button.

You will now see the SFTP Job configuration screen. In this screen you will provide the SFTP information that allows the SFTP Launcher to access and retrieve files from the SFTP server locations and redirect them to your flow’s HTTPs endpoint(API client or HTTPs/API connection) that was configured in the previous step.

General

Use this area to add a name for your SFTP Job and enter the details for your SFTP server.

• Name - a unique label for this SFTP Job and its server configuration.
• Server Address - the physical address of your SFTP server (i.e. https;//myfileserver.com/)
• Username - The username needed to log in to the SFTP server
• Password - The password you use to log into the SFTP server
• Port Number - Add the port number that your server uses to accept SFTP connections (The standard default is 22. Do not change this unless you have confirmed your server IS NOT using the default port.)
• Enable Insecure FTP (Optional) - Select this checkbox if you would like to force the connection to be insecure (Not Recommended)

Configuration Settings

Use this area to configure the specifics for this SFTP Job.

• Sub-folder - The folder location on the SFTP Server. (ex. /files or /files/documents). If you do not know which sub-folder to enter here, please ask your companies system administrator or refer to our support documentation on Adding a Chain Hosted SFTP
• File Criteria - This field is provided so you can isolate or work with specific files on your server. The criteria for the job needs to be written in Regular Expression. For example, If you wanted to only work with the Excel files you would write your criteria similar to the following: /+\xlsx/.
• Polling Interval - How often would you like this SFTP Job to run
• Delete, Archive, or Retained - After this job runs, would you like your files Deleted, Archived, or Retained on the server. Retained will leave them untouched.
• Maximum Number of Files - This is a numerical value of the maximum number of files you would like to process within this job.

Integration Endpoint Settings

Use this area to link this SFTP Job to your Integration Flow. You will need to enter in a couple of things which can be found in the Integration Flow you wish to associate to this.

• Web Address - In this field enter the URL and Sub Path from the API Client Host System or Integration Connection you setup within the Flow this SFTP Job will be used on.
• API Key - This will be the same ten character or more API Key you configured within the API Client Host System or Integration Connection.

NOTE: To be sure you copy the correct information, the fields for these elements in the Flow Configuration have a copy button

Advanced Settings

Use this area for more advanced settings.

Once you provide all of the applicable SFTP job configurations, click the Save button.

Upon saving, the SFTP Launcher will access the SFTP server you specified in the job. Please note that the first SFTP event (the attempt to retrieve a file from the SFTP) will occur based on the polling interval you configured in the SFTP Job. For example, if you had a polling interval set to 30 minutes and you saved the SFTP job at 1:00 pm, then the first SFTP event would occur at 1:30 pm.

You now have completed all of the necessary steps for configuring a flow that retrieves files from an SFTP.

Additional Notes:

You can review the status of the SFTP Events section under the SFTP Launcher page, these events will allow you to troubleshoot your SFTP Job if it is not successful due to incorrect credentials or configurations.