This page describes how to use Census with SFTP.
This guide shows you how to use Census to connect your SFTP server to your data warehouse and create your first sync.
Before you begin, you'll need the following:
- SFTP server: You'll need the host address, username, and password or private key.
- Have the proper credentials to access to your data source. See our docs for each supported data source for further information:
- 2.Click Add Service.
- 3.Select SFTP from the dropdown list.
- 4.Enter a Name for your destination. This is only for your reference – it can be anything that makes sense to you.
- 5.Enter authentication details for your SFTP server. Host and Username are always required. If your server requires a password instead of an SSH key, enter the Password. If your server uses SSH keys, you can leave the Password blank.
- 6.Click Save Connection.
- 7.If you're using SSH keys to authenticate your server, download the SFTP Public Key from this screen and upload it to your server. Then, click Test to verify that the connection works.
Your end state should look something like this: 👇
Connections page with SFTP server set up
The steps for connecting your data warehouse will depend on your technology. See the following guides:
After setting up your warehouse, your Connections page should look something like this: 👇
Connections page with data warehouse and SFTP server
When defining models, you'll write SQL queries to select the data you want to sync. This can be as simple as selecting everything in a specific database table or as complex as creating new calculated values.
- 2.Enter a name for your model. You'll use this to select the model later.
- 3.Enter your SQL query. If you want to test the query, use the Preview button.
- 4.Click Save Model.
Basic SQL query for a new model
The sync will move data from your warehouse to a new or existing CSV file on your SFTP server. In this step, you'll define how that will work.
- 2.Under What data do you want to sync?, choose your data warehouse as the Connection and your model as the Source.
- 3.Under Where do you want to sync data to?, choose the name you assigned in Step 1 (we used SFTP) as the Connection. Enter the File Path for the CSV file where data will sync. The path can accept variables that will populate when the sync runs. See File Path Variables. Confirm the file path in the Template Preview field.
- 4.Under How should changes to the source be synced?, Mirror will be automatically selected. This is the only supported sync behavior for SFTP.
- 5.Under Which properties should be updated?, choose whether to sync only Selected Properties or Sync All Properties. Syncing all properties will add new properties to the sync if the model changes.
- 6.To test your sync without actually syncing data, click Run Test and verify the results.
- 7.Click Next. This will open the Confirm Details page where you can see a recap of your setup.
- 8.If you want to start a sync immediately, set the Run a sync now? checkbox.
- 9.Click Create Sync.
When configuring your sync, the page should look something like this: 👇
Setting up a sync to an SFTP server
Once your sync is complete, it's time to check your data. Go to the specified path on your SFTP server and check that the file updated correctly.
If everything went well, that's it! You've started syncing data from your warehouse to your SFTP server! 🥳️
When defining the File Path for an S3 sync, you can use variables that will be set when the sync runs. This allows you to create and sync to new CSV files in the S3 bucket that reflect the date and time of the sync.
month with zero padding
month without zero padding
day with zero padding
day without zero padding
24 hour with zero padding
24 hour without zero padding
12 hour with zero padding
12 hour without zero padding
minute with zero padding
second with zero padding
Update or Create syncs upload your whole dataset on the first run and only new changes on subsequent runs. Each sync run saves to a different file. The first run saves with "full" at the end of the file name. For example,
filename_12_12_23_full.csvif it runs on 12/12/2023. Later syncs save with a timestamp at the end, like
filename_12_12_23_1702426195.csv, so you can see how your data changes over time.
Update or Create
- If you aren't using a password for your server, Census provides an RSA token with an OpenSSH-formatted public key.
- Data arrives in one file written to the designated SFTP file path.
- Files are written as a CSV with headers.
You can send our support team an email at [email protected] or start a conversation from the in-app chat.