SFTP

Getting started

This guide shows you how to use Census to connect your SFTP server to your data warehouse and create your first sync.

Prerequisites

Before you begin, you'll need the following:

• Census account: If you don't have this already, start with a free trial.

• SFTP server: You'll need the host address, username, and password or private key.

• Have the proper credentials to access to your data source. See our docs for each supported data source for further information:

  • Azure Synapse

  • Databricks

  • Elasticsearch

  • Google BigQuery

  • Google Sheets

  • MySQL

  • Postgres

  • Redshift

  • Snowflake

  • SQL Server

Step 1: Connect SFTP

1. Log into Census and navigate to Destinations.

2. Click New Destination.

3. Select SFTP from the dropdown list.

4. Enter a Name for your destination. This is only for your reference – it can be anything that makes sense to you.

5. Enter authentication details for your SFTP server. Host and Username are always required. If your server requires a password instead of an SSH key, enter the Password. If your server uses SSH keys, you can leave the Password blank.

6. Click Save Connection.

7. If you're using SSH keys to authenticate your server, download the SFTP Public Key from this screen and upload it to your server. Then, click Test to verify that the connection works.

• If you aren't using a password for your server, Census provides an RSA token with an OpenSSH-formatted public key.

Your end state should look something like this: 👇

Step 2: Connect your data warehouse

The steps for connecting your data warehouse will depend on your technology. See the following guides:

• Databricks

• Google BigQuery

• Google Sheets

• Postgres

• Redshift

• Snowflake

After setting up your warehouse, your Destinations page should look something like this: 👇

Step 3: Create your model

When defining models, you'll write SQL queries to select the data you want to sync. This can be as simple as selecting everything in a specific database table or as complex as creating new calculated values.

1. From inside your Census account, navigate to the Models page.

2. Enter a name for your model. You'll use this to select the model later.

3. Enter your SQL query. If you want to test the query, use the Preview button.

4. Click Save Model.

Step 4: Create your first sync

The sync will move data from your warehouse to a new or existing CSV file on your SFTP server. In this step, you'll define how that will work.

1. From inside your Census account, navigate to the Syncs page and click Add Sync.

2. Under What data do you want to sync?, choose your data warehouse as the Connection and your model as the Source.

3. Under Where do you want to sync data to?, choose the name you assigned in Step 1 (we used SFTP) as the Connection. Enter the File Path for the CSV file where data will sync. The path can accept variables that will populate when the sync runs. See File Path Variables. Confirm the file path in the Template Preview field.

4. Under How should changes to the source be synced?, Replace will be automatically selected. This is the only supported sync behavior for SFTP.

5. Under Which properties should be updated?, choose whether to sync only Selected Properties or Sync All Properties. Syncing all properties will add new properties to the sync if the model changes.

6. To test your sync without actually syncing data, click Run Test and verify the results.

7. Click Next. This will open the Confirm Details page where you can see a recap of your setup.

8. If you want to start a sync immediately, set the Run a sync now? checkbox.

9. Click Create Sync.

When configuring your sync, the page should look something like this: 👇

Step 5: Confirm the synced data

Once your sync is complete, it's time to check your data. Go to the specified path on your SFTP server and check that the file updated correctly.

If everything went well, that's it! You've started syncing data from your warehouse to your SFTP server! 🥳️

And if anything went wrong, contact the Census support team to get some help.

Supported Sync Behaviors

Update or Create Syncs

Update or Create syncs upload your whole dataset on the first run and only new changes on subsequent runs. Each sync run saves to a different file. The first run saves with "full" at the end of the file name. For example, filename_12_12_23_full.csv if it runs on 12/12/2023. Later syncs save with a timestamp at the end, like filename_12_12_23_1702426195.csv, so you can see how your data changes over time.

Let us know if you want Census to support additional sync behaviors for SFTP server connections.

File Path

When setting up a sync to SFTP, you can provide a file path for the file name Census will create/replace. The file path can include folders. Data arrives in one file to the designated server and file path.

Variables

When defining the File Path, you can use variables that will be set when the sync runs. This allows you to create and sync to new files that reflect the date and time of the sync.

Advanced Configuration

In addition to the file path, you can configure how the data is encoded as it is written. Primarily this is a question of file format:

• CSV - The standard comma separated values file. You can optionally specify an alternative delimeter such as |*, and you can enable/disable the header row.

• TSV - The tab separated values file. You can enable/disable the header row.

• JSON - A single JSON arraay of objects

• NDJSON - New line-delimited list of JSON objects

• Parquet - A columnar storage format that is more efficient for certain types of data.

• If your configured delimiter is present in data values, Census will automatically add double quotes around the value. Example: Hello, world is written as as "Hello, world" if the chosen delimiter is a comma.

In addition to file format, you can also provide a PGP Public Key to encrypt the data before it is written to the file. This is useful for ensuring that the data is secure in transit and at rest.

Make sure the key includes the full block, including the following headers and footers:

-----BEGIN PGP PUBLIC KEY BLOCK-----
<your-public-key-here>
-----END PGP PUBLIC KEY BLOCK-----

Need help connecting your server?

You can send our support team an email at [email protected] or start a conversation from the in-app chat.