Amazon S3

This page describes how to use Census with S3.

Getting started

This guide shows you how to use Census to connect your S3 account to your data warehouse.

Prerequisites

Before you begin, you'll need the following:

• Census account: If you don't have this already, start with a free trial.

• S3 account: You'll need an S3 bucket that's ready for use, along with credentials for an AWS user that can write to that bucket. See AWS documentation for setting up an S3 bucket. The specific permissions Census may need are covered later in Required permissions.

• Have the proper credentials to access to your data source.

Connecting to an S3 bucket

1. Enter authentication details for your AWS account and S3 bucket: Access Key ID, Secret Key, Bucket Name, and AWS Region. Census uses these details to authenticate – they must match your AWS setup. If you would like to use role based permissions instead, see the instructions below.

2. Click Save Connection.

Using role-based permissions

As an alternative to using keys you may opt to grant Census access to a role in your AWS account. This won't provide any additional functionality from Census, but may be preferable for your AWS configuration. This is a multi-step process with parts happening in Census and inside your AWS console.

Step 1: When configuring the S3 destination click the "Use role" checkbox. Provide your bucket and region, but leave access and secret key blank. Click Connect:

Step 2: The automated connection check will run at this point and fail, this is expected.

Step 3: Click the 'Back' button to return to editing the destination. You should now see an 'External ID' input box with a string in it. You will use this string in the following step.

Step 4: Open your AWS Console in a separate tab and browse to the IAM service. Click 'Roles' and 'Create role'.

• When creating the role choose 'AWS Account' for Trusted Entity Type and the 'Another AWS Account' radio button.

• Provide Census's AWS Account ID: 341876425553.

• Check the 'Require external ID' checkbox and enter the External ID string from Step 3.

• Finish setting up your Role. Note that it should have the required permissions to access the bucket you are using as your Census destination!

• When done, click on your role and copy its ARN. Go back to the tab where you're editing the Census S3 Destination and enter the role ARN.

• Click 'Connect'. The tester should re-run and succeed.

Supported Sync Behaviors

Behavior	Supported?	Objects
Update or Create	✅	All
Replace	✅	All

Update or Create Syncs

Update or Create syncs upload your whole dataset on the first run and only new changes on subsequent runs. Each sync run saves to a different file. The first run saves with "full" at the end of the file name. For example, filename_12_12_23_full.csv if it runs on 12/12/2023. Later syncs save with a timestamp at the end, like filename_12_12_23_1702426195.csv, so you can see how your data changes over time.

Learn more about all of our sync behaviors on our Core Concepts page.

Let us know if you want Census to support additional sync behaviors for S3.

File Path

When setting up a sync to S3, you can provide a file path for the file name Census will create/replace. The file path can include folders. Data arrives in one file to the designated bucket and file path.

Variables

When defining the File Path, you can use variables that will be set when the sync runs. This allows you to create and sync to new files that reflect the date and time of the sync.

Variable	Description	Example Values
%Y	4-digit year	1997
%y	2-digit year	97
%m	month with zero padding	07, 12
%-m	month without zero padding	7, 12
%d	day with zero padding	03, 23
%-d	day without zero padding	3, 23
%H	24 hour with zero padding	08, 18
%k	24 hour without zero padding	8, 18
%I	12 hour with zero padding	08, 12
%l	12 hour without zero padding	8, 12
%M	minute with zero padding	04, 56
%S	second with zero padding	06, 54

Advanced Configuration

In addition to the file path, you can configure how the data is encoded as it is written. Primarily this is a question of file format:

• CSV - The standard comma separated values file. You can optionally specify an alternative delimeter such as |*, and you can enable/disable the header row.

• TSV - The tab separated values file. You can enable/disable the header row.

• JSON - A single JSON arraay of objects

• NDJSON - New line-delimited list of JSON objects

• Parquet - A columnar storage format that is more efficient for certain types of data.

• If your configured delimiter is present in data values, Census will automatically add double quotes around the value. Example: Hello, world is written as as "Hello, world" if the chosen delimiter is a comma.

In addition to file format, you can also provide a PGP Public Key to encrypt the data before it is written to the file. This is useful for ensuring that the data is secure in transit and at rest.

Other Things to Know

• We support syncing files up to 100GB. Files larger than 5GB may require some additional permissions, see Required permissions below.

• We highly recommend adding default server-side encryption to your S3 buckets. Census supports syncing to buckets with encryption policies as long as the bucket uses an AWS provided key type like the Amazon S3 key (SSE-S3) or the AWS Key Management Service key (SSE-KMS). If the bucket uses SSE-KMS, make sure the IAM role credentials associated with the S3 connection have access to the AWS KMS key used for encryption. We do not support syncing to buckets using a customer-provided encryption key.

Contact us if your use cases don't work with these limitations. We plan on addressing at least a few of these in the future!

Required permissions

For most S3 uploads, the only permission that we require is the s3:PutObject action.

For files larger than 5GB, Census makes use of S3's Multi-part upload which requires the additional permissions:

• s3:AbortMultipartUpload

• s3:ListMultipartUploadParts

• s3:ListBucketMultipartUploads

For more details on how Multipart Uploads use these permissions, see the S3 documentation.

Example Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::example-bucket",
                "arn:aws:s3:::example-bucket/*"
            ]
        }
    ]
}

Need help connecting to S3?

You can send our support team an email at [email protected] or start a conversation from the in-app chat.