Google Cloud Storage
This page describes how to use Census with Google Cloud Storage.
- Have the proper credentials to access to your data source. See our docs for each supported data source for further information:
Our Google Cloud connector behaves a little differently than other Census connectors. Instead of going through an OAuth connection flow, we provide you a Google Identity address to which you grant Storage Object permissions.
- Click the New Destination button, and select Google Cloud Storage
- Paste the name of the GCS bucket, name the connection, and click the Save Connection button
- Your new Google Cloud Storage connection will include a GCP service account for that specific Census connection. Click the copy button () to save it to your clipboard.
- Now head to the Google Cloud Storage bucket that you want to sync to, and click into that bucket's details.
- Click the Permissions tab on the Bucket, then click "Add Permissions".
- Paste the credentials in the New Principals portion and select "Storage Object Admin."
You will be able to send a successful sync if the file path variables are unique per sync run by only granting this service account "Storage Object Creator", but the Test Connection in the next step will fail.
- Click save to save the permission.
You are good to go!
The sync will move data from your warehouse to your GCS bucket. In this step, you'll define how that will work.
- 1.
- 2.Under What data do you want to sync?, choose your data warehouse as the Connection and your source: either a Model from the Census Models tab or a database table.
- 3.Under Where do you want to sync data to?, choose the name you assigned in Step 1 (we used GCS) as the Connection. Enter the File Path for the file where data will sync. The path can accept variables that will populate when the sync runs. See File Path Variables. Confirm the file path in the Template Preview field.__If the service account only has "Storage Object Creator" permissions, this file path needs to be unique per sync run💡
- 4.Under How should changes to the source be synced?, Mirror will be automatically selected. This is the only supported sync behavior for GCS.
- 5.Under Which properties should be updated?, choose whether to sync only Selected Properties or Sync All Properties. Syncing all properties will automatically add new properties to the sync if the model or database table changes.
- 6.To test your sync without actually syncing data, click Run Test and verify the results.
- 7.Click the Next button to see the final preview which will have a recap of what will happen when you start the sync. If you're happy, check the Sync Now checkbox and save the sync.
- 8.Confirm the data arrives in the GCS bucket!
When defining the File Path for an GCS sync, you can use variables that will be set when the sync runs. This allows you to create and sync to new files in the GCS bucket that reflect the date and time of the sync.
Variable | Description | Example Values |
%Y | 4-digit year | 1997 |
%y | 2-digit year | 97 |
%m | month with zero padding | 07, 12 |
%-m | month without zero padding | 7, 12 |
%d | day with zero padding | 03, 23 |
%-d | day without zero padding | 3, 23 |
%H | 24 hour with zero padding | 08, 18 |
%k | 24 hour without zero padding | 8, 18 |
%I | 12 hour with zero padding | 08, 12 |
%l | 12 hour without zero padding | 8, 12 |
%M | minute with zero padding | 04, 56 |
%S | second with zero padding | 06, 54 |
Update or Create syncs upload your whole dataset on the first run and only new changes on subsequent runs. Each sync run saves to a different file. The first run saves with "full" at the end of the file name. For example,
filename_12_12_23_full.csv if it runs on 12/12/2023. Later syncs save with a timestamp at the end, like filename_12_12_23_1702426195.csv, so you can see how your data changes over time.Behavior | Supported? | Objects |
Mirror | ✅ | All |
- Currently, the connector only supports syncing for files up to 5GB.
- Data arrives in one file to the designated bucket and file path.
Contact us if your use cases don't work with these limitations. We plan on addressing at least a few of these in the future!
You can send our support team an email at sup[email protected] or start a conversation from the in-app chat.
Last modified 1mo ago