LogoLogo
  • 🦩Overview
  • 💾Datasets
    • Overview
    • Core Concepts
      • Columns & Annotations
      • Type & Property Mappings
      • Relationships
    • Basic Datasets
      • dbt Integration
      • Sigma Integration
      • Looker Integration
    • SaaS Datasets
    • CSV Datasets
    • Streaming Datasets
    • Entity Resolution
    • AI Columns
      • AI Prompts Recipe Book
    • Enrichment Columns
      • Quick Start
      • HTTP Request Enrichments
    • Computed Columns
    • Version Control
  • 📫Syncs
    • Overview
    • Triggering & Scheduling
    • Retry Handling
    • Live Syncs
    • Audience Syncs
    • Observability
      • Current Sync Run Overview
      • Sync History
      • Sync Tracking
      • API Inspector
      • Sync Alerts
      • Observability Lake
      • Datadog Integration
      • Warehouse Writeback
      • Sync Lifecycle Webhooks
      • Sync Dry Runs
    • Structuring Data
      • Liquid Templates
      • Event Syncs
      • Arrays and Nested Objects
  • 👥Audience Hub
    • Overview
    • Creating Segments
      • Segment Priorities
      • Warehouse-Managed Audiences
    • Experiments and Analysis
      • Audience Match Rates
    • Activating Segments
    • Calculated Columns
    • Data Preparation
      • Profile Explorer
      • Exclusion Lists
  • 🧮Data Sources
    • Overview
    • Available Sources
      • Amazon Athena
      • Amazon Redshift
      • Amazon S3
      • Azure Synapse
      • ClickHouse
      • Confluent Cloud
      • Databricks
      • Elasticsearch
      • Kafka
      • Google AlloyDB
      • Google BigQuery
      • Google Cloud SQL for PostgreSQL
      • Google Pub/Sub
      • Google Sheets
      • Greenplum
      • HTTP Request
      • HubSpot
      • Materialize
      • Microsoft Fabric
      • MotherDuck
      • MySQL
      • PostgreSQL
      • Rockset
      • Salesforce
      • SingleStore
      • Snowflake
      • SQL Server
      • Trino
  • 🛫Destinations
    • Overview
    • Available Destinations
      • Accredible
      • ActiveCampaign
      • Adobe Target
      • Aha
      • Airship
      • Airtable
      • Algolia
      • Amazon Ads DSP (AMC)
      • Amazon DynamoDB
      • Amazon EventBridge
      • Amazon Pinpoint
      • Amazon Redshift
      • Amazon S3
      • Amplitude
      • Anaplan
      • Antavo
      • Appcues
      • Apollo
      • Asana
      • AskNicely
      • Attentive
      • Attio
      • Autopilot Journeys
      • Azure Blob Storage
      • Box
      • Bloomreach
      • Blackhawk
      • Braze
      • Brevo (formerly Sendinblue)
      • Campaign Monitor
      • Canny
      • Channable
      • Chargebee
      • Chargify
      • ChartMogul
      • ChatGPT Retrieval Plugin
      • Chattermill
      • ChurnZero
      • CJ Affiliate
      • CleverTap
      • ClickUp
      • Constant Contact
      • Courier
      • Criteo
      • Crowd.dev
      • Customer.io
      • Databricks
      • Delighted
      • Discord
      • Drift
      • Drip
      • Eagle Eye
      • Emarsys
      • Enterpret
      • Elasticsearch
      • Facebook Ads
      • Facebook Product Catalog
      • Freshdesk
      • Freshsales
      • Front
      • FullStory
      • Gainsight
      • GitHub
      • GitLab
      • Gladly
      • Google Ads
        • Customer Match Lists (Audiences)
        • Offline Conversions
      • Google AlloyDB
      • Google Analytics 4
      • Google BigQuery
      • Google Campaign Manager 360
      • Google Cloud Storage
      • Google Datastore
      • Google Display & Video 360
      • Google Drive
      • Google Search Ads 360
      • Google Sheets
      • Heap.io
      • Help Scout
      • HTTP Request
      • HubSpot
      • Impact
      • Insider
      • Insightly
      • Intercom
      • Iterable
      • Jira
      • Kafka
      • Kevel
      • Klaviyo
      • Kustomer
      • Labelbox
      • LaunchDarkly
      • LinkedIn
      • LiveIntent
      • Loops
      • Mailchimp
      • Mailchimp Transactional (Mandrill)
      • Mailgun
      • Marketo
      • Meilisearch
      • Microsoft Advertising
      • Microsoft Dynamics
      • Microsoft SQL Server
      • Microsoft Teams
      • Mixpanel
      • MoEngage
      • Mongo DB
      • mParticle
      • MySQL
      • NetSuite
      • Notion
      • OneSignal
      • Optimizely
      • Oracle Database
      • Oracle Eloqua
      • Oracle Fusion
      • Oracle Responsys
      • Orbit
      • Ortto
      • Outreach
      • Pardot
      • Partnerstack
      • Pendo
      • Pinterest
      • Pipedrive
      • Planhat
      • PostgreSQL
      • PostHog
      • Postscript
      • Productboard
      • Qualtrics
      • Radar
      • Reddit Ads
      • Rokt
      • RollWorks
      • Sailthru
      • Salesforce
      • Salesforce Commerce Cloud
      • Salesforce Marketing Cloud
      • Salesloft
      • Segment
      • SendGrid
      • Sense
      • SFTP
      • Shopify
      • Singular
      • Slack
      • Snapchat
      • Snowflake
      • Split
      • Sprig
      • Statsig
      • Stripe
      • The Trade Desk
      • TikTok
      • Totango
      • Unify
      • Userflow
      • Userpilot
      • Vero Cloud
      • Vitally
      • Webhooks
      • Webflow
      • X Ads (formerly Twitter Ads)
      • Yahoo Ads (DSP)
      • Zendesk
      • Zoho CRM
      • Zuora
    • Custom & Partner Destinations
  • 📎Misc
    • Credits
    • Census Embedded
    • Data Storage
      • Census Store
        • Query Census Store from Snowflake
        • Query Census Store locally using DuckDB
      • General Object Storage
      • Bring Your Own Bucket
        • Bring your own S3 Bucket
        • Bring your own GCS Bucket
        • Bring your own Azure Bucket
    • Developers
      • GitLink
      • Dataset API
      • Custom Destination API
      • Management API
    • Security & Privacy
      • Login & SSO Settings
      • Workspaces
      • Role-based Access Controls
      • Network Access Controls
      • SIEM Log Forwarding
      • Secure Storage of Customer Credentials
      • Digital Markets Act (DMA) Consent for Ad Platforms
    • Health and Usage Reporting
      • Workspace Homepage
      • Product Usage Dashboard
      • Observability Toolkit
      • Alerts
    • FAQs
Powered by GitBook
On this page
  • What is a Sync?
  • Creating Syncs
  • 🔀 Sync Behaviors
  • 🔎 Sync Keys
  • 🖇 Field Mappings
  • Running Syncs
  • ⏱ Schedules and API
  • Sync History
  • 🔌 Sources and Destinations
  • Data Sources
  • Destination Services
  • Need Help?

Was this helpful?

  1. Syncs

Overview

What is a Sync?

A Sync is the core operation in Census that defines how data should flow from your selected source to your destination. Syncs are what make Census unique - unlike most integration tools that are event-based, Census works to keep your source and destination "in sync" (similar to how Dropbox or other cloud storage services work). This means you don't have to worry about dropped events or backfills. If you change your data in the source, Census automatically syncs it to your destination.

Creating Syncs

To create a sync, you'll need to define:

  1. A source - Where your data is coming from (Basic Dataset, Streaming Dataset, SaaS Dataset, CSV Dataset, or direct warehouse access)

  2. A destination - Where your data is going (CRM, marketing platform, etc.)

  3. A sync behavior - How Census should handle the data (update, create, mirror, etc.)

  4. Sync keys - How to match records between source and destination

  5. Field mappings - Which fields to sync and how they map between systems

🔀 Sync Behaviors

All of our syncs are incremental, which means we only sync records that are new or have their data changed since the previous sync.

Sync Behaviors tell Census the types of change it should apply to your data when a sync finds a matching (or not) record in a source and a destination. We support a few different sync behaviors:

Update or Create (aka Upsert)

Update existing destination records when IDs match, otherwise create new ones if they're missing.

Update Only

Update existing objects in the destination, don't create any new ones.

Create Only

Create a new object if it doesn't exist in the destination.

Mirror

Keep the destination in sync with the source.

  • If a row is added or edited in the source, update the destination.

  • If a previously synced row no longer is in the source, remove the matching object from the destination. Note: some services such as Braze offer other actions as well. See specific connector documentation for more details.

  • Mirror syncs identify changes by comparing against the data they have already sent -- not the data that might or might not already exist in the destination. This means that the first sync will be an upsert for all records, and the second and following syncs will account for changes from the source data.

Append Only

Treat the destination as an append only log of data suitable for Event data.

Delete

Delete syncs takes the list of provided IDs and deletes them from the destination. Be careful! Providing an incorrect set of IDs can result in data loss! Census recommends you backup data if you're concerned about accidentally deleting the wrong records.

Please note that some of these behaviors are only available for certain destinations. Visit our individual integration pages to view what's supported.

🔎 Sync Keys

Sync Keys let Census know how to associate data in the source with the destination. Both the source and destination need to provide a single, unique per record, identifying field. Census uses the identifiers to look for matches. When a match is found, or not found, it then can use your selected Sync Behavior to decide what to do.

For example, if Census sees your source and destination both have records with the identifier ABCD1234, it knows that it should update that record with data from the source when you've got an Update Only, Update or Create, or Mirror Sync Behavior configured.

🖇 Field Mappings

Once you've defined how data is related between your source and destination, the next step is to let Census know what properties should be updated. The field mapping step lets you specify how fields should be mapped from your source model to the destination object's fields. You can automatically add all matching fields, but even if names don't match, you can also provide the matching manually. If you remove fields from your mapping, Census will just stop updating those fields. We will not delete the values.

Using templates to transform source data

Sometimes the data in your source systems isn't in quite the format or style that the destination expects. If you need to transform records from the source before sending them to the destination, you can use a Templated Field.

Census supports templated fields using the Liquid template language. As a basic example, this Liquid template lets you combine a customer's city and country into a single piece of text:

{{ record['city'] }}, {{ record['country'] }}

To give you an idea of the capabilities available, here are a few more examples:

  • Customer → Sends just the word "Customer" to the destination field

  • Last updated at {{ "now" | date: "%Y-%m-%d %H:%M" }} → Prepends a timestamp with some initial text

  • {{ record['first_name'] | rstrip }} → Removes trailing whitespace from a customer's name

You can read all about the Liquid template system provided by Census here:

Templated fields have a few limitations:

  • Not all sources support templates yet; we are always adding support for new sources!

Conditional field mappings

Where available, Census supports two types of conditional field mappings:

  • Don't Sync Null Values - By default, Census will sync any Null values from the source to the destination. On some connections, you can disable sending any Null values. When disabled, the particular Null value for the property is ignored, but the rest of the recode is synced. Note that the the destinations may handle Null values differently. For example, Salesforce will convert Null values to empty strings, and Braze will delete the property completely when a Null value is synced.

  • Set If Empty - By default, Census will sync any values from the source to the destination, even if the destination already has a value. On some connections, you can set a property to only sync if the destination property is empty. This is useful if you want to set a default value for a property, but not overwrite any existing values.

Creating new fields on your destination object

For some destinations, such as Braze, Customer.io or Iterable (see full list below), you can create new fields directly from the Census app. This means the next time a sync runs, Census will create new properties or attributes for that object in the destination. This is useful if you want to automatically create new fields based on the columns in your source data.

Simply select "Sync All Properties" when setting up a sync.

The following destinations currently support automatically adding new properties:

Braze, Custom API, Customer.io, Google Cloud Storage, Google Sheets, Iterable, Klaviyo, Mixpanel, S3, Salesforce Marketing Cloud, Segment, SFTP, Slack, Vitally, Webhook

Running Syncs

Once syncs are created, you'll probably want to run them! That's where the real magic happens. There's a few things you should know about running and maintaining syncs though including how frequently they should run and what sync history tells you about the health of your sync.

⏱ Schedules and API

You can happily run a sync manually, but that's not all that useful on its own. The real power of Census is having your syncs run automatically. Once you've got your sync up and running, you can configure your sync to run automatically in three ways:

Pick the sync execution trigger that makes for your connection and Census will keep the data flowing to your schedule.

Sync History

You can dive deeper into why syncs failed, or what records were invalid from the source, or rejected by the destination, under the Sync History tab.

Failed Syncs Hover over the status label to see a detailed error.

Invalid or rejected records Click the number of invalid or rejected records to see a sample (up to 100), and the reason why they were invalid or rejected.

What is the difference between Invalid and Rejected records?

  • Invalid records are flagged and filtered by Census prior to syncing to your destination. Census will check the source data for records with NULL identifiers and duplicates.

  • Rejected records are records that were sent to the destination, but the destination did not accept them. The sample of rejected records will provide the specified reasons received from the destination.

🔌 Sources and Destinations

To create a sync, you'll need to connect both a source and a destination.

Data Sources

Census supports a variety of data sources for your syncs:

  • Basic Datasets - Connect directly to your data warehouse tables, views, or custom SQL queries

  • Streaming Datasets - Process real-time events from Kafka, Confluent Cloud, and other streaming sources

  • SaaS Datasets - Use data directly from your CRM systems like Salesforce or HubSpot

  • CSV Datasets - Upload and use CSV files for quick testing or one-time syncs

  • Direct Warehouse Access - Sync directly from warehouse tables without creating a dataset

This flexibility allows you to use the right data source for each use case, whether you need batch processing, real-time streaming, or direct access to your SaaS applications.

For instructions on connecting your specific data source, take a look at the Data Sources section on the left.

Data Source Permissions and Read-only Access

Census provides a powerful and customizable Advanced Sync Engine on top of your data sources. To enable all of this without storing your customer data outside your infrastructure, we create a scratch or bookkeeping schema in your data warehouse that we use to cache sync states. This requires write permission to this schema and only this schema. This schema will not be selectable as a source within the Census sync configuration UI.

If you don't have credentials with write access available, you can connect to your warehouse using our Basic Sync Engine which instead stores this state outside the warehouse.

Datasets and segments

Datasets in Census provide a flexible way to organize and prepare your data for syncing. You can create datasets from various sources:

  • Basic Datasets from your data warehouse using SQL or table selection

  • Streaming Datasets for real-time data activation

  • SaaS Datasets from your connected business applications

  • CSV Datasets for simple file uploads

Destination Services

Census helps you keep your data in sync across all the business apps you use to engage with your customers, from advertising and marketing, through sales, all the way to finance. The goal is to keep all of those apps in sync, powered by the exact same set of data.

The process of connecting a service is a little different for each. Sometimes, it's as simple as an OAuth confirmation flow or a straightforward API key. Other times, a few more steps are required.

Once you've connected a source, we'll be able to access a set of objects that service makes available. Typically this is a Contact or a Company, but there are often others and in some cases, even custom defined objects are supported.

All of the connection steps, the objects, and other details for each service are covered here in the docs.

Need Help?

That's it! Now you've got everything you need to know to get the most out of Census syncs, but don't worry, you can still get into the nitty gritty on individual data sources or destinations. And we're always here to help. If you have any questions, just let us know!

PreviousVersion ControlNextTriggering & Scheduling

Last updated 3 months ago

Was this helpful?

Read more about .

Templated fields operate on one record at a time. If you need to bring multiple records together, take a look at , which allow you to use SQL to prepare your source data for syncing, or at the Census , which includes a powerful point-and-click and .

including with Cron

Either option may be right for you, read more about the differences between .

You can also connect dataset integrations like , , or .

Datasets are optional in Census—you can also sync data directly from a data source table or view—but datasets give you a simple way to create authoritative locations for the full set of all of your paying customers, invoices, or whatever other reusable data concept matters for your business. And once you've built your datasets, Census makes it easy to quickly select and sync as well.

And don't forget, our library of connections grows every week! Keep an eye on our for all of our available destinations and let us know if we're missing something you need!

📫
https://github.com/sutrolabs/census-docs/blob/main/structuring-data/liquid-templates.md
Programmatically via API or Orchestration tool
Automatically with dbt Cloud
Basic and Advanced Sync Engines
dbt
Looker
Sigma
Segments
integrations page
sending event data
Datasets
Audience Hub
Schedules
visual segment builder
calculated fields
Also Sync Null Values in the source field editor
Set only if destination field is empty
Simply select "Sync All Properties"
Output of invalid records diagnostic log