Triggering & Scheduling
You can happily run a sync manually, but that's not all that useful on its own. The real power of Census is having your syncs run automatically. Once you've got your sync up and running, you can configure your sync to run automatically in several ways:
- Schedule
- Programmatically via API
- Via Airflow
- Via dbt Cloud
- In a Sequence
Schedules let you specify a time and frequency that Census can use to run your sync automatically. You can choose options from weekly all the way to Continuous, which means Census checks your source roughly every minute for new changes.
To remove a schedule from a sync, click the edit icon and select Manual from the drop down list.
The last scheduling option is Cron. Cron lets you schedule syncs on arbitrary schedules such as every 3 hours, or only week days. Census accepts standard Cron definitions up to minute granularity (second-level granularity cron definitions are not supported). A Cron schedule is specified by a series of five values, separated by spaces. In order, the values are Minute-of-hour, hours-of-day, Days-of-month, Months-of-year, and Day-of-week. The timezone is UTC.
Here's a few examples of common Cron schedules:
- Once an hour, on the hour, every four hours:
0 */4 * * * - Hourly during weekdays:
0 * * * 1,2,3,4,5 - Minutely on the 5th, 6th, and 7th hours of the day in UTC timezone:
0 5,6,7 * * *, executes sync jobs at 9pm, 10pm and 11pm PST or 12am, 1am and 2am EST
If you're using dbt Cloud to run your dbt project, you can configure Census to automatically run syncs whenever your models have been rebuilt. Note that using dbt Cloud to trigger syncs is complementary to using dbt Models but is not required.
- You may use both User API keys and Service Account tokens. We strongly recommend you use Service Account tokens.
- As of May 2021, according to dbt Cloud, tokens may only be created or modified by users with Account Admin (Enterprise plan) or Owner (Team plan) permissions on an account, so that this token has these privileges.
With your token in hand, you can now connect dbt Cloud to your dbt project.
- 1.Visit the Settings page and select the Integrations tab
- 2.Then copy your dbt Cloud API key, Verify your key is correct, and Save your settings.
Now, you'll be able to use a dbt Cloud job to trigger syncs. Visit the Configuration tab of any of your syncs.
With this enabled, the sync will run automatically as soon as dbt Cloud has successfully finished the job you've selected. Nothing more to do!
Each sync can also be triggered via API. On the sync configuration page, you can access the trigger API endpoint for the sync.
An empty HTTP POST call to this endpoint will trigger the sync (no need to provide any data in the body). You can use this API to automatically trigger Census syncs as part of your data pipeline, running syncs once the models they depend on have been rebuilt.
Request
Response
curl -X POST https://bearer:[API_TOKEN]@app.getcensus.com/api/v1/syncs/[SYNC_ID]/trigger
{
"status": "success",
"data": {
"sync_run_id": 1234567890
}
}
Response Property | Description |
|---|---|
status | success or error indicating whether the sync was triggered. |
data | Present if successful. An object containing the sync_run_id |
message | Present if error. Contains message describing the error. |
You can use the
sync_run_id returned when successfully triggering a sync execution and get status on its progress or determine when it has completed.Request
Response
curl https://bearer:[API_TOKEN]@app.getcensus.com/api/v1/sync_runs/[SYNC_RUN_ID]
{
"status": "success",
"data": {
"error_message": null,
"records_failed": 15,
"records_invalid": 5,
"records_processed": 100,
"records_updated": 80,
"status": "completed"
}
}
Response Property | Description |
|---|---|
status | success if sync_run was found |
data | Present if successful. Contains the following properties: |
status |
|
records_processed | Number of new or updated records retrieved from the source |
records_updated | Number of records successfully sent to the destination |
records_invalid | Number of records skipped by Census because of data quality issues. |
records_failed | Number of records rejected by the destination. |
Heads up: Unlike Airflow 2, Airflow 1 doesn't show any non-"core" providers (i.e. Census!) in the connections UI. If you're using Airflow 1, Census should be configured as an "HTTP" Conn Type, as documented here.
Whether you're using Astronomer or self-hosting your own instance, you can use Census's Airflow Provider to trigger and monitor Census syncs.
Visit the Census Airflow Provider GitHub repository for more details on how to use it for your project.
If your syncs have dependencies and you'd like to organize them to run in order, you can use a Sequence. A Sequence runs a dependent sync whenever its specified parent sync completes successfully. Sequences can be found on the sync configurations page:
Sequences do not currently support specifying multiple parent syncs. If you are interested in multi-parent functionality, please email [email protected].
Last modified 2mo ago