Confluent Cloud
This page describes how to configure Confluent Cloud as a source for Census.
Last updated
This page describes how to configure Confluent Cloud as a source for Census.
Last updated
Census can connect to Confluent Cloud clusters over the public internet. Census always connects to your cluster via TLS.
To read data from your Confluent Cloud cluster, Census needs the following permissions:
Operation | Type | Resource |
---|---|---|
When used as the source for a Live Sync, Census will also create error topics in your cluster for messages that can’t be processed. To do so, Census needs the following permissions:
Operation | Type | Resource |
---|---|---|
Log in to your existing Confluent Cloud account, or start a free trial.
Create a resource API key for Census to use to collect to your Confluent Cloud cluster.
Refer to the Confluent Cloud documentation for step-by-step instructions for creating a resource API key.
Note the API key and secret as you will need them to connect Census to Confluent Cloud.
In Census, navigate to Sources.
Click New Source and select Confluent Cloud.
Enter the hostname of your bootstrap server.
Enter the API key and secret you created in step 2.
Click Connect.
After you create a Confluent Cloud connection, you need to set up message schemas for your topics.
Before you can use a Confluent Cloud topic as the source for a sync, you must define the schema of the messages on the topic.
In Census, navigate to Models.
From the Source dropdown, select your Confluent Cloud connection.
Census automatically pulls the list of topics from your cluster. You can also click Refresh topics to manually refresh the list.
Click on the name of the topic you want to define a schema for.
Select the format of the messages on your topic from the Format drop down. The only supported format is JSON.
Click Import sample message and enter a sample message. Your sample message should be a JSON object containing any top-level fields you want to use in your pipelines. Make sure the values of the fields in your sample message match the data type of the values in your actual messages.
Census does not handle sample messages as customer data. Do not include real data in a sample message.
Click Save Topic. Review the listed properties and types to ensure the sample message you provided was interpreted as intended.
Contact us via support@getcensus.com or start a conversation with us via the in-app chat.
Read
Topic
Any topic you want to sync data from
Read
Group
census_group_*
Create, Write
Topic
census_*