Kafka
Last updated
Last updated
Census can connect to Kafka clusters over the public internet. Census always connects to your cluster via TLS using static IP addresses that you can add to an allow list.
Census supports the following SASL authentication mechanisms for Kafka:
Plain
SCRAM SHA-256
SCRAM SHA-512
To read data from your Kafka cluster, Census needs the following permissions:
Operation | Type | Resource |
---|---|---|
When used as the source for a Live Sync, Census will also create error topics in your cluster for messages that can’t be processed. To do so, Census needs the following permissions:
Operation | Type | Resource |
---|---|---|
In Census, navigate to Sources.
Click New Source and select Kafka.
Enter the hostname and port number of your bootstrap server.
Select the SASL authentication method your brokers are configured to use.
Enter the username and password Census should use to authenticate with your brokers.
Click Connect.
After you create a Kafka connection, you need to set up message schemas for your topics.
Before you can use a Kafka topic as the source for a sync, you must define the schema of the messages on the topic.
In Census, navigate to Datasets.
From the Source dropdown, select your Kafka connection.
Census automatically pulls the list of topics from your cluster. You can also click Refresh topics to manually refresh the list.
Click on the name of the topic you want to define a schema for.
Select the format of the messages on your topic from the Format drop down. The only supported format is JSON.
Click Import sample message and enter a sample message. Your sample message should be a JSON object containing any top-level fields you want to use in your pipelines. Make sure the values of the fields in your sample message match the data type of the values in your actual messages.
Census stores message samples as part of the Dataset definition. Do not include customer PII or otherwise sensitive data in a sample message.
Click Save Topic. Review the listed properties and types to ensure the sample message you provided was interpreted as intended.
Contact us via support@getcensus.com or start a conversation with us via the in-app chat.
Read
Topic
Any topic you want to sync data from
Read
Group
census_group_*
Create, Write
Topic
census_*