Comment on page
This page describes how to use Census with a Kafka destination.
In this guide, we will show you how to connect your Kafka destination to Census.
- 1.Go to the Destinations tab and click New Destination.
- 2.Select Kafka from the menu.
- 3.To connect to Kafka, you'll need to provide the following credentials:
- Bootstrap Servers: One (or more) host/port combinations to use for establishing the initial connection to the Kafka cluster. For example,
kafka-1.example.com:9092,kafka-2.example.com:9092. Sometimes called
- Protocol: The protocol to use for communicating with Kafka. Currently, only
- Mechanism: SASL mechanism to use for authentication, such as
- Username: The username of your Kafka instance. Called API Key by Confluent.
- Password: The password of your Kafka instance. Called API Secret by Confluent.
Sending a Kafka Message involves a number of configuration properties, in addition to the actual message payload. Census allows you to configure these properties in the Advanced Configuration section of the destination setup, as well as the Mappings section of the sync setup.
Within Advanced Configuration, you can set the following properties:
- Preserve message ordering: If enabled, Census will send messages to Kafka in the same order they were received by Census. Otherwise, Census will send messages to Kafka as fast as possible, which may result in messages being received out of order.
- Created/Updated/Deleted record tag: Census will automatically add an
operationproperty to each message sent to Kafka (see below). You can use these properties to change the value of the
operationproperty, which can be useful for downstream processing.
- Message Template: You can optionally provide a mustache-formatted template to structure your message. By default with no template provided, Census will send all properties as a single flat JSON object.
Within the Mappings section of the sync setup, you can set the following properties:
- Topic (required): The Kafka topic to send the message to.
- Key: The key to use for the message.
- Partition: The partition to send the message to.
- Partition Key: The partition key to use for the message.
- Timestamp: The timestamp to use for the message.
By default, Census will structure messages as a single flat JSON object. You can optionally provide a Message Template to structure your message in a different way. In addition to the properties you provide, Census will automatically add the following properties to each message:
- operation: The operation that triggered the sync.
- synced_at: The time the sync was triggered.