Kafka Topic example config

Overview

This example syncs from a Kafka Topic source to a Cinchy Table target.

We want to sync the following data from Kafka and map it to the appropriate column in the "Sync Target 2" table in the "Kafka Sync" domain.

Kafka Source	Cinchy Column
$.employeeId	Employee Id
$.name	Name

UI Example

This is what the Connections UI will look like with the aforementioned example parameters and data.

Source tab

Your source tab should be set to "Kafka Topic" and have the following information (Image 1):

Tip: Click on an image in this document to enlarge it.

Column 1 (Standard Column) Parameters	Example Data
Name	$.employeeid
Alias	Employee Id
Data Type	Number

Column 2 ( Standard Column) Parameters	Example Data
Name	$.name
Alias	Name
Data Type	Text
Trim Whitespace	True

Image 1: Inserting your source data

Destination tab

Your destination tab should be set to Cinchy Table, and have the following information (Image 2):

Domain: The domain where your destination table resides. This example uses the "Kafka Sync" domain.

Table: The name of your destination table. This example uses the "Sync Target 2" table.

Degree of Parallelism: This is the number of parallel batch inserts and updates that can be run. Set this to 1 for our example.

Column 1 (Standard Column) Parameters	Example Data
Source Column	Employee Id
Target Column	Employee Id

Column 2 (Standard Column) Parameters	Example Data
Source Column	Name
Target Column	Name

Image 2: Inserting your destination data

Sync behaviour

Under the Sync Behaviour tab, we want to use the following parameters:

• Synchronization Pattern: Full File

• Sync Key Column Reference Name: Employee Id

• New Record Behaviour: Insert

• Dropped Record Behaviour: Delete

• Change Record Behavior: Update

XML example

The following code is what the XML for our example connection would look like:

<?xml version="1.0" encoding="utf-16"?>
<BatchDataSyncConfig name="Kafka Sync" version="1.0.0" xmlns="http://www.cinchy.co">
    <KafkaTopicDataSource>
        <Schema>
            <Column name="$.employeeId" label="Employee Id" dataType="Number" isMandatory="false" validateData="false"/>
            <Column name="$.name" label="Name" dataType="Text" trimWhitespace="true" isMandatory="false" validateData="false"/>
        </Schema>
    </KafkaTopicDataSource>
    <CinchyTableTarget reconcileData="true" domain="Kafka Sync" table="Sync Target 2" suppressDuplicateErrors="false" degreeOfParallelism="1">
        <ColumnMappings>
            <ColumnMapping sourceColumn="$.employeeId" targetColumn="Employee Id"/>
            <ColumnMapping sourceColumn="$.name" targetColumn="Name"/>
        </ColumnMappings>
        <SyncKey>
            <SyncKeyColumnReference name="Employee Id"/>
        </SyncKey>
        <NewRecordBehaviour type="INSERT"/>
        <DroppedRecordBehaviour type="DELETE"/>
        <ChangedRecordBehaviour type="UPDATE"/>
        <PostSyncScripts/>
    </CinchyTableTarget>
</BatchDataSyncConfig>