Cinchy Platform Documentation
Cinchy v5.6
Cinchy v5.6
  • Data Collaboration Overview
  • Release Notes
    • Release Notes
      • 5.0 Release Notes
      • 5.1 Release Notes
      • 5.2 Release Notes
      • 5.3 Release Notes
      • 5.4 Release Notes
      • 5.5 Release Notes
      • 5.6 Release Notes
  • Getting Help
  • Cinchy Glossary
  • Frequently Asked Questions
  • Deployment Guide
    • Deployment Installation Guides
      • Deployment Planning Overview and Checklist
        • Deployment Architecture Overview
          • Kubernetes Deployment Architecture
          • IIS Deployment Architecture
        • Deployment Prerequisites
          • Single Sign-On (SSO) Integration
            • Enabling TLS 1.2
            • Configuring ADFS
            • AD Group Integration
      • Kubernetes Deployment Installation
        • Disabling your Kubernetes Applications
        • Changing your File Storage Configuration
        • Configuring AWS IAM for Connections
        • Using Self-Signed SSL Certs (Kubernetes Deployments)
        • Deploying the CLI (Kubernetes)
      • IIS Deployment Platform Installation
    • Upgrade Guides
      • Upgrading Cinchy Versions
        • Cinchy Upgrade Utility
        • Kubernetes Upgrades
          • v5.1 (Kubernetes)
          • v5.2 (Kubernetes)
          • v5.3 (Kubernetes)
          • v5.4 (Kubernetes)
          • v5.5 (Kubernetes)
          • v5.6 (Kubernetes)
          • Upgrading AWS EKS Kubernetes Version
          • Updating the Kubernetes Image Registry
          • Upgrading AKS (Azure Kubernetes Service)
        • IIS Upgrades
          • v4.21 (IIS)
          • v4.x to v5.x (IIS)
          • v5.1 (IIS)
          • v5.2 (IIS)
          • v5.3 (IIS)
          • v5.4 (IIS)
          • v5.5 (IIS)
          • v5.6 (IIS)
      • Upgrading from v4 to v5
  • Guides for Using Cinchy
    • User Guides
      • Overview of the Data Browser
      • The Admin Panel
      • User Preferences
        • Personal Access Tokens
      • Table Features
      • Data Management
      • Queries
      • Version Management
        • Versioning Best Practices
      • Commentary
    • Builder Guides
      • Best Practices
      • Creating Tables
        • Attaching Files
        • Columns
        • Data Controls
          • Data Entitlements and Access Controls
          • Data Erasure
          • Data Compression
        • Formatting Rules
        • Indexing and Partitioning
        • Linking Data
        • Table and Column GUIDs
        • System Tables
      • Deleting Tables
        • Restoring Tables, Columns, and Rows
      • Saved Queries
      • CinchyDXD Utility
        • Building the Data Experience (CinchyDXD)
        • Packaging the Data Experience (CinchyDXD)
        • Installing the Data Experience (CinchyDXD)
        • Updating the Data Experience (CinchyDXD)
        • Repackaging the Data Experience (CinchyDXD)
        • Reinstalling the Data Experience (CinchyDXD)
      • Multi-Lingual Support
      • Integration Guides
    • Administrator Guide
    • Additional Guides
      • Monitoring and Logging on Kubernetes
        • Grafana
        • Opensearch Dashboards
          • Setting up Alerts
        • Monitoring via ArgoCD
      • Maintenance
      • System Properties
      • Enable Data At Rest Encryption
      • MDQE
      • Application Experiences
        • Network Map
          • Custom Node Results
          • Custom Results in the Network Map
        • Setting Up Experiences
  • API Guide
    • API Overview
      • API Authentication
      • API Saved Queries
      • ExecuteCQL
      • Webhook Ingestion
  • CQL
    • The Basics of CQL
      • CQL Examples
      • CQL Functions Master List
      • CQL Statements Overview
        • Cinchy DML Statements
        • Cinchy DDL Statements
      • Cinchy Supported Functions
        • Cinchy Functions
        • Cinchy System Values
        • Cinchy User Defined Functions
          • Table-Valued Functions
          • Scalar-Valued Functions
        • Conversion Functions
        • Date and Time Types and Functions
          • Return System Date and Time Values
          • Return Date and Time Parts
          • Return Date and Time Values From Their Parts
          • Return Date and Time Difference Values
          • Modify Date and Time Values
          • Validate Date and Time Values
        • Logical Functions
        • Mathematical Functions
        • String Functions
        • Geometry and Geography Data Type and Functions
          • OGC Methods on Geometry & Geography Instances
          • Extended Methods on Geometry & Geography Instances
        • Full Text Search Functions
        • Connections Functions
        • JSON Functions
  • Meta Forms
    • Introduction to Meta-Forms
    • Meta-Forms Deployment Installation Guide
      • Deploying Meta-Forms (Kubernetes)
      • Deploying Meta-Forms (IIS)
    • Forms Data Types
    • Meta-Forms Builders Guides
      • Creating a Dynamic Meta-Form (Using Tables)
      • Creating a Dynamic Meta-Form Example (Using Form Designer)
      • Adding Links to a Form
      • Rich Text Editing in Forms
  • Data Syncs
    • Getting Started with Data Syncs
    • Installation & Maintenance
      • Prerequisites
      • Installing Connections
      • Installing the Worker/Listener
      • Installing the CLI and the Maintenance CLI
    • Building Data Syncs
      • Types of Data Syncs
      • Common Design Patterns
      • Sync Behaviour
      • Columns and Mappings
        • Calculated Column Examples
      • Listener Configuration
      • Advanced Settings
        • Filters
        • Parameters
        • Auth Requests
        • Request Headers
        • Post Sync Scripts
        • Pagination
      • Batch Data Sync Example
      • Real-Time Sync Example
      • Scheduling a Data Sync
      • Connection Functions
    • CLI Commands List
    • Error Logging and Troubleshooting
    • Supported Data Sync Sources
      • Cinchy Event Broker/CDC
        • Cinchy Event Broker/CDC XML Config Example
      • Cinchy Table
        • Cinchy Table XML Config Example
      • Cinchy Query
        • Cinchy Query XML Config Example
      • Copper
      • DB2 (Query and Table)
      • Dynamics 2015
      • Dynamics
      • DynamoDB
      • File Based Sources
        • Binary File
        • Delimited File
        • Excel
        • Fixed Width File
        • Parquet
      • Kafka Topic
        • Kafka Topic Example Config
        • Apache AVRO Data Format
      • LDAP
      • MongoDB Collection
        • MongoDB Collection Source Example
      • MongoDB Collection (Cinchy Event Triggered)
      • MS SQL Server (Query and Table)
      • ODBC Query
      • Oracle (Query and Table)
      • Polling Event
        • Polling Event Example Config
      • REST API
      • REST API (Cinchy Event Triggered)
      • SAP SuccessFactors
      • Salesforce Object (Bulk API)
      • Salesforce Platform Event
      • Salesforce Push Topic
      • Snowflake
        • Snowflake Source Example Config
      • SOAP 1.2 Web Service
    • Supported Data Sync Destinations
      • Cinchy Table
      • DB2 Table
      • Dynamics
      • Kafka Topic
      • MongoDB Collection
      • MS SQL Server Table
      • Oracle Table
      • REST API
      • Salesforce Object
      • Snowflake Table
      • SOAP 1.2 Web Service
    • Supported Real-Time Sync Stream Sources
      • Cinchy Event Broker/CDC
      • Data Polling
      • Kafka Topic
      • MongoDB
      • Salesforce Push Topic
      • Salesforce Platform Event
  • Other Resources
    • Angular SDK
    • JavaScript SQK
Powered by GitBook
On this page
  • 1. Overview
  • 2. UI Example
  • 2.1 Source Tab
  • 2.2 Destination Tab
  • 2.3 Sync Behaviour
  • 3. XML Example

Was this helpful?

Export as PDF
  1. Data Syncs
  2. Supported Data Sync Sources
  3. Kafka Topic

Kafka Topic Example Config

1. Overview

In this example, we are syncing from a Kafka Topic source to a Cinchy Table target.

We want to sync the following data from Kafka and map it to the appropriate column in the "Sync Target 2" table in the "Kafka Sync" domain.

Kafka Source
Cinchy Column

$.employeeId

Employee Id

$.name

Name

2. UI Example

This is what the Connections UI will look like with the aforementioned example parameters and data.

2.1 Source Tab

Your source tab should be set to "Kafka Topic" and have the following information (Image 1):

Tip: Click on an image in this document to enlarge it.

Column 1 (Standard Column) Parameters
Example Data

Name

$.employeeid

Alias

Employee Id

Data Type

Number

Column 2 ( Standard Column) Parameters
Example Data

Name

$.name

Alias

Name

Data Type

Text

Trim Whitespace

True

2.2 Destination Tab

Your destination tab should be set to "Cinchy Table", and have the following information (Image 2):

Domain: The domain where your destination table resides. In our example we are using the "Kafka Sync" domain.

Table: The name of your destination table. In our example we are using the "Sync Target 2" table.

Degree of Parallelism: This is the number of parallel batch inserts and updates that can be run. Set this to 1 for our example.

Column 1 (Standard Column) Parameters
Example Data

Source Column

Employee Id

Target Column

Employee Id

Column 2 (Standard Column) Parameters
Example Data

Source Column

Name

Target Column

Name

2.3 Sync Behaviour

Under the Sync Behaviour tab, we want to use the following parameters:

  • Synchronization Pattern: Full File

  • Sync Key Column Reference Name: Employee Id

  • New Record Behaviour: Insert

  • Dropped Record Behaviour: Delete

  • Change Record Behavior: Update

3. XML Example

The following code is what the XML for our example connection would look like:

<?xml version="1.0" encoding="utf-16"?>
<BatchDataSyncConfig name="Kafka Sync" version="1.0.0" xmlns="http://www.cinchy.co">
    <KafkaTopicDataSource>
        <Schema>
            <Column name="$.employeeId" label="Employee Id" dataType="Number" isMandatory="false" validateData="false"/>
            <Column name="$.name" label="Name" dataType="Text" trimWhitespace="true" isMandatory="false" validateData="false"/>
        </Schema>
    </KafkaTopicDataSource>
    <CinchyTableTarget reconcileData="true" domain="Kafka Sync" table="Sync Target 2" suppressDuplicateErrors="false" degreeOfParallelism="1">
        <ColumnMappings>
            <ColumnMapping sourceColumn="$.employeeId" targetColumn="Employee Id"/>
            <ColumnMapping sourceColumn="$.name" targetColumn="Name"/>
        </ColumnMappings>
        <SyncKey>
            <SyncKeyColumnReference name="Employee Id"/>
        </SyncKey>
        <NewRecordBehaviour type="INSERT"/>
        <DroppedRecordBehaviour type="DELETE"/>
        <ChangedRecordBehaviour type="UPDATE"/>
        <PostSyncScripts/>
    </CinchyTableTarget>
</BatchDataSyncConfig>
PreviousKafka TopicNextApache AVRO Data Format

Last updated 2 years ago

Was this helpful?

Image 1: Inserting your source data
Image 2: Inserting your destination data