Data Sync Events

This page offers a detailed overview of the connecting components within the BDB Data Pipeline, specifically focusing on Data Sync Events.

What is a Data Sync Event?

A Data Sync Event in the Data Pipeline module is designed to simplify and accelerate the process of writing data directly to a configured database without the need to introduce additional components in the pipeline flow. This streamlines the process by reducing the number of writer components within the pipeline, offering a centralized and seamless method for data writing.

Important Notes:

• Caution: Before modifying the Data Sync settings, it is important to assess the potential impact on any pipelines utilizing this feature.

• Admin Access: Administrators can view a list of pipelines that rely on Data Sync, enabling them to make well-informed decisions before making any changes to the configuration.

Benefits of using the Data Sync Event

• It doesn't need a Kafka event to read the data. It can be connected with any component to read the data, and it writes it to the tables of the respective databases.

• Pipeline complexity is reduced because the Kafka event and writer are not needed in the pipeline.

• Since writers are not used, the resource consumption is low.

Pre-requisites: Before creating the Data Sync Event, the user has to configure the driver settings in the Data Sync section under the Pipeline Settings admin module.

• If no driver configuration is done for the Data Sync, a clickable link "Configure data sync settings" will appear under the Data Sync section of the Events tab, redirecting the user to configure a driver for the Data Sync.

  
• Clicking the link redirects the user to the Data Sync page, located within the Pipeline Settings Admin module.

  • Click the Create Data Sync Connection to configure new driver settings for Data Sync.

    
  • The configured driver for the Data Sync will be listed under the Create Sync Event dialog window of the Events tab. For example, the "Mongo NON SSL" driver in the given image.

    
• Once the Data Sync settings are configured, multiple Data Sync events can be created for the same driver configuration, and the data can be written to multiple tables.

Please refer to the Data Sync section of the Pipeline Settings to understand the driver configuration process in detail.

Creating and Configuring a Data Sync Event (from the Events Panel)

Follow the steps below to create and configure a Data Sync Event in the Pipeline Workflow Editor.

1

Access the Components & Events Panel

• Navigate to the Pipeline Workflow Editor page.

• Click the Add Component/Event icon.

• In the side panel, select the Events tab.

• Expand the DB Sync section.

• Click the Add New Data Sync icon.

  

2

Create a Data Sync Event

• The Create Sync Event window opens.

• Provide the required details:

  • Display Name: Enter a name for the new Data Sync.

  • Driver: Select a driver from the drop-down list (only pre-configured drivers appear).

  • Is Failover: Check this option if you want to create a failover Data Sync. (In this example, leave it unchecked.)

• Click Add Sync Event.

  

• A new Data Sync is created and appears under the DB Sync section in the Events tab.

Please note:

• Only the drivers configured under the Settings page are available for selection.

• If the "Is Failover" option is enabled, the Data Sync will be created as a failover Data Sync.

3

Add the Data Sync Event to the Workflow

• Drag and drop the newly created Data Sync Event onto the Pipeline Workflow Editor canvas.

• The Data Sync Event automatically connects with the nearest component in the pipeline workflow.

  

4

Configure Data Sync Properties

• Select the Data Sync Event on the canvas.

• Open the Basic Info tab.

• Provide the following information:

  • Table Name: Specify the database table name.

  • Driver: Select a driver using the dropdown menu. All the configured drivers for the data sync event will be listed.

  • Save Mode: Select one of the following from the drop-down list:

    • Append: Adds new records without affecting existing ones.

    • Upsert: Updates existing records based on a key and inserts new ones.

  • Composite Key: (Optional) Appears only when Upsert is selected as the save mode. Use it to define the key fields for record updates.

• Click the Save Data Sync icon to save your configuration.

  

• A success message confirms that the Data Sync information has been saved.

5

Update and Activate the Pipeline

• Update the pipeline with the new configuration.

• Activate the pipeline.

• A success notification confirms that the logs are loaded.

6

View Logs

• Click the Logs icon.

• Open the Logs tab.

• Review the logs to see details of the data being written to the output event (the connected DB Sync Event).

  

7

Please Note: The supported drivers for the Data Sync component are as listed below:

• ClickHouse

• MongoDB

• MSSQL

• MySQL

• Oracle

• PostgreSQL

• Snowflake

• Redshift

Auto Connecting Data Sync

This feature automatically connects the Kafka Event to the component when it is dragged from the Events panel. To use this feature, users must ensure that the "Auto connect components on drag" option is enabled in the Events panel (it comes enabled in the default setting).

Mapping Failover Data Sync to a Component

The Failover Data Sync Event component is a safeguard mechanism in resilient pipeline designs. It ensures data reliability and consistency across systems during failovers, enabling smooth transitions and uninterrupted operations in mission-critical environments.

Please note: To create a Failover Data Sync Event, follow the same steps outlined for creating a regular Data Sync Event. Additionally, enable the Is Failover option to designate it as a failover Data Sync Event.

Pipeline

Prerequisites

• A pipeline workflow is available in the Pipeline Workflow Editor.

• The Failover Data Sync Event is already available under the DB Sync section in the Events tab.

Follow the steps below to configure and map a Failover Data Sync Event:

1

Add Failover Data Sync to Workflow

• Navigate to the Events tab in the Pipeline Workflow Editor.

• Locate the Failover Data Sync Event under the DB Sync section.

• Drag and drop the Failover Data Sync event into the workflow editor canvas.

2

Configure Failover Data Sync

• Click on the Failover Data Sync event in the workflow editor.

• In the Basic Info tab, provide the following details:

  • Table Name: Specify the table where failed data should be written.

  • Save Mode: Choose from the available options:

    • Append – Adds failed records to the existing table.

    • Upsert – Inserts or updates records in the target table.

• Click the Save Data Sync icon.

• A success message confirms that the Failover Data Sync has been configured.

3

Map Failover Data Sync with Pipeline Component

• Open the pipeline component in the existing workflow that requires failover handling.

• In the Basic Information tab of the selected component, locate the Failover Event drop-down menu.

• Select the configured Failover Data Sync Event.

• Save the component.

• Upon success, update the pipeline.

• Activate the pipeline workflow once the update is successful.

4

Validate Configuration & Logs

• Open the Log Panel after pipeline activation.

• Confirm pipeline activation via the success message.

• Navigate to the Logs tab within the panel.

• Verify notification messages indicating that the Failover Data Sync Event has been triggered.

  • For example, if mapped with a ClickHouse Reader component, any failure will result in the failed records being written to the designated failover table.

• The logs explicitly confirm when data is written to the failover table.

Important Notes:

• The UI does not display configuration details when hovering over the Failover Data Sync Event.

• However, when a user hovers over the Failover Data Sync Event component in either the pipeline canvas or component view, the pipeline components mapped to this event are visually highlighted.

  • This interactive highlighting provides a clear indication of the linkage between the Failover Data Sync and the components it supports.

    

✅ At this point, the Failover Data Sync Event is fully configured and mapped, ensuring data resiliency in case of pipeline component failures.

Adding a Data Sync Event to a Pipeline Workflow

The Data Sync Event feature allows users to establish a direct connection between a pipeline component and a synchronization event. This can be performed quickly by using the context menu on the pipeline workspace canvas.

Steps to Add a Data Sync Event

1

Right-click on the component

• On the pipeline workspace canvas, right-click on the desired pipeline component that has already been dragged onto the canvas.

2

Select Add Data Sync

• From the context menu, choose the Add Data Sync option.

  

3

Open the Create Sync Event dialog

• The Create Sync Event dialog box will appear automatically.

4

Add the Sync Event

• In the dialog box, click Add Sync Event to create the synchronization event.

5

Verify the Connection

• The newly created Data Sync Event will be directly connected to the selected component within the pipeline workflow.