Events [Kafka and DB Sync]

Creating a Kafka Event

• Navigate to the Pipeline Editor page.

• Click the Add New Event icon from the Event Panel.

• The New Event window opens.

• Provide a name for the new Event.

• Select Event Duration: Select an option from the below-given options.

* Short (4 hours)

* Medium (8 hours)

* Long (48 hours)

* Full Day (24 hours)

* Long (48 hours)

* Week (168 hours)

Please Note: The event data gets erased after 7 days if no duration option is selected from the available options. The Offsets expire as well.

• Provide No. of Partitions (1-10).

• No. of Outputs: The user can define the number of outputs using this field.

• Checkmark the 'Is Failover' option to enable the Failover Event.

• Select a Pipeline using the drop-down menu if you have selected the ‘Is Shared?’ option.

• Click the Add Event option.

• A notification message appears.

• The newly created Event gets added to the Private tab in the Events panel.

Updating a Kafka Event

The user can use the below-given steps to update a Kafka Event.

• Drag the created Event component to the Pipeline Editor canvas.

• Click the dragged Event component to open the Basic Info configuration fields.

• The user can edit the following information (except No. of Partition and Event Duration):

• Event Name: Modify the event name.

• No. of Outputs: Set the number of outputs (the maximum allowed number of outputs are 3).

• ‘Is Failover?’: Enable this option to create a failover event.

• Select Pipeline: Select one pipeline or multiple pipelines using the drop-down list.

• Click the Save Event icon to save the changes.

• The user gets a notification message stating that the pipeline update is a success.

• The targeted Event component gets updated.

Flushing Events

Kafka events can be flushed to delete all records present. Flushing an event retains the offsets of the topic by setting the start-offset value to the end-offset. Events can be flushed by using the "Flush Event" button beside the respective event in the event panel, and all events can be flushed at once by using the "Flush All" button. This button is present at the top of the event panel.

Connecting Event to a Component

• Drag a (reader) component to the canvas.

• Configure the parameters of the dragged reader.

• Drag the Event from the Events Panel.

• Connect the dragged reader component as the input connection to the dragged Event.

• Click the Update Pipeline icon to save the pipeline workflow.

Creating a DB Sync Event

Support is available for the following drivers:

1. ClickHouse

2. MongoDB

3. MS-SQL

4. MySQL

5. Oracle

6. PostgreSQL

• Navigate to the Pipeline Editor page.

• Click on the DB Sync tab. Click on the Add New DB Sync icon from the Event Panel.

• The Create Data Sync window opens.

  • Provide a display name for the new DB Sync.

  • Select Connection id: [pre- defined by the user in the settings page]

  • Click the Save option.

• Drag and drop DB Sync Event to the workflow editor.

• Click on the dragged Data Sync component.

• The Basic Information tab appears with the following fields:

  • Table name: Specify table name.

  • Driver: this field will be pre-selected.

  • Primary Key: This field is optional.

  • Save Mode: select anyone save mode from the drop-down.

  • Click on the Save Data Sync icon to save the DB Sync information.
• Connect the dragged DB Sync Event to the reader component as displayed below:

Note: In Save mode, we have two options.

i. Append

ii. Upsert: One extra field will be displayed for upsert save mode i.e.: Composite Key.