Events [Kafka and DB Sync]

Creating a Kafka Event

  • Click the Add New Event icon from the Event Panel.

  • The New Event window opens.

  • Provide a name for the new Event.

  • Select Event Duration: Select an option from the below-given options.

* Short (4 hours)

* Medium (8 hours)

* Long (48 hours)

* Full Day (24 hours)

* Long (48 hours)

* Week (168 hours)

Please Note: The event data gets erased after 7 days if no duration option is selected from the available options. The Offsets expire as well.

  • Provide No. of Partitions (1-10).

  • No. of Outputs: The user can define the number of outputs using this field.

  • Checkmark the 'Is Failover' option to enable the Failover Event.

  • Select a Pipeline using the drop-down menu if you have selected the ‘Is Shared?’ option.

  • Click the Add Event option.

  • A notification message appears.

  • The newly created Event gets added to the Private tab in the Events panel.

Updating a Kafka Event

The user can use the below-given steps to update a Kafka Event.

  • Drag the created Event component to the Pipeline Editor canvas.

  • Click the dragged Event component to open the Basic Info configuration fields.

  • The user can edit the following information (except No. of Partition and Event Duration):

  • Event Name: Modify the event name.

  • No. of Outputs: Set the number of outputs (the maximum allowed number of outputs are 3).

  • ‘Is Failover?’: Enable this option to create a failover event.

  • Select Pipeline: Select one pipeline or multiple pipelines using the drop-down list.

  • Click the Save Event icon to save the changes.

  • The user gets a notification message stating that the pipeline update is a success.

  • The targeted Event component gets updated.

Connecting Event to a Component

  • Drag a (reader) component to the canvas.

  • Configure the parameters of the dragged reader.

  • Drag the Event from the Events Panel.

  • Connect the dragged reader component as the input connection to the dragged Event.

  • Click the Update Pipeline icon to save the pipeline workflow.

Creating a DB Sync Event

  • Navigate to the Pipeline Editor page.

  • Click on the DB Sync tab. Click on the Add New DB Sync icon from the Event Panel.

  • The Create Data Sync window opens.

    • Provide a display name for the new DB Sync.

    • Select Connection id: [pre- defined by the user in the settings page]

    • Click the Save option.

  • Drag and drop DB Sync Event to the workflow editor.

  • Click on the dragged Data Sync component.

  • The Basic Information tab appears with the following fields:

    • Table name: Specify table name.

    • Driver: this field will be pre-selected.

    • Primary Key: This field is optional.

    • Save Mode: select anyone save mode from the drop-down.

    • Click on the Save Data Sync icon to save the DB Sync information.

  • Connect the dragged DB Sync Event to the reader component as displayed below:

Note: In Save mode, we have two options.

i. Append

ii. Upsert: One extra field will be displayed for upsert save mode i.e.: Composite Key.

Last updated