Settings

This page is for the Advanced configuration of the Data Pipeline. The following list of the configurations get displayed on the Settings Page:

• Logger

• Default Configuration

• Pod Status

• Data Sync

• Job BaseInfo

Logger

This gives the info and config details of the Kafka topic that is set for logging.

Logger Configuration

Default Configuration

This is for setting the Default component configuration for Low, Medium and, High configuration Pipelines.

Default Configuration

Please Note: The Non-Spark means Docker under the Default Configuration section.

Pod Status

The Pod Status page under the Settings option gives us monitoring capability on the health of the System Components.

System Component Health

Please Note: The system components should be up for proper functionality of the Pipeline module.

Data Sync

Data Sync in Settings is given to globally configure the Data Sync feature. This way we can enable a DB connection and use the same connection in the pipeline without using any extra resources.

DB Sync page under the Pipeline Settings option

• Go to the Settings page.

• Click the Data Sync option.

• The Data Sync page opens.

• Click the Plus icon.

Please Note: We support seven(7) drivers at present

• MongoDB

• Postgres

• MySQL

• MSSQL

• Oracle

• ClickHouse

• Snowflake

• The New Settings dialog box appears.

• Specify all the required connection details

• Click the Save option.

• A notification message appears.

• Enable the action button.

Edit Connection String

• Go to the Settings page.

• Click the Data Sync option - the DB Sync Settings page opens.

• Click the Edit icon.

• The Edit Settings dialog box opens.

• Edit the connection string if required.

• Click the Update option.

Disconnect Connection String

• Go to the Settings page.

• Click the Data Sync option.

• The Data Sync Settings page opens.

• Click on the Disconnect button to disconnect.

• The Disconnect Setting confirmation dialog box appears.

• Click the DISCONNECT option.

• A success notification message appears on the top when DB Sync gets disconnected.

Data Sync in a Pipeline

• All the Pipelines containing the DB Sync Event component get listed under the Pipeline using DB Sync section.

• The user can see the following information:

  • Name: Name of the pipeline where Data Sync is used.

  • Running status: This indicates if the pipeline is active or deactivated.

  • Actions: The user will be able to view the pipeline where Data Sync is used.