Enrichment Component

Enrichment Component

Enrich your data from the master table/collection with few simple steps.This component helps users to enrich the incoming data from an in-Event by querying lookup table in the RDBMS and MongoDB reader components.All component configurations are classified broadly into the following sections:

• ​Basic ​

• Metadata

• ​Resource Configuration​

• ​Connection Validation​

Follow the steps given in the demonstration to configure the Enrichment component.

Configuring Enrichment Component

Please Note: The supported drivers are as mentioned below:

• MySQL

• MS-SQL

• MongoDB

• PostgreSQL

• Oracle

• ClickHouse

• Snowflake

Steps to configure the Enrichment component

• Drag and drop the Enrichment Component to the Workflow Editor.

• Create two Events and drag them to the Workspace.

• Connect the input event and the output event (The data in the input event can come from any Ingestion, Reader, or shared events).

• Click the Enrichment Component to get the component properties tabs.

Using the Enrichment component as a part of Pipeline Workflow

Basic Information

The Basic Information tab opens by default.

• Select an Invocation type from the drop-down menu to confirm the running mode of the reader component. Select either Real-Time or Batch option from the drop-down menu.

• Deployment Type: It displays the deployment type for the component. This field comes pre-selected.

• Container Image Version: It displays the image version for the docker container. This field comes pre-selected.

• Failover Event: Select a failover Event from the drop-down menu.

• Batch Size: Provide the maximum number of records that you want to be processed in one execution cycle.

Meta Information

Open the Meta Information tab and fill in all the connection-specific details for the Enrichment Component.

• Driver (*): Select Database type (MYSQL, MSSQL, Oracle, Postgres, MongoDB, ClickHouse, Snowflake)

• Port (*): Host server port number

• Host IP Address (*): IP Address

• Username (*): Username for Authentication.

• Password (*): Password for Authentication.

• Database Name (*): Provide the Database name.

• Table Name: Provide the table name to read the data.

• Refresh rate: The value of this field has to be provided in seconds. It refreshes the master table and fetch the changes in every cycle of the given value. For example, if refresh rate value is given as 3600 seconds, it will refresh the master table in every 3600 seconds and fetch the changes.

• Connection Type: This option will show if the user selects MongoDB from the Driver field. A User can configure the MongoDB driver via two connection types (Standard or SRV) that are explained below:

  • Standard - Port field does appear with the Standard connection type option.

  • SRV - Port field does not appear with the SRV connection type option.

• Conditions: Select conditions type (Remove or Blank option).

• Master table query: Write a SQL query to get the data from the table which has been mentioned in the Table name field.

• Query: Write a SQL query for the in-event data.

Configuring meta information of enrichment component

Selected Columns

The users can select some specific columns from the table to read data instead of selecting a complete table; this can be achieved via the Selected Columns section. Select the columns which you want to read and if you want to change the name of the column, then put that name in the alias name section otherwise keep the alias name the same as of column name and then select a Column Type from the drop-down menu.

or

Use the Download Data and Upload File options to select the desired columns.

• Upload File: The user can upload the existing system files (CSV, JSON) using the Upload File icon (file size must be less than 2 MB).

• Download Data (Schema): Users can download the schema structure in JSON format by using the Download Data icon.

Saving the Component Configuration

• After doing all the configurations click the Save Component in Storage icon provided in the reader configuration panel to save the component.

• A notification message appears to inform about the component configuration success.

Query Example

select * from inputDF left join departments on inputDF.department_id = departments.department_id;

select * from inputDF left join departments on inputDF.department_id = departments.department_id;

Please Note:

• The data of previous event is taken as inputDF as the table name in the query field of enrichment component as shown in the above query example.

• Remove [works like an inner join]– This option completely removes the row If any row of incoming data is not matching with query condition of the lookup table. Only passes the matching records.

• Blank – This option sets blank for the cell of a row if complete incoming data from the in-event and matching rows from the lookup table.