Enrichment Component

Enrich your data from the master table/collection with few simple steps. This component helps users to enrich the incoming data from an in-Event by querying lookup table in the RDBMS and MongoDB reader components. All component configurations are classified broadly into the following sections:

• ​​Basic Information​​

• Meta Information

• ​Resource Configuration​​

• ​Connection Validation​

Follow the steps given in the demonstration to configure the Enrichment component.

Configuring Enrichment Component

Supported Drivers

• MySQL

• MS-SQL

• MongoDB

• PostgreSQL

• Oracle

• ClickHouse

• Snowflake

• Redshift

Please Note: If the selected driver is MongoDB, then write Mongo Aggregation query in Master table query field. Please refer the below given demonstration to configure the Enrichment component for MongoDB driver.

Configuring Enrichment Component for MongoDB driver

Configuring Enrichment Component for Redshift driver

Sample Query for MongoDB Driver

  {
    $match: {
      state: "Uttar-Pradesh"
    }
  },
  {
    $group: {
      _id: "$District",
      totalPopulation: { $sum: "$Population" },
      totalArea: { $sum: "$Area_(km2)" },
      averageDensity: { $avg: "$Pop._Density" }
    }
  },
  {
    $sort: {
      totalPopulation: -1
    }
  }

Steps to configure the Enrichment component

• Drag and drop the Enrichment Component to the Workflow Editor.

• Create two Events and drag them to the Workspace.

• Connect the input event and the output event (The data in the input event can come from any Ingestion, Reader, or shared events).

• Click the Enrichment Component to get the component properties tabs.

Using the Enrichment component as a part of Pipeline Workflow

Basic Information

The Basic Information tab opens by default.

• Select an Invocation type from the drop-down menu to confirm the running mode of the reader component. Select either Real-Time or Batch option from the drop-down menu.

• Deployment Type: It displays the deployment type for the component. This field comes pre-selected.

• Container Image Version: It displays the image version for the docker container. This field comes pre-selected.

• Failover Event: Select a failover Event from the drop-down menu.

• Batch Size: Provide the maximum number of records that you want to be processed in one execution cycle.

Meta Information

Open the Meta Information tab and fill in all the connection-specific details for the Enrichment Component.

• Driver (*): Select Database type (MYSQL, MSSQL, Oracle, Postgres, MongoDB, ClickHouse, Snowflake)

• Port (*): Host server port number

• Host IP Address (*): IP Address

• Username (*): Username for Authentication.

• Password (*): Password for Authentication.

• Database Name (*): Provide the Database name.

• Enable SSL: Check this box to enable SSL for this components. Enable SSL feature in DB reader component will appear only for three(3) drivers: Mongodb, PostgreSQL and ClickHouse.

• Certificate Folder: This option will appear when the Enable SSL field is checked-in. The user has to select the certificate folder from drop down which contains the files which has been uploaded to the admin settings. Please refer the below given images for the reference.

• Table Name: Provide the table name to read the data.

• Refresh rate in Secs: The value of this field has to be provided in seconds. It refreshes the master table and fetches the changes in every cycle of the given value. For example, if Refresh rate value is given as 3600 seconds, it will refresh the master table in every 3600 seconds and fetch the changes (Default value for this field is 3600 seconds).

Please Note: The Refresh rate value can be changed according to your use-case.

• Connection Type: This option will show if the user selects MongoDB from the Driver field. A User can configure the MongoDB driver via two connection types (Standard or SRV) that are explained below:

  • Standard - Port field does appear with the Standard connection type option.

  • SRV - Port field does not appear with the SRV connection type option.

• Conditions: Select conditions type (Remove or Blank option).

• Master table query: Write a Spark SQL query to get the data from the table which has been mentioned in the Table name field.

• Query: Enter a Spark SQL query to join the master table and the data coming from the previous event. Take inputDf as the table name for the previous event data.

Meta information of Enrichment Component

Selected Columns

The users can select some specific columns from the table to read data instead of selecting a complete table; this can be achieved via the Selected Columns section. Select the columns which you want to read and if you want to change the name of the column, then put that name in the alias name section otherwise keep the alias name the same as of column name and then select a Column Type from the drop-down menu.

or

Use the Download Data and Upload File options to select the desired columns.

• Upload File: The user can upload the existing system files (CSV, JSON) using the Upload File icon (file size must be less than 2 MB).

• Download Data (Schema): Users can download the schema structure in JSON format by using the Download Data icon.

Saving the Component Configuration

• After doing all the configurations click the Save Component in Storage icon provided in the reader configuration panel to save the component.

• A notification message appears to inform about the component configuration success.

Query Example

select * from inputDf left join departments on inputDf.department_id = departments.department_id;

select * from inputDf left join departments on inputDf.department_id = departments.department_id;

Please Note: The data of previous event is taken as inputDf as the table name in the query field of the Enrichment component as shown in the above query example.