Kafka Consumer

The Kafka Consumer component consumes the data from the given Kafka topic. It can consume the data from the same environment and external environment with CSV, JSON, XML, and Avro formats. This comes under the Consumer component group.

All component configurations are classified broadly into the following sections:

• ​​Basic Information​​

• Meta Information

• ​Resource Configuration​​

Check out the steps provided in the demonstration to configure the Kafka Consumer component.

Configuring the Kafka Consumer component

Please Note: It currently supports SSL and Plaintext as Security types.

Component being used to read data from the external broker.

This Component can read the data from external Brokers as well with SSL as the security type and host Aliases:

Kafka consumer Meta Information tab

Steps to Configure

• Drag and drop the Kafka Consumer Component to the Workflow Editor.
• Click on the dragged Kafka Consumer component to get the component properties tabs.

• Configure the Basic Information tab.

• Select an Invocation type from the drop-down menu to confirm the running mode of the component. Select the Real-Time option from the drop-down menu.

• Deployment Type: It displays the deployment type for the component. This field comes pre-selected.

• Container Image Version: It displays the image version for the docker container. This field comes pre-selected.

• Failover Event: Select a failover Event from the drop-down menu.

• Batch Size (min 10): Provide the maximum number of records to be processed in one execution cycle (Min limit for this field is 10

• Enable Auto-Scaling: Component pod scale up automatically based on a given max instance, if component lag is more than 60%.

• Click on the Meta Information tab to open the properties fields and configure the Meta Information tab by providing the required fields.

• Topic Name: Specify the topic name that the user wants to consume data from Kafka.

• Start From: It contains the following start from:

• Processed: Using this option consumes live processed data and already processed data that has never been consumed by the component and

• Beginning: Using this option consumes live processed data and already processed data from the beginning

• Latest: Using this option consumes the latest processed data.

• Timestamp: Using this option consumes data between given interval times.

• Is External: The user can consume external topic data from the external bootstrap server by enabling the Is External option. The Bootstrap Server and Config fields will display after enabling the Is External option.

• Bootstrap Server: Enter external bootstrap details.

• Config: Enter configuration details of external details.

• Input Record Type: It contains the following input record types:

• CSV: The user can consume CSV data using this option. The Headers and Separator fields will display if the user selects choose CSV input record type.

  • Header: In this field, the user can enter column names of CSV data that consume from the Kafka topic.

  • Separator: In this field, the user can enter separators like comma (,) that used CSV data.

• JSON: The user can consume JSON data using this option.

• XML: The user can consume parquet data using this option.

• AVRO: The user can consume Avro data using this option.

• Security Type: It contains the following security types:

  • Plain Text: Choose the Plain Text option if there environment without SSL.

  • Host Aliases: This option contains the following fields:

  • IP: Provide the IP address.

  • Host Names: Provide the Host Names.

• SSL: Choose the SSL option if there environment with SSL. It will display the following fields:

  • Trust Store Location: Provide the trust store path.

  • Trust Store Password: Provide the trust store password.

  • Key Store Location: Provide the key store path.

  • Key Store Password: Provide the key store password.

  • SSL Key Password: Provide the SSL key password.

  • Host Aliases: This option contains the following fields:

  • IP: Provide the IP.

  • Host Names: Provide the host names.

Please Note: The Host Aliases can be used with SSL and Plain text Security types.

• After doing all the configurations click the Save Component in Storage icon provided in the configuration panel to save the component.

s

• A notification message appears to inform about the component configuration saved.

Please Note: The user should know the Kafka details and topic name.