Component Base Configuration
This page pays attention to describe the Basic Info tab provided for the pipeline components. This tab has to be configured for all the components.
This page pays attention to describe the Basic Info tab provided for the pipeline components. This tab has to be configured for all the components.
The Invocation-Type config decides the type of deployment of the component. There are following two types of invocations:
Real-Time
Batch
When the Component has the real-time invocation, the component never goes down when the pipeline is active. This is for situations where you want to keep the component ready all the time to consume data.
When "Realtime" is selected as the invocation type, we have an additional option to scale up the component called "Intelligent Scaling."
Please refer to the following page to learn more about Intelligent Scaling.
Please Note: The First Component of the pipeline must be in real-time invocation.
When the component has the batch invocation type then the component needs a trigger to initiate the process from the previous event. Once the process of the component is finished and there are no new events to process the component goes down.
These are really helpful in Batch or scheduled operations where the data is not streaming or real-time.
Please Note: When the users select the Batch invocation type, they get an additional option of the Grace Period (in sec). This grace period is the time that the component will take to go down gracefully. The default value for Grace Period is 60 seconds and it can be configured by the user.
The pipeline components process the data in micro-batches. This batch size is given to define the maximum number of records that you want to process in a single cycle of operation; This is really helpful if you want to control the number of records being processed by the component if the unit record size is huge. You can configure it in the base config of the components.
The below given illustration displays how to update the Batch Size configuration.
We can create a failover event and map it in the component base configuration, so that if the component fails it audits all the failure messages with the data (if available) and timestamp of the error.
Go through the illustration given below to understand the Failover Events.
