Component Base Configuration

This is components base configurations that has to be configured for all the components

Invocation Type

The Invocation-Type config decides the type of deployment of the component. There are following two types of invocations:

1. Real-Time

2. Batch

Real-Time:

Real-Time Configuration

When the Component has the real-time invocation the component never goes down when the pipeline is active. This is for situations where you want to keep the component ready all the time to consume data.

Note: The First Component of the pipeline must be in real-time invocation.

Batch:

Batch Configuration

When the component has the batch invocation type then the component needs a trigger to initiate the process from the previous event. Once the process of the component is finished and there are no new events to process the component goes down.

These are really helpful in batch or scheduled operations where the data is not streaming or real-time.

When we select batch invocation type we get an additional option of the grace period. This grace period is the time the component will take to go down gracefully.

The default is 60 seconds and can be configured.

Batch Size

The pipeline components process the data in micro-batches. This batch size is given to define the maximum number of records that you want to process in a single cycle of operation; This is really helpful if you want to control the number of records being processed by the component if the unit record size is huge. You can configure it in the base config of the components.

Failover Events

We can create a failover event and map it in the component base config so that if the component fails it audits all the failure messages with the data(if available) and timestamp of error.

configuring a failover event