Intelligent Scaling

A way to scale up the processing speed of components.

A feature to scale your component to the max number of instances to reduce the data processing lag automatically. This feature detects the need to scale up the components in case of higher data traffic.

Please Note: This feature is available both in Spark & Docker components. This feature will only work with the real-time as invocation type.

Intelligent Scaling as a part of the ClickHouse Writer

All components have option of Intelligent scaling which is ability of the system to dynamically adjust the scale or capacity of the reader component based on the current demand and available resources. It involves automatically optimizing the resources allocated to the component to ensure efficient and effective processing of tasks.

Please Note:

• If you have selected intelligent scaling option make sure to give enough resources to component so that it can auto-scale based on the load.

• Components will scale up if there is a lag of more than 60% and if the lag goes less than 10% component pods will automatically scale down. This lag percentage is configurable.