Memory and CPU allocations

configuration Tab.

For each component that gets deployed, we have an option to configure the resources i.e., Memory and CPU.

We have two deployment types:

  • Docker

  • Spark

Docker

Go through the given illustration to understand how to configure a component using the Docker deployment type.

Docker Components Conjuration steps

After we save the component and pipeline, the component gets saved with the default configuration of the pipeline i.e., Low, Medium, and High. After we save the pipeline, we can see the configuration tab in the component. There are multiple things.

  • For the Docker components, we have the Request and Limit configurations.

  • We can see the CPU and Memory options to be configured.

CPU: This is the CPU configuration where we can specify the number of cores that we need to assign to the component.

Memory: This option is to specify how much memory you want to dedicate to that specific component.

Instances: The number of instances is used for parallel processing. If we give N no. of instances those many pods will get deployed.

Spark

Go through the below given walk-through to understand the steps to configure a component with Spark configuration type.

Spark Component Configuration Steps

The Spark Components configuration is slightly different from the Docker components. When the spark components are deployed, there are two pods that come up:

  • Driver

  • Executor

Provide the Driver and executor configurations separately.

Instances: The number of instances is used for parallel processing. If we give N no. of instances in executors configuration those many executors pods will get deployed.

Last updated

Was this helpful?