Scheduler

The Scheduler Component allows users to schedule tasks or pipeline executions automatically at predefined times. It uses cron expressions to define execution intervals ranging from minutes to years.

This component is ideal for automating recurring workflows, batch jobs, or timed data ingestion processes.

Key Capabilities

  • Schedule workflows to run at fixed times or recurring intervals.

  • Define scheduling using cron expressions with minute-to-year granularity.

  • Support for time zones to ensure region-specific scheduling.

  • Monitor scheduled pipelines through the Scheduler List Page.

Configuration Overview

All Scheduler configurations are grouped into:

  • Basic Information

  • Meta Information

  • Resource Configuration

Steps to Configure the Scheduler Component

  1. Add Component

    • Drag and drop the Scheduler Component into the Workflow Editor.

  2. Connect Events

    • Connect the Scheduler to a Reader or Data Loading Component (input event).

  3. Basic Information Tab

    • Invocation Type – Only Real-Time is supported.

    • Deployment Type – Pre-selected.

    • Container Image Version – Pre-selected.

    • Failover Event – Select a failover event if scheduling fails.

    • Batch Size – Minimum of 10; defines maximum number of records processed per execution cycle.

  4. Meta Information Tab

    • Scheduler Name – Provide a descriptive name.

    • Payload – Enter the payload in the required format.

    • Timezone – Select from drop-down (e.g., UTC, Asia/Kolkata).

    • Scheduler Time – Time derived from the Cron Generator.

    • Cron Generator – Create cron expressions to define when the pipeline should run.

Cron Generator

The Scheduler supports various time-based patterns.

Supported Units of Time

  • Minutes – Execute at fixed intervals of minutes/seconds.

  • Hourly – Execute at fixed intervals of hours, minutes, and seconds.

  • Daily – Execute once daily at a specified time.

    • Option: Run every working day at the specified time.

  • Weekly – Execute on selected days of the week at a specified time.

  • Monthly – Execute on a given date of the month at a specified time.

    • Option: Run on the nearest weekday if the scheduled date is a weekend.

    • Option: Schedule by first/last occurrence of a weekday (e.g., Second Monday of each month).

  • Yearly – Execute annually on a specified date.

    • Option: Run on the nearest weekday.

    • Option: Schedule by first/last occurrence of a weekday in a given month.

Value Ranges

  • Hours: 0–23

  • Minutes: 0–59

  • Seconds: 0–59

  • Months: 1–12 or names (January–December)

  • Weekday: Monday–Sunday

  • Day of Month: 1–31 (February supports up to 29)

Example:

  • 0 0 9 * * MON → Run every Monday at 09:00.

  • 0 30 10 1 * * → Run on the 1st of every month at 10:30.

Scheduler List Page

The Scheduler List Page provides visibility into all scheduled workflows.

  • Displays meta-information of each scheduler.

  • Shows:

    • Number of times triggered

    • Next scheduled run

  • Accessed via the Scheduler icon on the left-side panel of the Pipeline landing page.

  • Click on a pipeline name to:

    • View trigger history.

    • Open the pipeline in the editor.

Example Use Cases

  • Run daily sales ingestion pipelines at midnight.

  • Schedule data validation jobs every Monday at 08:00.

  • Trigger monthly reports on the last working day of each month.

  • Automate hourly log processing jobs.

PreviousPySpark ScriptNextAlerts