Job Editor Page

The Job Editor Page provides the user with all the necessary options and components to add a task and eventually create a Job workflow.

Adding Tasks to a Job Workflow

Once a Spark Job is saved, users can design a workflow by adding and configuring Tasks. Tasks allow you to define how data is read, transformed, and written within the job execution pipeline.

Please note: Task components are available only for Spark Jobs.

Step-by-step process for adding tasks to a Job Workflow

1

Accessing the Job Editor

• Navigate to the Jobs List page.

• Select a job from the list. The job type is displayed in the Type column.

• Click the View icon for the selected job.

  • If continuing immediately after job creation, the Job Editor opens automatically.

  • If returning later, use the above steps to re-open the Job Editor.

Please note: The Job Editor displays the Task Panel and the workflow canvas.

2

Task Panel Overview

• The Task Panel is accessible by clicking the Add Task icon in the Job Editor.

• Located on the right-hand side, the Task Panel includes:

  • Search Bar – Quickly search for task components.

  • Task Categories:

    • Readers: Input sources (databases, files, or SaaS applications).

    • Writers: Destinations (databases or data warehouses where processed data is written).

    • Transforms: Processing steps to cleanse, enrich, or prepare data for analytics.

For more information, refer to the Task Components page.

3

Searching for a Task

• In the Search Task field, type the name of the required task.

• Matching task components appear as suggestions.

• Select a suggestion to filter the Task Panel and display only relevant tasks.

4

Creating a Job Workflow

• From the Task Panel, drag the required task onto the canvas.

  • Example: Drag a DB Reader to the workspace.

• Click the dragged task icon to open its configuration dialog.

• Configure the Meta Information for the task.

• Click the Validate Connection icon to test connectivity.

• Click Save Task in Storage.

  • A notification confirms the task has been saved.

5

Activating the Job

• Once all tasks are configured, click the Update Job icon.

• Click the Activate Job icon.

• In the confirmation dialog, click Yes.

• A success message confirms activation.

Please note: Once activated, the View option replaces the Edit option for a job.

6

Development Mode

Jobs can also be run in Development Mode:

• Preview up to 10 records in the Preview Data tab for each task.

• Writer tasks will insert only 10 records into the target database.

• If not in Development Mode, no preview data is displayed.

• The job status updates on the Jobs List page to reflect whether it is running in Development Mode or Activated.

7

Job Task Preview

• Navigate to the Job Editor page.

• Select a task and open the Preview Data tab.

• Available actions:

  • Preview up to 10 records.

  • Download data in CSV, JSON, or Excel format.

  • Copy records as a list of dictionaries.

  • Adjust Column Widths by dragging column separators.

  • Filter Data by:

    • Latest records

    • Beginning records

    • Timestamp (choose from time ranges or define start and end dates).

8

Toggle Log Panel

The Toggle Log Panel provides detailed job logs.

1. In the Job Editor, click the Toggle Log Panel icon from the header.

2. The Log Panel opens with:

   • Logs Tab: Displays collective logs of the job.

   • Advanced Logs Tab: Provides detailed execution logs.

   • Job Status Tab: Displays the pod status of the job.

Important: Orphan tasks (unlinked tasks in the workflow) cause job failure. For example, a DB Writer without an input source is considered an orphan task. Always ensure tasks are connected properly.

9

Version Update

If a newer version of the job is available:

• The Update Job Version icon displays a red dot.

• Click the icon.

• A confirmation dialog appears to update the job version; click Yes.

• A notification confirms the version upgrade.

• The icon will be disabled once the job is up to date.

By following these steps, users can design Spark Job workflows by adding, configuring, and managing tasks, ensuring their jobs are version-controlled, tested in development, and activated for production use.