Python Job
A Python Job in the Data Pipeline module allows you to write, execute, and manage Python scripts as part of your data workflows. Python Jobs are created using scripts developed in the Data Science Lab under the Python Environment. These scripts can be exported and executed seamlessly in the Jobs section of the Data Pipeline module.
Prerequisites
Before creating a Python Job:
Create a project in the Data Science Lab under Python Environment.
Activate the project and create a Notebook.
Write your Python script in the Notebook, then save and export it to make it available in the Data Pipeline module.
📎 Refer to the Data Science Lab Projects guide for detailed steps on creating and exporting Python scripts.
Create a Python Job
Navigation path: Data Pipeline > Jobs > Create Job
From the Data Pipeline homepage, click Create.
In the right-hand panel, select Job → Create.
Fill in the following fields:
Name: Enter a name for the job.
Description (Optional): Provide details about the job.
Job Base Info: Select Python Job from the drop-down menu.
Trigger By: Configure how the job should be triggered:
On Success: The job will start when the selected job completes successfully.
On Failure: The job will start if the selected job fails.
Is Scheduled?
Schedule the job for a specific UTC timestamp. The job runs automatically at the defined schedule.
Or select On Demand to run the job manually. Learn more about On-Demand Jobs.
Docker Configuration: Select resource allocation: Low, Medium, or High.
Resources: Define CPU and memory allocation.
Limit: Maximum CPU and memory allowed for the job.
Request: Minimum CPU and memory required at job start.
Instances: Number of instances to run.
Alerts: Configure Job Alerts to monitor execution and failures.
Click Save to create the Python Job.
Once saved, you are redirected to the Job Editor workspace.
Configure a Python Job
Navigation path: Data Pipeline > Jobs > Job Editor > Meta Information tab
After creating a Python Job, configure its metadata:
Project Name: Select the project where the Notebook was created.
Script Name: Choose the exported Notebook (appears in the drop-down list).
External Library: List any additional libraries required (comma-separated).
Example:
pandas,numpy,scikit-learn
Start Function: Select the entry function to execute from the script.
Script: The exported script appears in this section.
Input Data: Provide parameters if the start function requires inputs.
Key = parameter name
Value = parameter value