Register as Job
The Register Feature enables users to directly create jobs from DsLab notebook. Using this feature, users can create the following types of jobs:
Creating a Python Job or Python Job (On demand) using register feature.
Follow the below-given steps to create a Python Job using the register feature:
Create a project in the DSLab module under the Python environment. Refer to the below given image for guidance.
Once the project is created, create a Notebook and write the scripts. It supports scripts running through multiple code cells.
After writing the script, select the Register option from the Notebook options. Refer to the following image for reference.
The Register as Job panel will open on the right side of the window with two options:
Register as New: This will create a new Job.
Re-Register: If the Job is already created and the user wants to make changes to the scripts, it will update the existing Job with the latest changes.
Register as New:
Please go through the walkthrough below to register as a new Job.
After clicking on the Register option and selecting the Register as New option, the user needs to choose the desired notebook cell that should be executed with the Job. The user can select multiple cells from the notebook according to requirement and then click on the Next option.
Now, the user needs to validate the script of the selected cell by clicking on the Validate option. Additionally, the user can select the External Libraries used in the project by clicking on the External Libraries option next to the Validate option. The Next option won't be enabled until the user validates the script. After validating the script, click on the Next option.
Enter scheduler name: Name of the job. (It will create a scheduled job if the user is creating a Python Job).
Scheduler description: Description of the job.
Start function: Select the start function from the validated script to start the job from the given start function.
Job baseinfo: Python (This field will be pre-selected if the DSLab project is created under the Python environment.)
Docker config: Select the desired configuration for the job and provide the resources (CPU & Memory) for executing the job.
Provide the resources required to run the Python Job in the Limit and Request section.
Limit: Enter the max CPU and Memory required for the Python Job.
Request: Enter the CPU and Memory required for the job at the start.
Instances: Enter the number of instances for the Python Job.
On demand: Check-in this option if Python Job (On demand) has to be created. In this scenario, the Job will not be scheduled.
Payload: This option will appear if the On demand option is checked in. Enter the payload in the form of a list of dictionaries. For more details about the Python Job (On demand), refer to this link: Python Job(On demand)
Concurrency Policy: Select the desired concurrency policy. For more details about the Concurrency Policy, check this link: Concurrency Policy
Alert: This feature in the Job allows the users to send an alert message to the specified channel (Teams or Slack) in the event of either the success or failure of the configured Job. Users can also choose both success and failure options to send an alert for the configured Job. Check the following link to configure the Alert: Job Alerts
Click on the Save option to create a job.
Re-Register:
This feature allows the user to update an existing job with the latest changes. Select the Re-register option if the Job is already created and the user wants to make changes to the scripts.
After selecting the Re-Register option, the system will display all previous versions of registered Jobs from the chosen notebook.
The user must choose the Job to be Re-Registered with the latest changes and proceed by clicking Next.
Validate the script of the selected cell by clicking on the Validate option. Additionally, the user can choose External Libraries by clicking on the option next to Validate. The Next option remains disabled until the script is validated.
After script validation, proceed by clicking Next.
Now, the user needs to provide all the required information for Re-registering the job, following the same steps outlined in the Register as New feature.
Finally, click on the Save option to complete the Re-Registration of the job.
Please Note: The user can Register as New or Re-Register a PySpark Job in the same way as Python Job as mentioned above. The only difference is that the user needs to create the project under PySpark environment in the DSLab module.
Last updated
