Workflow 5

Automating Python (On Demand) Job Execution Using Job Trigger

This workflow focuses on automating the initiation of Python (On Demand) jobs using the Job Trigger component. The Job Trigger uses in-event data as a payload to automatically trigger Python jobs, reducing manual work and ensuring efficient, real-time execution.

This integration streamlines data processing and helps maintain consistency, speed, and control across your pipeline processes.

Create Data Pipeline

Navigate to the Data Pipeline module using app menu from the BDB homepage.

Select data pipeline, Click the “Create” icon. Enter the pipeline name, description, and resource allocation (High, Medium, Low). Click Save to create the pipeline.

Add Python Component

From the Components section, search for Python Script. Drag and drop the component into the workspace.

Set the invocation type to Real-Time. Go to the Meta Information tab. Provide a name like Payload. Select the data frame from in-event data. Choose Custom Script as the execution type. Write a simple function in the script editor:

def display():

return [{"id":101,"name":"jashika","age":20},{"id":102,"name":"siya","age":40}]

Choose display as the start function. Validate and Save the component.

Add and Configure Event

Click the Event Panel icon in the toolbar. Click the "+" icon to add a new event. Rename the event if needed. Drag and drop it onto the canvas and connect it to the Python component.

Add and Configure Job Trigger Component

Search for Job Trigger from the Components library. Drag and drop it onto the canvas.Set the invocation type to Real-Time. In Meta Information, select Workflow 4 from the job dropdown (only Python On Demand jobs will be listed). Click Save.

Repeat the process to add another event and connect it to the Job Trigger component.

Update the Pipeline

Click the Update Pipeline icon in the toolbar.

Configure Python Component in Workflow 4

Click the component to configure the recently exported custom script. Choose the Project Name and Script Name from the dropdowns. Set the Start Function. Enter the necessary input arguments including DB credentials. Save the configuration.

Now activate the pipeline. once the pipeline Execution is completed successfully, it will automatically trigger the job workflow 4.

click on the activated job from the job list. In the Advanced Job Status section, Ensure all pods are operational.

Click Logs to track progress and validate execution. Once the job completes, a success message will appear.

Conclusion

Congratulations! You’ve successfully completed Workflow 5 by automating a Python (On Demand) job using the Job Trigger component. This streamlined workflow showcases the power of dynamic job chaining, efficient orchestration, and real-time processing on the BDB platform.

Once done, don’t forget to deactivate the pipeline to release resources.