Executing External Scripts Using the Script Executor Job

This workflow demonstrates how to configure and run the Script Executor job in the BDB Platform to execute scripts written in multiple programming languages — including Go, Julia, and Python. The Script Executor seamlessly integrates with Git repositories (GitHub or GitLab) to fetch, execute, and monitor scripts within the platform, providing flexibility for cross-language automation and reproducible workflows.

Overview

The Script Executor job enables users to:

• Fetch and execute scripts directly from remote Git repositories.

• Support multiple programming languages such as Go, Julia, and Python.

• Automate execution across different environments without manual intervention.

• Maintain consistent and version-controlled code execution within BDB’s data orchestration layer.

This workflow is ideal for data engineers, automation developers, and ML practitioners who require flexibility in executing diverse scripts across a unified platform.

Prerequisites:

Before you begin, ensure that:

• You have access to the Data Pipeline module.

• A Git repository (GitHub or GitLab) is available and accessible with valid credentials.

• Your personal access token (PAT) or API token is generated and authorized for repository access.

• The desired script file is available in your Git repository, or you have permission to upload it.

Step 1: Create the Script Executor Job

Procedure

1. Navigate to the Data Pipeline module from the Apps menu.

2. Select the Jobs tab.

3. Click the Create icon to initiate job creation.

4. In the job setup window, provide the following details:

   • Job Name: Workflow6

   • Description: Script Executor Job

5. From the Job Type dropdown, select Script Executor.

6. Click Save.

• A new Script Executor job is created, and you are redirected to the Job Canvas for configuration.

Step 2: Configure the Script Executor Component

Procedure

1. Click on the newly created Script Executor job component on the canvas.

2. In the Meta Information tab, configure the Git connection:

   

Field	Description
Git Configuration	Select Personal
Git URL	https://github.com or https://gitlab.com
Username	Your Git username
Access Token	Personal Access Token (PAT) or API Token
Branch	Name of the branch containing the script
Script Type	Choose one: Go, Julia, or Python
Start Script	Filename (e.g., main.py)
Start Function	Function to execute (e.g., execute_script)
Repository Name	Name of the Git repository

1. If your script is not yet uploaded to Git:

   • Go to your GitHub or GitLab repository.

   • Click Upload File and select your local script.

   • Commit the file to your branch.

     
2. Return to the BDB Platform and ensure all required fields are filled correctly.

• The Script Executor component is now linked to your Git repository and configured for execution.

Step 3: Activate and Execute the Script Executor Job

Procedure

1. Click the Update icon to save configurations.

2. Click the Activate icon (green button at the top right).

3. You will be redirected to the Advanced Job Status section, where you can:

   • Monitor job status and execution progress.

   • Verify that all pods are running properly.

   • Click Logs to view detailed execution logs.

     

• Once the script executes successfully, a confirmation message appears.

Verification

• Open the Logs tab to verify that the script was fetched and executed correctly from the Git repository.

• Ensure that console output or print statements are logged as expected.

Step 4: Sample Python Script for Script Executor

Below is a sample Python script you can use for testing your Script Executor setup.

# Example Python script for Script Executor component

def execute_script():
    # Sample Logic: Adding two numbers
    num1 = 10
    num2 = 20
    result = num1 + num2
    print(f"The sum of {num1} and {num2} is {result}")

# Ensure that the script runs when the file is executed
if __name__ == "__main__":
    execute_script()

Notes:

• Save the file as script.py or any name consistent with the Start Script value configured in the platform.

• Ensure that the function name (execute_script) matches the Start Function field in the configuration.

• The Script Executor job fetches the file from your Git repository, runs it, and logs the output (e.g., “The sum of 10 and 20 is 30”).

Results

After successful completion:

• The BDB Platform fetches the script from Git, executes it, and displays logs in real time.

• Users can monitor the job lifecycle from the Advanced Job Status section.

• All results and execution details are captured within the centralized log view.

Notes and Recommendations:

• Ensure that the Git access token has appropriate read permissions for the target repository.

• Use branch-specific configuration to control which version of a script runs in production.

• For sensitive scripts, use secret management to securely store credentials.

• Validate that the script’s entry point (main function) matches the Start Function specified during setup.

Best Situation to Use

Use the Script Executor Job when:

• You need to run external scripts across multiple languages (Go, Julia, Python) directly from Git.

• Your development team wants to centralize code execution within the BDB ecosystem without manual deployment.

• You require version-controlled, automated, and environment-agnostic execution of scripts.

• Integrating third-party logic or analytical models hosted outside the platform is necessary.

Conclusion

You’ve completed the Script Executor Job on the BDB Platform.

You have learned how to:

• Connect to a Git repository.

• Execute Go, Julia, or Python scripts directly inside the BDB environment.

• Monitor and verify job execution using centralized logs.

This workflow demonstrates the flexibility, scalability, and reusability of the BDB Platform, empowering data teams to integrate and automate diverse programming workflows seamlessly.