Data Pipeline
  • Data Pipeline
    • About Data Pipeline
    • Design Philosophy
    • Low Code Visual Authoring
    • Real-time and Batch Orchestration
    • Event based Process Orchestration
    • ML and Data Ops
    • Distributed Compute
    • Fault Tolerant and Auto-recovery
    • Extensibility via Custom Scripting
  • Getting Started
    • Homepage
      • List Pipelines
      • Create
        • Creating a New Pipeline
          • Adding Components to Canvas
          • Connecting Components
            • Events [Kafka and Data Sync]
          • Memory and CPU Allocations
        • Creating a New Job
          • Job Editor Page
          • Task Components
            • Readers
              • HDFS Reader
              • MongoDB Reader
              • DB Reader
              • S3 Reader
              • Azure Blob Reader
              • ES Reader
              • Sandbox Reader
            • Writers
              • HDFS Writer
              • Azure Writer
              • DB Writer
              • ES Writer
              • S3 Writer
              • Sandbox Writer
              • Mongodb Writer
              • Kafka Producer
            • Transformations
          • PySpark Job
          • Python Job
      • List Jobs
      • List Components
      • Delete Orphan Pods
      • Scheduler
      • Data Channel
      • Cluster Event
      • Trash
      • Settings
    • Pipeline Workflow Editor
      • Pipeline Toolbar
        • Pipeline Overview
        • Pipeline Testing
        • Search Component in Pipelines
        • Push Pipeline (to VCS/GIT)
        • Pull Pipeline
        • Full Screen
        • Log Panel
        • Event Panel
        • Activate/Deactivate Pipeline
        • Update Pipeline
        • Failure Analysis
        • Pipeline Monitoring
        • Delete Pipeline
        • Pipeline Component Configuration
        • Pipeline Failure Alert History
      • Component Panel
      • Right-side Panel
    • Testing Suite
    • Activating Pipeline
    • Monitoring Pipeline
    • Job Monitoring
  • Components
    • Adding Components to Workflow
    • Component Architecture
    • Component Base Configuration
    • Resource Configuration
    • Intelligent Scaling
    • Connection Validation
    • Readers
      • S3 Reader
      • HDFS Reader
      • DB Reader
      • ES Reader
      • SFTP Stream Reader
      • SFTP Reader
      • Mongo DB Reader
        • MongoDB Reader Lite (PyMongo Reader)
        • MongoDB Reader
      • Azure Blob Reader
      • Azure Metadata Reader
      • ClickHouse Reader (Docker)
      • Sandbox Reader
      • Azure Blob Reader
    • Writers
      • S3 Writer
      • DB Writer
      • HDFS Writer
      • ES Writer
      • Video Writer
      • Azure Writer
      • ClickHouse Writer (Docker)
      • Sandbox Writer
      • MongoDB Writers
        • MongoDB Writer
        • MongoDB Writer Lite (PyMongo Writer)
    • Machine Learning
      • DSLab Runner
      • AutoML Runner
    • Consumers
      • SFTP Monitor
      • MQTT Consumer
      • Video Stream Consumer
      • Eventhub Subscriber
      • Twitter Scrapper
      • Mongo ChangeStream
      • Rabbit MQ Consumer
      • AWS SNS Monitor
      • Kafka Consumer
      • API Ingestion and Webhook Listener
    • Producers
      • WebSocket Producer
      • Eventhub Publisher
      • EventGrid Producer
      • RabbitMQ Producer
      • Kafka Producer
      • Synthetic Data Generator
    • Transformations
      • SQL Component
      • Dateprep Script Runner
      • File Splitter
      • Rule Splitter
      • Stored Producer Runner
      • Flatten JSON
      • Email Component
      • Pandas Query Component
      • Enrichment Component
      • Mongo Aggregation
      • Data Loss Protection
      • Data Preparation (Docker)
      • Rest Api Component
      • Schema Validator
    • Scripting
      • Script Runner
      • Python Script
        • Keeping Different Versions of the Python Script in VCS
    • Scheduler
  • Custom Components
  • Advance Configuration & Monitoring
    • Configuration
      • Default Component Configuration
      • Logger
    • Data Channel
    • Cluster Events
    • System Component Status
  • Version Control
  • Use Cases
Powered by GitBook
On this page
  • Configuring the Script Runner Component
  • Accessing the Script Runner Component
  • Basic Information Tab
  • Meta Information Tab
  • Input Arguments
  • Saving the Component Configuration

Was this helpful?

  1. Components
  2. Scripting

Script Runner

PreviousScriptingNextPython Script

Was this helpful?

This component can be used for connecting it to a remote server/machine and running script files present there based on some events.

All component configurations are classified broadly into the following sections:

  • ​​

  • Meta Information

  • ​​

Please follow the given steps in the demonstration to use the Script Runner component in a pipeline workflow.

Configuring the Script Runner Component

Accessing the Script Runner Component

The Script Runner component is provided under the Scripting section of the Component pallet.​

  • Drag and drop Script Runner Component to the Workflow Editor.

  • Open the dragged Script Runner component to open the component configuration tabs.

Basic Information Tab

The Basic Information tab opens by default.

  • Invocation Type: Select an Invocation type from the drop-down menu to confirm the running mode of the script runner component. The supported invocation types are Real-Time and Batch.

  • Deployment Type: It displays the deployment type for the component. This field comes pre-selected.

  • Container Image Version: It displays the image version for the docker container. This field comes pre-selected.

  • Failover Event: Select a failover event from the drop-down menu.

  • Batch Size (min 10): Provide the maximum number of records to be processed in one execution cycle (Min limit for this field is 10).

Meta Information Tab

Open the Meta Information tab and configure the required information.

  • Host: Host IP of the remote server/machine

  • Username: Username of the remote server/machine.

  • Port: Provide machine Port number.

  • Authentication: Select an authentication option from the drop-down menu.

    • Password: By selecting this option the user needs to pass the password.

    • PEM/PPK File: By selecting this option the user needs to pass the authentication file to connect to the server.

  • Script type: Choose the type of script file that You want to run out of SSH/ PERL/command options.

  • File path: Path of the file that is stored at the remote server.

  • File Name: The script file that you want to execute.

  • Event File Location: this is the location of the file sent through the file monitor (Non-mandatory).

Please Note: The displayed fields may vary based on the selected Authentication option.

Component Properties when the Authentication option is Password.

Component Properties when the Authentication option is PEM/PPK File.

Input Arguments

  • Manual Arguments (Optional): These are the arguments to the parameter of the script that the user can provide manually.

  • Event Arguments (Optional): These are the arguments to the parameter coming from the previous event/Kafka topic.

Saving the Component Configuration

  • Click the Save Component in Storage icon (A notification message appears to confirm the action completion).

  • The Script Runner component gets configured, and the notification message appears to inform the same.

Please Note: The component can connect to the remote machine using the details provided. It will pick the file from the location in that machine using the file name and file path respectively and finally execute the script after passing arguments (if any).

Limitations

a. It accepts only lists as input i.e. the in-event data should be a list.

b. It sends data on the out-event only when there is a print statement as output in the script if not there will be no data on the out-event.

c. The data produced from the script is of a list type.

​

Script Runner Meta Information
​Basic Information​
Resource Configuration​
Configuring the Script Runner component in a Pipeline Workflow
Script Runner Basic Information