Data Pipeline
  • Data Pipeline
    • About Data Pipeline
    • Design Philosophy
    • Low Code Visual Authoring
    • Real-time and Batch Orchestration
    • Event based Process Orchestration
    • ML and Data Ops
    • Distributed Compute
    • Fault Tolerant and Auto-recovery
    • Extensibility via Custom Scripting
  • Getting Started
    • Homepage
      • List Pipelines
      • Creating a New Pipeline
        • Adding Components to Canvas
        • Connecting Components
          • Events [Kafka and Data Sync]
        • Memory and CPU Allocations
      • List Jobs
      • Create Job
        • Job Editor Page
        • Task Components
          • Readers
            • HDFS Reader
            • MongoDB Reader
            • DB Reader
            • S3 Reader
            • Azure Blob Reader
            • ES Reader
            • Sandbox Reader
          • Writers
            • HDFS Writer
            • Azure Writer
            • DB Writer
            • ES Writer
            • S3 Writer
            • Sandbox Writer
            • Mongodb Writer
            • Kafka Producer
          • Transformations
        • PySpark Job
        • Python Job
      • List Components
      • Delete Orphan Pods
      • Scheduler
      • Data Channel
      • Cluster Event
      • Trash
      • Settings
    • Pipeline Workflow Editor
      • Pipeline Toolbar
        • Pipeline Overview
        • Pipeline Testing
        • Search Component in Pipelines
        • Push Pipeline (to VCS/GIT)
        • Pull Pipeline
        • Full Screen
        • Log Panel
        • Event Panel
        • Activate/Deactivate Pipeline
        • Update Pipeline
        • Failure Analysis
        • Pipeline Monitoring
        • Delete Pipeline
      • Component Panel
      • Right-side Panel
    • Testing Suite
    • Activating Pipeline
    • Monitoring Pipeline
  • Components
    • Adding Components to Workflow
    • Component Architecture
    • Component Base Configuration
    • Resource Configuration
    • Intelligent Scaling
    • Connection Validation
    • Readers
      • S3 Reader
      • HDFS Reader
      • DB Reader
      • ES Reader
      • SFTP Stream Reader
      • SFTP Reader
      • Mongo DB Reader
        • MongoDB Reader Lite (PyMongo Reader)
        • MongoDB Reader
      • Azure Blob Reader
      • Azure Metadata Reader
      • ClickHouse Reader (Docker)
      • Sandbox Reader
      • Azure Blob Reader
    • Writers
      • S3 Writer
      • DB Writer
      • HDFS Writer
      • ES Writer
      • Video Writer
      • Azure Writer
      • ClickHouse Writer (Docker)
      • Sandbox Writer
      • MongoDB Writers
        • MongoDB Writer
        • MongoDB Writer Lite (PyMongo Writer)
    • Machine Learning
      • DSLab Runner
      • AutoML Runner
    • Consumers
      • SFTP Monitor
      • MQTT Consumer
      • Video Stream Consumer
      • Eventhub Subscriber
      • Twitter Scrapper
      • Mongo ChangeStream
      • Rabbit MQ Consumer
      • AWS SNS Monitor
      • Kafka Consumer
      • API Ingestion and Webhook Listener
    • Producers
      • WebSocket Producer
      • Eventhub Publisher
      • EventGrid Producer
      • RabbitMQ Producer
      • Kafka Producer
    • Transformations
      • SQL Component
      • Dateprep Script Runner
      • File Splitter
      • Rule Splitter
      • Stored Producer Runner
      • Flatten JSON
      • Email Component
      • Pandas Query Component
      • Enrichment Component
      • Mongo Aggregation
      • Data Loss Protection
      • Data Preparation (Docker)
      • Rest Api Component
      • Schema Validator
    • Scripting
      • Script Runner
      • Python Script
        • Keeping Different Versions of the Python Script in VCS
    • Scheduler
  • Custom Components
  • Advance Configuration & Monitoring
    • Configuration
      • Default Component Configuration
      • Logger
    • Data Channel
    • Cluster Events
    • System Component Status
  • Version Control
  • Use Cases
Powered by GitBook
On this page
  • Please follow the below demonstration to configure SFTP Monitor component.
  • Steps to configure the SFTP Monitor Component
  • Basic Information Tab
  • Meta Information Tab
  • Saving the Component Configuration
  1. Components
  2. Consumers

SFTP Monitor

PreviousConsumersNextMQTT Consumer

Last updated 1 year ago

SFTP (Secure File Transfer Protocol) Monitor is used to monitor and manage files transfer over SFTP servers. It is designed to keep track of file transfers over SFTP.

All component configurations are classified broadly into the following sections:

  • ​​

  • Meta Information

  • ​​

  • ​

Please follow the below demonstration to configure SFTP Monitor component.

Steps to configure the SFTP Monitor Component

  • Drag and Drop the SFTP Monitor consumer component which is inside the consumer section of the system component part to the Workflow Editor.

  • Click the dragged ingestion component to get the component properties tabs.

Basic Information Tab

The Basic Information Tab is the default tab for the component.

  • Select the Invocation Type (at present only the Real-Time option is provided).

  • Deployment Type: It comes preselected based on the component.

  • Container Image Version: It comes preselected based on the component.

  • Failover Event: Select a failover event from the drop-down menu.

  • Batch Size (min 10): Provide the maximum number of records to be processed in one execution cycle (Min limit for this field is 10).

Meta Information Tab

Configure the Meta Information tab for the dragged SFTP Monitor component.

  • Host: Broker IP or URL

  • Username: If authentication is required then give username

  • Port: Provide the Port number

  • Authentication: Select an authentication option using the drop-down list.

    • Password: Provide a password to authenticate SFTP Monitor.

    • PEM/PPK File: Choose a file to authenticate the SFTP Monitor component. The user needs to upload a file if this authentication option has been selected.

  • Directory Path: Fill the monitor folder path using forward-slash (/). E.g., /home/monitor

  • Copy Directory Path: Fill in the copy folder name where you want to copy the uploaded file. E.g., /home/monitor_copy

Please Note: Don't use a nested directory structure in the directory path and copy directory path. Else the component won't behave in an expected manner.

Don't use the dirpath and copy-path as follows: dirpath: home/monitor/datacopy-dir:home/monitor/data/copy_data

  • Channel: Select a channel option from the drop-down menu (the supported channel is SFTP).

Meta Information Fields when the Authentication option is Password

Meta Information Fields when the Authentication option is PEM/PPK

Saving the Component Configuration

  • Click the Save Component in Storage icon to save configured details of the SFTP Monitor component.

  • A notification message appears to confirm the same.

Please Note:

a. The SFTP Monitor component monitors the file coming to the monitored path and copies the file in the Copy Path location for SFTP Reader to read.

b. The SFTP Monitor component requires an Event to send output.​

c. The SFTP Monitor send the file name to the out event along with File size, last modified time and ingestion time (Refer the below image).

d. Only one SFTP monitor will read and move the file if multiple monitors are set up to monitor the same file path at the same time.

​

​Basic Information​
Resource Configuration​
Connection Validation
Out Event data preview for SFTP Monitor