Data Pipeline
CtrlK
  • Data Pipeline
    • About Data Pipeline
    • Design Philosophy
    • Low Code Visual Authoring
    • Real-time and Batch Orchestration
    • Event based Process Orchestration
    • ML and Data Ops
    • Distributed Compute
    • Fault Tolerant and Auto-recovery
    • Extensibility via Custom Scripting
  • Getting Started
    • Homepage
      • List Pipeline
      • Creating a New Pipeline
        • Adding Components to Canvas
        • Connecting Components
        • Events [Kafka and Data Sync]
        • Memory and CPU Allocations
      • List Components
      • Delete Orphan Pods
      • Scheduler
      • Data Channel
      • Cluster Event
      • Trash
      • Settings
    • Pipeline Workflow Editor
      • Pipeline Toolbar
        • Pipeline Overview
        • Pipeline Testing
        • Search Component in Pipelines
        • Push Pipeline (to VCS/GIT)
        • Pull Pipeline
        • Full Screen
        • Log Panel
        • Event Panel
        • Activate/Deactivate Pipeline
        • Update Pipeline
        • Failure Analysis
        • Pipeline Monitoring
        • Delete Pipeline
      • Component Panel
      • Right-side Panel
    • Testing Suite
    • Activating Pipeline
    • Monitoring Pipeline
  • Components
    • Adding Components to Workflow
    • Component Architecture
    • Component Base Configuration
    • Resource Configuration
    • Intelligent Scaling
    • Connection Validation
    • Readers
      • S3 Reader
      • HDFS Reader
      • DB Reader
      • Elastic Search Reader
      • SFTP Stream Reader
      • SFTP Reader
      • Mongo DB Reader
        • Mongodb Reader Lite
        • Mongo DB Reader
      • Azure Blob Reader
      • Azure Metadata Reader
      • ClickHouse Reader (Docker)
      • Sandbox Reader
    • Writers
      • S3 Writer
      • DB Writer
      • HDFS Writer
      • ES Writer
      • Mongo Writers
        • Mongo Writer (Spark)
        • PyMongo Writer
      • Video Writer
      • Azure Writer
      • ClickHouse Writer (Docker)
      • Sandbox Writer
    • Machine Learning
      • DSLab Runner
      • AutoML Runner
    • Consumers
      • SFTP Monitor
      • MQTT Consumer
      • Video Consumer
      • Eventhub Subscriber
      • Twitter Scrapper
      • Mongo Change Stream
      • Rabbit MQ Consumer
      • AWS SNS Monitor
      • Kafka Consumer
      • Schema Validator
      • API Ingestion and Webhook Listener
    • Producers
      • WebSocket Producer
      • Eventhub Publisher
      • EventGrid Producer
      • RabbitMQ Producer
      • Kafka Producer
    • Transformations
      • Data Preparation (Docker)
      • SQL Component
      • Dateprep Script Runner
      • File Splitter
      • Rule Splitter
      • Stored Producer Runner
      • Flatten JSON
      • Email Component
      • Pandas Query Component
      • Enrichment Component
      • Mongo Aggregation
      • Rest Api Component
      • Data Loss Protection
    • Scripting
      • Script Runner
      • Python Script
        • Keeping Different Versions of the Python Script in VCS
    • Scheduler
  • Custom Components
  • Advance Configuration & Monitoring
    • Configuration
      • Kafka Configuration
      • Default Component Configuration
      • Logger
    • Data Channel
    • Cluster Events
    • System Component Status
  • Version Control
  • Jobs
    • Transformations
  • Use Cases
Powered by GitBook
On this page
  1. Components

Transformations

Data Preparation (Docker)SQL ComponentDateprep Script RunnerFile SplitterRule SplitterStored Producer RunnerFlatten JSONEmail ComponentPandas Query ComponentEnrichment ComponentMongo AggregationRest Api ComponentData Loss Protection
PreviousKafka ProducerNextData Preparation (Docker)

Last updated 2 years ago