Pipeline Components

This section describes various pipeline components in details.

Pipeline components are classified into two main types: System Components and Custom Components.

System Components

The BDB Data Pipeline offers a comprehensive set of system components designed to perform diverse tasks, enabling the delivery of well-structured and meaningful data to users for faster and more effective data analysis.

• AI Agents: These functional units are designed to apply artificial intelligence or machine learning techniques to incoming data as it flows through the pipeline.

• Readers: Your repository of data can serve as a valuable resource for you. It could be a database, a file, or a SaaS application. Read Readers.

• Writers: The databases or data warehouses to which the Pipelines load the data. Read Writers.

• Transforms: The series of transformation components that help to cleanse, enrich, and prepare data for smooth analytics. Read Transformations.​

• Producers: Producers are the components that can be used to produce/generate streaming data to external sources. Read Producers.

• Machine Learning: The Model Runner components allow users to use the models created using the Python workspace of the Data Science Workbench or saved models from the Data Science Lab to be consumed in a pipeline. Read Machine Learning.​

• Consumers: These are the real-time / Streaming component that ingests data or monitor for changes in data objects from different sources to the pipeline. Read Consumers.​

• Alerts: These components facilitate user notification on various channels like Teams, Slack, and email based on their preferences. Notifications can be delivered for success, failure, or other relevant events, depending on the user's requirements. Read Alerts.

• Scripting: The Scripting components allow users to write custom scripts and integrate them into the pipeline as needed. Read Scripting.

• Scheduler: The Scheduler component enables users to schedule their pipeline at a specific time according to their requirements.

Custom Components

In addition to the predefined system components, the BDB Data Pipeline provides users with the flexibility to create and integrate custom components, enabling them to tailor the pipeline according to their specific operational requirements and use cases.