# Understanding Data Ingestion Patterns -Pipeline

### **Streaming Ingestion**

#### **Definition**

Ingests data continuously in real time as it arrives.

#### **Use Cases**

* Capturing sensor data from IoT devices
* Collecting real-time user activity logs (e.g., telecom, automotive)

#### Supported Components

* Kafka Consumer
* MQTT Consumer
* RabbitMQ Consumer
* Azure Event Hub Subscriber
* SFTP Stream Reader

#### **Best Practices**

* Ensure data is idempotent to prevent duplicates
* Implement backpressure handling to manage data flow
* Use windowing techniques for time-based aggregation

### **API-Based Ingestion**

#### **Definition**

Ingests data via APIs exposed by external systems or services.

#### **Use Cases**

* Integrating with SaaS applications
* Fetching data from third-party APIs

#### **Supported Components**

* API Ingestion Module

#### **Best Practices**

* Manage API rate limits and authentication securely
* Use caching strategies to reduce the number of API calls

### **Change Data Capture (CDC)**

#### **Definition**

Captures and replicates data changes (inserts, updates, deletes) from source systems in real time.

#### **Use Cases**

* Synchronizing changes from source databases (e.g., MongoDB)

#### **Supported Components**

* MongoDB Change Stream Listener

#### **Best Practices**

* Ensure the source database supports CDC features
* Monitor replication lag and system performance


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.bdb.ai/bdb-user-documentation/implementation-playbook/bdb-implementation-playbook/discovery-phase/understanding-data-ingestion-patterns-pipeline.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.