# Twitter Scrapper

The **Twitter Scraper** component is used to fetch tweets from Twitter based on a **specified hashtag**. It supports fetching both **historical tweets** and **real-time streams**, making it useful for:

* Social media sentiment analysis
* Brand monitoring
* Event tracking and trend analysis

### Configuration Sections

All configurations are classified into the following sections:

* **Basic Information**
* **Meta Information**
* **Resource Configuration**
* **Connection Validation**

### Basic Information Tab

The *Basic Information* tab defines general execution settings.

| Field                       | Description                                                               | Required |
| --------------------------- | ------------------------------------------------------------------------- | -------- |
| **Invocation Type**         | Select execution mode: **Batch** or **Real-Time**.                        | Yes      |
| **Deployment Type**         | Displays the deployment type of the component (pre-selected).             | Yes      |
| **Container Image Version** | Displays the Docker image version used (pre-selected).                    | Yes      |
| **Failover Event**          | Select a failover event to handle retries or errors.                      | Optional |
| **Batch Size**              | Maximum number of records processed in one execution cycle (minimum: 10). | Yes      |

### Meta Information Tab

The *Meta Information* tab defines authentication and query parameters for fetching tweets.

| Field                       | Description                                                                                                                   | Required |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------- |
| **Consumer API Key**        | API key provided by Twitter Developer Portal.                                                                                 | Yes      |
| **Consumer API Secret Key** | Secret key (acts as password) associated with the Consumer API Key.                                                           | Yes      |
| **Filter Text**             | Hashtag or keyword to filter tweets (e.g., `#AI`, `#BigData`).                                                                | Yes      |
| **Twitter Data Type**       | Select one of the following options: **History** (fetch past tweets) or **Real-Time** (fetch live tweets as they are posted). | Yes      |

### Saving the Configuration

1. Enter API credentials and filter details in the **Meta Information** tab.
2. Click the **Save Component** (Storage icon).
3. A success message confirms that the component properties have been saved.
4. Activate the pipeline to start fetching tweets.

### Example Workflow

1. Configure Twitter Scraper with:
   * **Consumer API Key**: `xxxxx`
   * **Consumer API Secret Key**: `yyyyy`
   * **Filter Text**: `#ClimateChange`
   * **Twitter Data Type**: `Real-Time`
2. Start the pipeline.
3. Tweets containing `#ClimateChange` are ingested into the pipeline and passed to downstream components for sentiment analysis and dashboard visualization.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.bdb.ai/bdb-user-documentation/platform-modules/11.0/data-engineering/data-pipelines/pipeline-editor/pipeline-components/consumers/twitter-scrapper.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.