API Ingestion and Webhook Listener

The API Ingestion and Webhook Listener components enable pipelines to receive data directly from third-party services or applications.

  • API Ingestion: Receives data via an HTTP POST request to a generated endpoint.

  • Webhook Listener: Provides a listening endpoint to capture webhook calls from external systems.

Both methods allow external applications to push JSON data into the pipeline for real-time processing.

Configuration Sections

All component configurations are classified into:

  • Basic Information

  • Meta Information

  • Resource Configuration

Basic Information Tab

The Basic Information tab defines execution and deployment settings.

Field
Description
Required

Invocation Type

Select execution mode: Batch or Real-Time.

Yes

Deployment Type

Displays the deployment type of the component (pre-selected).

Yes

Container Image Version

Displays the Docker image version used (pre-selected).

Yes

Failover Event

Select a failover event for error handling.

Optional

Batch Size

Maximum number of records processed in one cycle (minimum: 10).

Yes

Meta Information Tab

The Meta Information tab configures ingestion details and authentication.

Field
Description
Required

Ingestion Type

Select from: API Ingestion or Webhook.

Yes

Ingestion ID

Predefined unique identifier for the component. Used as a header for authentication.

Yes

Ingestion Secret

Predefined secret key for authentication. Used as a header for authentication.

Yes

Once the component is saved, a Component Instance ID URL is generated automatically. This URL serves as the endpoint for API or webhook requests.

Workflow: Configuring API Ingestion

Step 1: Configure Component

  1. Drag and drop the API Ingestion component into the pipeline editor.

  2. Configure the Meta Information tab with:

    • Ingestion Type: API Ingestion

    • Ingestion ID: Auto-generated

    • Ingestion Secret: Auto-generated

  3. Save the component and pipeline.

Step 2: Generate Endpoint

  • After saving the pipeline, the Component Instance ID URL appears in the Meta Information tab.

  • This URL serves as the endpoint to which API requests must be sent.

Step 3: Send Data via Postman

  1. Open Postman (or a similar API testing tool).

  2. Create a new request:

    • Method: POST

    • URL: Component Instance ID URL

  3. Go to Headers tab and add:

    • ingestionId: <value from component>

    • ingestionSecret: <value from component>

  4. Go to Body tab:

    • Select raw → JSON.

    • Enter JSON payload (e.g., { "customerId": 123, "orderId": "ORD456" }).

  5. Click Send.

The API Ingestion component will process the JSON data and pass it to the connected out event in the pipeline.

Workflow: Configuring Webhook Listener

For Webhook ingestion, external systems (e.g., payment processors, SaaS tools) can push data to the generated Component Instance ID URL.

  • Use the same Ingestion ID and Ingestion Secret headers for authentication.

  • The pipeline processes incoming webhook events in real time.

Example Use Case

  • API Ingestion: An e-commerce app posts order data via API calls to the pipeline.

  • Webhook Listener: A payment gateway sends webhook notifications about completed transactions.

Both methods feed real-time JSON data into downstream components for analytics, alerting, or storage.

PreviousKafka ConsumerNextProducers