Notification Subscriber

The Notification Subscriber is a reader component within the BDB Data Pipeline that enables pipelines to consume messages from external cloud-based notification and messaging services.

The Notification Subscriber is a robust cloud-integrated reader component that enables BDB Data Pipelines to operate in event-driven and alert-driven architectures. With support for both AWS SNS/SQS and Google Pub/Sub, it allows seamless ingestion of real-time notifications and operational triggers into enterprise data workflows.

This component is typically used to integrate event-driven workflows, automate data movements based on upstream triggers, or consume alerts generated by external systems.

Overview

The Notification Subscriber listens to a configured cloud messaging endpoint and retrieves incoming messages during pipeline execution. Once messages are received, they are passed as output to subsequent steps—enabling fully event-driven data pipelines.

Supported cloud services include:

Provider	Supported Subscription Mode
AWS SNS/SQS	Subscribe to SNS topics using SQS endpoints
Google Pub/Sub	Subscribe to a Pub/Sub topic subscription

The configuration panel for the component includes two tabs:

• Basic Information – Component name and description

• Meta Information – Cloud provider–specific subscription and authentication details

The images provided demonstrate how the Meta Information panel dynamically updates based on the selected Notification Type.

Component Placement

The Notification Subscriber is added to the pipeline canvas from:

Data Engineering → Data Pipelines → Components → Readers

It serves as a Reader, meaning it introduces messages into the pipeline at runtime. It can be placed at the beginning of a pipeline or midstream to trigger conditional branches based on received notifications.

Basic Information Tab

This tab contains standard fields such as:

• Component Name

• Description (optional)

These fields help identify the subscriber within large enterprise workflows.

Meta Information Tab

The Meta Information tab contains all required configuration details for connecting to an external cloud notification service. The fields displayed depend on the selected Notification Type.

AWS SNS/SQS Configuration

When SNS is selected as the Notification Type, the panel displays AWS subscription fields, as shown below:

Field	Description
Notification Type*	Select SNS to subscribe to AWS SNS messages via an SQS queue.
Access Key*	AWS Access Key ID used to authenticate the subscriber.
Secret Key*	AWS Secret Access Key.
SQS URL	Fully-qualified SQS queue URL configured as an SNS subscription endpoint (e.g., https://sqs.us-east-1.amazonaws.com/123456789012/my-queue).
Region	AWS region where the SQS queue resides.

Operational Notes:

• AWS SNS does not support direct message consumption; therefore, an SQS queue subscribed to the SNS topic is required.

• The configured Access Key must have permissions:

  • sqs:ReceiveMessage

  • sqs:DeleteMessage

  • sns:ListSubscriptions (optional)

• The queue policy must allow the SNS topic to publish messages.

Google Pub/Sub Configuration

When Google Pub/Sub is selected, the configuration panel updates as shown.

Field	Description
Notification Type*	Select Google Pub/Sub to read messages from a Pub/Sub subscription.
Project ID*	Google Cloud project identifier that owns the Pub/Sub resources.
Subscription Name*	Name of the Pub/Sub subscription the pipeline will read from. Ex: my-event-subscription.
Authentication File Name	JSON key file for a Google Cloud Service Account with Pub/Sub Subscriber role.

Operational Notes:

• The Service Account must have:

  • pubsub.subscriptions.pull

  • pubsub.subscriptions.acknowledge

• Ensure the subscription exists and is active under the provided Project ID.

• A single subscription can have multiple consumers, but this may impact message delivery semantics, depending on the subscription type (pull mode).

Execution Behavior

During execution:

1. The subscriber component connects to the configured cloud endpoint.

2. It fetches messages available in the queue/subscription.

3. Messages are passed forward in the pipeline as structured output.

4. For SQS and Pub/Sub, messages are acknowledged or deleted after successful processing.

Failure Handling

If authentication fails or the subscription is unreachable:

• The pipeline run is marked Failed.

• Error logs specify the underlying cloud-provided error.

• No messages are deleted or acknowledged unless they are successfully consumed.

Typical Use Cases

Event-Driven Pipeline Execution

Trigger downstream transformations when an external service publishes updates—for example:

• A data ingestion event from an ERP system

• A notification about the file arrival

• A threshold breach from a monitoring system

Alert Consumption

Use the Notification Subscriber together with the Notification Publisher to build:

• Bidirectional alerting systems

• Audit pipelines

• Automated incident response flows

Streaming-style Message Processing

Consume continuous messages and push them into:

• Databases

• Data Lake zones

• Alert dashboards

• Machine learning inference pipelines

Best Practices

Credential Management

• Rotate keys periodically.

• Use least-privilege IAM/Service Account roles.

• Ensure the authentication file is securely uploaded and stored.

Subscription Architecture

• For AWS, always create one SQS queue per subscriber group to avoid message contention.

• For Pub/Sub, consider dead-letter queues for unprocessable messages.

Data Handling

• Standardize message payload formats (JSON recommended).

• Include timestamps, event source IDs, and message type identifiers.