Notification Publisher
The Notification Publisher component can be added at any point in the pipeline to configure proactive notifications.
The Notification Publisher is a powerful and flexible component within the BDB Data Pipeline, enabling integration with enterprise-grade cloud messaging services such as AWS SNS and Google Pub/Sub. By configuring alert components, users can implement proactive monitoring and ensure that operational teams receive timely updates, improving pipeline reliability and observability.
When placed inside a pipeline, the component publishes a message every time it is executed—typically after a success, warning, or failure node.
Using the Notification Publisher
After dragging the Notification Publisher into the canvas, clicking the component opens its configuration panel, which has two tabs:
Basic Information
Meta Information (platform-specific settings)
The panel dynamically updates based on the selected Notification Type, as shown in the provided images.
Basic Information Tab
This tab contains general details such as component name and optional descriptive metadata. These fields help the user identify and manage alert components in large, production-scale pipelines.
Meta Information Tab
The Meta Information section defines cloud-specific connection parameters and message-publishing details. The configuration options vary depending on the selected notification provider.
If the user selects SNS as the Notification Type, the following fields are displayed:
Notification Type*
Select SNS to publish messages using Amazon Simple Notification Service.
Access Key*
AWS Access Key for authentication.
Secret Key*
AWS Secret Access Key.
Region
AWS region where the SNS topic is hosted (example: us-east-1).
Topic ARN
Full Amazon Resource Name of the SNS topic (example: arn:aws:sns:us-east-1:123456789012:MyTopic).
Configuring Google Pub/Sub
If Google Pub/Sub is selected, the meta panel updates to show Google-specific fields:
Notification Type*
Select Google Pub/Sub to push notifications to a Pub/Sub topic.
Pub/Sub Topic ID*
Identifier of the Pub/Sub topic to which the notification will be published.
Authentication File Name
JSON key file of a Google Cloud service account with Pub/Sub Publisher role.
Execution Behavior
Once configured and placed in a pipeline:
The component receives input (typically a text message or structured JSON).
It publishes the content to SNS or Pub/Sub.
External subscribers receive the alert based on their subscription type (email, SMS, webhook, analytics service, etc.).
If the component fails to publish the alert, the pipeline logs an error in the execution view.
Best Practices
To align with enterprise alerting standards, consider the following:
Message Design
Use structured messages (JSON) for multi-system integrations.
Include pipeline name, timestamp, error details, and run ID for quick troubleshooting.
Security
Rotate cloud keys periodically.
Avoid embedding credentials in scripts—use secure vaults when available.
Restrict IAM roles to publisher-only permissions.
Operational Reliability
Include alerts for both failure and successful completion in production pipelines.
Use different SNS topics or Pub/Sub channels based on severity levels (info, warning, critical).
Example Use Cases
Pipeline Completion Notification
Send a message when a batch job completes, and downstream workflows must start.
Data Quality Alerts
Trigger notifications when row-count validation or schema checks fail.
Threshold-Based Monitoring
Publish alerts when KPIs such as data volume, latency, or record anomalies exceed expected limits.
Last updated