AWS SNS Monitor
The AWS SNS (Simple Notification Service) Monitor component enables monitoring of SNS topics and messages in real time. It integrates with Amazon SQS queues and S3 bucket event notifications to capture and forward activity to downstream pipeline events.
This component provides a reliable way to track:
S3 bucket object events (create, update, delete).
Delivery status of SNS messages.
Notifications from subscribed SNS topics.
By configuring the monitor with the appropriate AWS credentials and queue URL, you can seamlessly integrate AWS event-driven workflows into your pipeline.
Configuration Sections
All component configurations are grouped into the following sections:
Basic Information
Meta Information
Resource Configuration
Connection Validation
Basic Information Tab
The Basic Information tab defines execution and deployment settings.
Invocation Type
Select execution mode: Batch or Real-Time.
Yes
Deployment Type
Displays the deployment type of the component (pre-selected).
Yes
Container Image Version
Displays the Docker image version used (pre-selected).
Yes
Failover Event
Select a failover event to handle retries or errors.
Optional
Batch Size
Maximum number of records processed in one execution cycle (minimum: 10).
Yes
Meta Information Tab
The Meta Information tab configures AWS credentials and SNS/SQS integration details.
Access Key
AWS Access Key for authentication.
Yes
Secret Key
AWS Secret Key for authentication.
Yes
Region
AWS region where the SNS topic and SQS queue are created.
Yes
SQS URL
URL of the SQS queue subscribed to the SNS topic. The SNS Monitor fetches messages from this queue.
Yes
End-to-End Setup Guide
To configure monitoring with AWS SNS Monitor, complete the following steps:
Step 1: Create an SNS Topic
Sign in to the AWS Management Console.
Navigate to Simple Notification Service (SNS).
Click Create topic and provide a name, display name, and optional description.
Once created, go to the Subscriptions tab → Create subscription.
Select Amazon SQS as the protocol.
Choose an existing SQS queue or create a new one.
Click Create subscription. The SQS URL will be available for configuration.
Step 2: Create an SQS Queue
In the AWS Console, navigate to Simple Queue Service (SQS).
Click Create queue and enter a name.
Configure optional settings (message retention, visibility timeout, etc.).
Once created, select the queue from the list.
In Queue Actions, select Subscribe to SNS topic.
Choose the SNS topic created earlier.
Save the subscription. The SQS URL will be displayed in subscription details.
Step 3: Configure S3 Event Notification
In the AWS Console, go to S3 and select the bucket to monitor.
Navigate to the Properties tab → Events → Create event notification.
Provide a name for the event.
Select the events to monitor (e.g., All object create events, All object delete events).
Set the destination to Amazon SNS and choose the SNS topic created earlier.
Optionally add filters (prefix/suffix) to restrict events.
Save the event notification.
Step 4: Configure AWS SNS Monitor
Open the pipeline editor and add the AWS SNS Monitor component.
Enter the following details in the Meta Information Tab:
Access Key / Secret Key
Region
SQS URL
Save the configuration.
Activate the pipeline.
When an event occurs in the monitored S3 bucket, it is published to the SNS topic, forwarded to the SQS queue, and consumed by the AWS SNS Monitor, which then sends it to the pipeline’s output event.
Notes
Ensure the S3 bucket, SNS topic, and SQS queue are in the same AWS region.
Only one SNS Monitor should be configured for a given SQS URL to avoid duplicate processing.
Example Workflow
Create an S3 event notification for a bucket storing customer documents.
Link it to an SNS topic → subscribed by an SQS queue.
Configure AWS SNS Monitor with the SQS URL.
When a new document is uploaded, the SNS Monitor pushes metadata about the event into the pipeline for logging and downstream analytics.