Data Pipeline
  • Data Pipeline
    • About Data Pipeline
    • Design Philosophy
    • Low Code Visual Authoring
    • Real-time and Batch Orchestration
    • Event based Process Orchestration
    • ML and Data Ops
    • Distributed Compute
    • Fault Tolerant and Auto-recovery
    • Extensibility via Custom Scripting
  • Getting Started
    • Homepage
      • Create
        • Creating a New Pipeline
          • Adding Components to Canvas
          • Connecting Components
            • Events [Kafka and Data Sync]
          • Memory and CPU Allocations
        • Creating a New Job
          • Job Editor Page
          • Spark Job
            • Readers
              • HDFS Reader
              • MongoDB Reader
              • DB Reader
              • S3 Reader
              • Azure Blob Reader
              • ES Reader
              • Sandbox Reader
              • Athena Query Executer
            • Writers
              • HDFS Writer
              • Azure Writer
              • DB Writer
              • ES Writer
              • S3 Writer
              • Sandbox Writer
              • Mongodb Writer
              • Kafka Producer
            • Transformations
          • PySpark Job
          • Python Job
          • Python Job(On demand)
          • Script Executer Job
          • Job Alerts
        • Register as Job
        • Exporting a Script From Data Science Lab
        • Utility
        • Git Sync
      • Overview
        • Jobs
        • Pipeline
      • List Jobs
      • List Pipelines
      • Scheduler
      • Data Channel & Cluster Events
      • Trash
      • Settings
    • Pipeline Workflow Editor
      • Pipeline Toolbar
        • Pipeline Overview
        • Pipeline Testing
        • Search Component in Pipelines
        • Push & Pull Pipeline
        • Pull Pipeline
        • Full Screen
        • Log Panel
        • Event Panel
        • Activate/Deactivate Pipeline
        • Update Pipeline
        • Failure Analysis
        • Delete Pipeline
        • Pipeline Component Configuration
        • Pipeline Failure Alert History
        • Format Flowchart
        • Zoom In/Zoom Out
        • Update Component Version
      • Component Panel
      • Right-side Panel
    • Testing Suite
    • Activating Pipeline
    • Pipeline Monitoring
    • Job Monitoring
  • Components
    • Adding Components to Workflow
    • Component Architecture
    • Component Base Configuration
    • Resource Configuration
    • Intelligent Scaling
    • Connection Validation
    • Readers
      • GCS Reader
      • S3 Reader
      • HDFS Reader
      • DB Reader
      • ES Reader
      • SFTP Stream Reader
      • SFTP Reader
      • Mongo DB Reader
        • MongoDB Reader Lite (PyMongo Reader)
        • MongoDB Reader
      • Azure Blob Reader
      • Azure Metadata Reader
      • ClickHouse Reader (Docker)
      • Sandbox Reader
      • Azure Blob Reader (Docker)
      • Athena Query Executer
    • Writers
      • S3 Writer
      • DB Writer
      • HDFS Writer
      • ES Writer
      • Video Writer
      • Azure Writer
      • ClickHouse Writer (Docker)
      • Sandbox Writer
      • MongoDB Writers
        • MongoDB Writer
        • MongoDB Writer Lite (PyMongo Writer)
    • Machine Learning
      • DSLab Runner
      • AutoML Runner
    • Consumers
      • GCS Monitor
      • Sqoop Executer
      • OPC UA
      • SFTP Monitor
      • MQTT Consumer
      • Video Stream Consumer
      • Eventhub Subscriber
      • Twitter Scrapper
      • Mongo ChangeStream
      • Rabbit MQ Consumer
      • AWS SNS Monitor
      • Kafka Consumer
      • API Ingestion and Webhook Listener
    • Producers
      • WebSocket Producer
      • Eventhub Publisher
      • EventGrid Producer
      • RabbitMQ Producer
      • Kafka Producer
      • Synthetic Data Generator
    • Transformations
      • SQL Component
      • File Splitter
      • Rule Splitter
      • Stored Producer Runner
      • Flatten JSON
      • Pandas Query Component
      • Enrichment Component
      • Mongo Aggregation
      • Data Loss Protection
      • Data Preparation (Docker)
      • Rest Api Component
      • Schema Validator
    • Scripting
      • Script Runner
      • Python Script
        • Keeping Different Versions of the Python Script in VCS
    • Scheduler
    • Alerts
      • Alerts
      • Email Component
    • Job Trigger
  • Custom Components
  • Advance Configuration & Monitoring
    • Configuration
      • Default Component Configuration
      • Logger
    • Data Channel
    • Cluster Events
    • System Component Status
  • Version Control
  • Use Cases
Powered by GitBook
On this page
  • Configuring the Meta Information of AWS SNS Monitor
  • Steps to create an SNS topic and Get SQS URL
  • Steps to Create an SQS Queue
  • Steps to create an Event Notification in S3 Bucket
  1. Components
  2. Consumers

AWS SNS Monitor

PreviousRabbit MQ ConsumerNextKafka Consumer

Last updated 1 year ago

In AWS, SNS (Simple Notification Service) is a fully managed messaging service that enables you to send notifications and messages to distributed systems and components. SNS Monitor is a feature or functionality related to SNS that allows users to monitor the activity, health, and performance of their SNS topics and messages. It provides metrics and insights into the delivery status, throughput, success rates, and other relevant information about the messages sent through SNS topics.

All component configurations are classified broadly into 3 section:

  • ​

  • Meta Information

  • ​

  • ​​

Configuring the Meta Information of AWS SNS Monitor

  • Access Key: Enter the AWS access key.

  • Secret Key: Enter the AWS secret key.

  • Region: Select the region of the SNS topic.

  • SQS URL: Enter the SQS URL obtained after creating an SQS queue, which will fetch the notification and send it to the out event if there is any modification in the S3 bucket.

Please Note:

  • Follow the below-given steps to set up monitoring for an S3 bucket using AWS SNS monitor:

    • Create an SNS topic in your AWS account.

    • Create an SQS queue that will subscribe to the SNS topic you created earlier.

    • After setting up the SQS queue, obtain the SQS URL associated with it.

    • With the SNS topic and SQS queue configured, you need to create an event notification for the S3 bucket that needs to be monitored.

    • This event notification will be configured to send notifications to the specified SNS topic.

    • Whenever there is a modification in the S3 bucket, the SNS topic will trigger notifications, which will be fetched by the SQS queue using its URL.

    • Finally, these notifications will be sent to the out Event, allowing you to monitor activity within the S3 bucket effectively.

  • Please go through the below given steps to create an SNS topic, SQS queue and Event Notification.

Steps to create an SNS topic and Get SQS URL

  • Sign in to the AWS console.

  • Navigate to the "Services" option or use the search option at the top and select "Simple Notification Service (SNS)".

  • Once redirected to the SNS page, go to the "Topics" option and click on "Create topic".

  • Enter a name and display name for the topic, and optionally, provide a description.

  • Click on "Create topic" to create the SNS topic.

  • Once the topic is created, go to the "Subscriptions" tab and click on "Create subscription".

  • Choose "Amazon SQS" as the protocol.

  • Select the desired SQS queue from the drop-down list or create a new queue if needed.

  • Click on "Create subscription" to link the SQS queue to the SNS topic.

  • After successfully creating the subscription, the SQS URL will be displayed. This URL can be used to receive messages from the SNS topic.

Steps to Create an SQS Queue

  • Sign in to the AWS console.

  • Navigate to the "Services" option or use the search option at the top and select "Simple Queue Service (SQS)".

  • Once redirected to the SQS page, where the list of available queues will be displayed, go to the "Topics" option and click on "Create queue".

  • Enter the queue name and configure any required settings such as message retention period, visibility timeout, etc.

  • Click on the "Create queue" button to create the queue.

  • After successfully creating the queue, select the queue from the list.

  • In the queue details page, navigate to the "Queue Actions" dropdown menu and select "Subscribe to SNS topic".

  • Choose the SNS topic to which you want to subscribe from the dropdown menu.

  • Configure any required parameters such as filter policies and delivery retry policies.

  • Click on the "Subscribe" button to create the subscription.

  • Once the subscription is created successfully, the SQS URL will be displayed in the subscription details.

  • The Obtained URL can be used in the SQS URL field in the AWS SNS monitor component.

Once the SNS and SQS are configured, the user has to create an event notification in a bucket to monitor the activity of the S3 bucket using the AWS SNS monitor component in the pipeline. This involves setting up event notifications within an S3 bucket to trigger notifications whenever certain events occur, such as object creation, deletion, or modification. By configuring these event notifications, users can ensure that relevant events in the S3 bucket are captured and forwarded to the specified SNS topic for further processing or monitoring. This integration allows for seamless monitoring of S3 bucket activities using the AWS SNS monitor component within the pipeline.

Steps to create an Event Notification in S3 Bucket

  • Sign in to the AWS Management Console.

  • Navigate to the "Services" option and select "S3" from the list of available services.

  • Once redirected to the S3 dashboard, locate and click on the desired bucket for which you want to create an event notification.

  • In the bucket properties, navigate to the "Properties" tab.

  • Scroll down to the "Events" section and click on "Create event notification".

  • Provide a name for the event notification configuration.

  • Choose the events that the user wants to trigger notifications for, such as "All object create events", "All object delete events", or specific events based on prefixes or suffixes.

  • Specify the destination for the event notification. Select "Amazon SNS" as the destination type.

  • Choose the SNS topic to which the user wants to publish the event notifications.

  • Optionally, configure any additional settings such as filters based on object key prefixes or suffixes.

  • Review the configuration and click "Save" or "Create" to create the event notification.

  • Once saved, the event notification will be configured for the selected S3 bucket, and notifications will be sent to the specified SNS topic whenever the configured events occur within the bucket. Subsequently, these notifications will be fetched by the SQS URL subscribed to that SNS topic.

Go through the below given demonstration to create SNS topic, SQS ques and Event notification in AWS.

Please Note: The user should ensure that the AWS Bucket, SNS topic, and SQS topic are in the same region to create an event notification.

​Basic Information​
Resource Configuration​
Connection Validation
AWS SNS Monitor
Creating an SNS topic
Getting SQS URL