Azure Metadata Reader

Azure Metadata Reader is designed to read and access metadata associated with Azure resources. Azure Metadata Readers typically authenticate with Azure using Azure Active Directory credentials or other authentication mechanisms supported by Azure.

All component configurations are classified broadly into the following sections:

Please Note: Please go through the below given demonstration to configure Azure Metadata Reader in the pipeline.

Please Note: Before starting to use the Azure Reader component, please follow the steps below to obtain the Azure credentials from the Azure Portal:

Accessing Azure Blob Storage: Shared Access Signature (SAS), Secret Key, and Principal Secret

This document outlines three methods for accessing Azure Blob Storage: Shared Access Signatures (SAS), Secret Keys, and Principal Secrets.

Understanding Security Levels:

  • Shared Access Signature (SAS): This is the recommended approach due to its temporary nature and fine-grained control over access permissions. SAS tokens can be revoked, limiting potential damage if compromised.

  • Secret Key: Secret keys grant full control over your storage account. Use them with caution and only for programmatic access. Consider storing them securely in Azure Key Vault and avoid hardcoding them in scripts.

  • Principal Secret: This applies to Azure Active Directory (Azure AD) application access. Similar to secret keys, use them cautiously and store them securely (e.g., Azure Key Vault).

1. Shared Access Signature (SAS):

Benefits:

  • Secure: Temporary and revocable, minimizing risks.

  • Granular Control: Define specific permissions (read, write, list, etc.) for each SAS token.

Steps to Generate an SAS Token:

  1. Navigate to Azure Portal: Open the Azure portal (https://azure.microsoft.com/en-us/get-started/azure-portal) and log in with your credentials.

  2. Access Blob Storage Account: Locate "Storage accounts" in the left menu and select your storage account.

  3. Configure SAS Settings: Find and click on "Shared access signature" in the settings. Define the permissions, expiry date, and other parameters for your needs.

  4. Generate SAS Token: Click on "Generate SAS and connection string" to create the SAS token.

  5. Copy and Use SAS Token: Copy the generated SAS token. Use this token to securely access your Blob Storage resources in your code.

2. Secret Key:

Use with Caution:

  • High-Risk: Grants full control over your storage account.

  • Secure Storage: Store them securely in Azure Key Vault, never hardcode them in scripts.

Steps to Obtain Secret Key:

  1. Navigate to Azure Portal: Open the Azure portal and log in.

  2. Access Blob Storage Account: Locate and select your storage account.

  3. View Secret Keys: Click on "Access keys" to view your storage account keys. Do not store these directly in code. Consider Azure Key Vault for secure storage.

3. Principal Secret (Azure AD Application):

Use for Application Access:

  • Grants access to your storage account through an Azure AD application.

  • Secure Storage: Store them securely in Azure Key Vault, never hardcode them in scripts.

Steps to Obtain Principal Secret:

  1. Navigate to Azure AD Portal: Open the Azure AD portal (https://azure.microsoft.com/en-us/get-started/azure-portal) and log in with your credentials.

  2. Access App Registrations: Locate "App registrations" in the left menu.

  3. Select Your Application: Find and click on the application for which you want to obtain the principal secret.

  4. Access Certificates & Secrets: Inside your application, go to "Certificates & secrets" in the settings menu.

  5. Generate New Client Secret (Principal Secret):

    • Under "Client secrets," click on "New client secret."

    • Enter a description, select the expiry duration, and click "Add" to generate the new client secret.

    • Copy the generated client secret immediately as it will be hidden afterward.

Configuring the Meta Information of Azure Blob Reader

Read Using: There are three authentication methods available to connect with Azure in the Azure Blob Reader Component:

  • Shared Access Signature

  • Secret Key

  • Principal Secret

Read using Shared Access Signature

Provide the following details:

  1. Shared Access Signature: This is a URI that grants restricted access rights to Azure Storage resources.

  2. Account Name: Provide the Azure account name.

  3. Container: Provide the container name from where the file is located and which has to be read.

  4. Path type: There are options available under it:

    • Null: If Null is selected as the Path Type, the component will read the metadata of all the blobs from the given container. The user does not need to fill the Blob Name field in this option.

    • Directory Path: Enter the directory path to read the metadata of files located in the specified directory. For example: employee/joining_year=2010/department=BI/designation=Analyst/.

    • Blob Name: Specify the blob name to read the metadata from that particular blob.

Read using Secret Key

Provide the following details:

  1. Account Key: It is be used to authorize access to data in your storage account via Shared Key authorization.

  2. Account Name: Provide the Azure account name.

  3. Container: Provide the container name from where the file is located and which has to be read.

  4. Path type: There are options available under it:

    • Null: If Null is selected as the Path Type, the component will read the metadata of all the blobs from the given container. The user does not need to fill the Blob Name field in this option.

    • Directory Path: Enter the directory to read the metadata of files located in the specified directory. For example: employee/joining_year=2010/department=BI/designation=Analyst/.

    • Blob Name: Specify the blob name to read the metadata from that particular blob.

Read using Principal Secret

Provide the following details:

  1. Client ID: The client ID is the unique Application (client) ID assigned to your app by Azure AD when the app was registered.

  2. Tenant ID: It is a globally unique identifier (GUID) that is different than your organization name or domain.

  3. Client Secret: The client secret is the password of the service principal.

  4. Account Name: Provide the Azure account name.

  5. Container: Provide the container name from where the file is located and which has to be read.

  6. Path type: There are options available under it:

    • Null: If Null is selected as the Path Type, the component will read the metadata of all the blobs from the given container. The user does not need to fill the Blob Name field in this option.

    • Directory Path: Enter the directory to read the metadata of files located in the specified directory. For example: employee/joining_year=2010/department=BI/designation=Analyst/.

    • Blob Name: Specify the blob name to read the metadata from that particular blob.

Output Metadata

Once the component runs successfully, it will send the following metadata to the output event:

  • Container: Name of the container where the blob is present.

  • Blob: Name of the blob present in the specified path.

  • blobLastModifiedDateAndTime: Date and time when the blob was last modified.

  • blobLength: Size of the blob.

PreviousAzure Blob ReaderNextClickHouse Reader (Docker)

Last updated