Azure Writer

Azure Writer component is designed to write or store data in Microsoft Azure's storage services, such as Azure Blob Storage. Azure Writers typically authenticate with Azure using Azure Active Directory credentials or other authentication mechanisms supported by Azure.

All component configurations are classified broadly into the following sections:

Please go through the demonstration to configure Azure Writer in the pipeline.

Please Note: Before starting to use the Azure Reader component, please follow the steps below to obtain the Azure credentials from the Azure Portal:

Accessing Azure Blob Storage: Shared Access Signature (SAS), Secret Key, and Principal Secret

This document outlines three methods for accessing Azure Blob Storage: Shared Access Signatures (SAS), Secret Keys, and Principal Secrets.

Understanding Security Levels:

  • Shared Access Signature (SAS): This approach is recommended due to its temporary nature and fine-grained control over access permissions. SAS tokens can be revoked, limiting potential damage if compromised.

  • Secret Key: Secret keys grant full control over your storage account. Use them with caution and only for programmatic access. Consider storing them securely in Azure Key Vault and avoid hardcoding them in scripts.

  • Principal Secret: This applies to Azure Active Directory (Azure AD) application access. Similar to secret keys, use them cautiously and store them securely (e.g., Azure Key Vault).

1. Shared Access Signature (SAS):

Benefits:

  • Secure: Temporary and revocable, minimizing risks.

  • Granular Control: Define specific permissions (read, write, list, etc.) for each SAS token.

Steps to Generate an SAS Token:

  1. Navigate to Azure Portal: Open the Azure portal (https://azure.microsoft.com/en-us/get-started/azure-portal) and log in with your credentials.

  2. Access Blob Storage Account: Locate Storage accounts from the left menu and select your storage account.

  3. Configure SAS Settings: Find and click on "Shared access signature" in the settings. Define the permissions, expiry date, and other parameters for your needs.

  4. Generate SAS Token: Click on "Generate SAS and connection string" to create the SAS token.

  5. Copy and Use SAS Token: Copy the generated SAS token. Use this token to access your Blob Storage resources in your code securely.

2. Secret Key:

Use with Caution:

  • High-Risk: Grants full control over your storage account.

  • Secure Storage: Store them securely in Azure Key Vault, never hardcode them in scripts.

Steps to Obtain Secret Key:

  1. Navigate to Azure Portal: Open the Azure portal and log in.

  2. Access Blob Storage Account: Locate and select your storage account.

  3. View Secret Keys: Click the Access keys to view your storage account keys. Do not store these directly in code. Consider Azure Key Vault for secure storage.

3. Principal Secret (Azure AD Application):

Use for Application Access:

  • Grants access to your storage account through an Azure AD application.

  • Secure Storage: Store them securely in Azure Key Vault, never hardcode them in scripts.

Steps to Obtain Principal Secret:

  1. Navigate to Azure AD Portal: Open the Azure AD portal (https://azure.microsoft.com/en-us/get-started/azure-portal) and log in with your credentials.

  2. Access App Registrations: Locate "App registrations" in the left menu.

  3. Select Your Application: Find and click on the application you want to obtain the principal secret.

  4. Access Certificates & Secrets: Go to Certificates & secrets in the Settings menu inside your application.

  5. Generate New Client Secret (Principal Secret):

    • Click on the New client secret option under the Client secrets section.

    • Enter a description, select the expiry duration, and click the Add option to generate the new client secret.

    • Copy the generated client secret immediately as it will be hidden afterward.

Configuring the Meta Information of Azure Writer

Write Using: There are three authentication methods available to connect with Azure in the Azure Writer Component:

  • Shared Access Signature

  • Secret Key

  • Principal Secret

Write Using Shared Access Signature

Provide the following details:

  • Shared Access Signature: This is a URI that grants restricted access rights to Azure Storage resources.

  • Account Name: Provide the Azure account name.

  • Container: Provide the container name from where the blob is located. A container is a logical unit of storage in Azure Blob Storage that can hold blobs. It is similar to a directory or folder in a file system, and it can be used to organize and manage blobs.

  • Blob Name: Enter the blob name. A blob is a type of object storage used to store unstructured data, such as text or binary data, like images or videos.

  • File Format: Four (4) types of file types are available. Select the file format in which the data has to be written:

    • CSV

    • JSON

    • PARQUET

    • AVRO

  • Save Mode: Select the save mode from the drop-down menu:

    • Append: It will append the data in the blob.

    • Overwrite: It will overwrite the data in the blob.

  • Schema File Name: Upload a Spark schema file of the data that has to be written in JSON format.

  • Column Filter: Enter the column names here. Only the specified columns will be fetched from the data from the previous connected event to the Azure Writer. In this field, the user needs to fill in the following information:

    • Name: Enter the name of the column that has to be written from the previous event. The user can add multiple columns by clicking on the "Add New Column" option.

    • Alias: Enter the alias name for the selected column name.

    • Column Type: Enter the data type of the column.

    • Upload: This option allows the user to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

    • Download Data: This option will download the data filled in the Column Filter field in the JSON format.

    • Delete Data: This option will clear all the information filled in the Column Filter field.

  • Partition Column: This feature enables users to partition the data when writing to Azure Blob. Users can specify multiple columns for partitioning by clicking the "Add Column Name" option.

Write Using Secret Key Option:

Provide the following details:

  • Account Key: Enter the Azure account key. In Azure, an account key is a security credential that is used to authenticate access to storage resources, such as blobs, files, queues, or tables, in an Azure storage account.

  • Account Name: Provide the Azure account name.

  • Container: Provide the container name from where the blob is located. A container is a logical unit of storage in Azure Blob Storage that can hold blobs. It is similar to a directory or folder in a file system, and it can be used to organize and manage blobs.

  • Blob Name: Enter the Blob name. A blob is a type of object storage that is used to store unstructured data, such as text or binary data, like images or videos.

  • File Format: There are four (4) types of file extensions available:

    • CSV

    • JSON

    • PARQUET

    • AVRO

  • Save Mode: Select the save mode from the drop-down menu:

    • Append: It will append the data in the blob.

    • Overwrite: It will overwrite the data in the blob.

  • Schema File Name: Upload a Spark schema file of the data which has to be written in JSON format.

  • Column Filter: Enter the column names here. Only the specified columns will be fetched from the data from the previous connected event to the Azure Writer. In this field, the user needs to fill in the following information:

    • Name: Enter the name of the column which has to be written from the previous event. The user can add multiple columns by clicking on the "Add New Column" option.

    • Alias: Enter the alias name for the selected column name. The column name given here will be written in the container.

    • Column Type: Enter the data type of the column.

    • Upload: This option allows users to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

    • Download Data: This option will download the data filled in the Column Filter field in the JSON format.

    • Delete Data: This option will clear all the information filled in the Column Filter field.

  • Partition Column: This feature enables users to partition the data when writing to Azure Blob. Users can specify multiple columns for partitioning by clicking the "Add Column Name" option.

Write using Principal Secret:

Provide the following details:

  • Client ID: Provide Azure Client ID. The client ID is the unique Application (client) ID assigned to your app by Azure AD when the app was registered.

  • Tenant ID: Provide the Azure Tenant ID. Tenant ID (also known as Directory ID) is a unique identifier that is assigned to an Azure AD tenant and represents an organization or a developer account. It is used to identify the organization or developer account that the application is associated with.

  • Client Secret: Enter the Azure Client Secret. Client Secret (also known as Application Secret or App Secret) is a secure password or key that is used to authenticate an application to Azure AD.

  • Account Name: Provide the Azure account name.

  • Container: Provide the container name from where the blob is located. A container is a logical unit of storage in Azure Blob Storage that can hold blobs. It is similar to a directory or folder in a file system, and it can be used to organize and manage blobs.

  • Blob Name: Enter the Blob name. A blob is a type of object storage that is used to store unstructured data, such as text or binary data, like images or videos.

  • File Format: There are four (4) types of file extensions available under it:

    • CSV

    • JSON

    • PARQUET

    • AVRO

  • Save Mode: Select the save mode from the drop-down menu:

    • Append: It will append the data in the blob.

    • Overwrite: It will overwrite the data in the blob.

  • Schema File Name: Upload a Spark schema file of the data which has to be written in JSON format.

  • Column Filter: Enter the column names here. Only the specified columns will be fetched from the data from the previous connected event to the Azure Writer. In this field, the user needs to fill in the following information:

    • Name: Enter the column name that must be written from the previous event. The user can add multiple columns by clicking the Add New Column option.

    • Alias: Enter the alias name for the selected column name. The column name given here will be written in the container.

    • Column Type: Enter the data type of the column.

    • Upload: This option allows the user to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

    • Download Data: This option will download the data filled in the Column Filter field in JSON format.

    • Delete Data: This option will clear all the information filled in the Column Filter field.

  • Partition Column: This feature enables users to partition the data when writing to Azure Blob. Users can specify multiple columns for partitioning by clicking the "Add Column Name" option.

PreviousVideo WriterNextClickHouse Writer (Docker)

Last updated