Azure Blob Reader

Azure Blob Reader is designed to read and access data stored in Azure Blob Storage. Azure Blob Readers typically authenticate with Azure Blob Storage using Azure Active Directory credentials or other authentication mechanisms supported by Azure. This is a spark based component.

All component configurations are classified broadly into the following sections:

• ​​Basic Information​​

• Meta Information

• ​Resource Configuration​​

• ​Connection Validation

Please Note: Please go through the below given demonstration to configure Azure Blob Reader in the pipeline.

Azure Blob Reader

Please Note: Before starting to use the Azure Reader component, please follow the steps below to obtain the Azure credentials from the Azure Portal:

Accessing Azure Blob Storage: Shared Access Signature (SAS), Secret Key, and Principal Secret

This document outlines three methods for accessing Azure Blob Storage: Shared Access Signatures (SAS), Secret Keys, and Principal Secrets.

Understanding Security Levels:

• Shared Access Signature (SAS): This is the recommended approach due to its temporary nature and fine-grained control over access permissions. SAS tokens can be revoked, limiting potential damage if compromised.

• Secret Key: Secret keys grant full control over your storage account. Use them with caution and only for programmatic access. Consider storing them securely in Azure Key Vault and avoid hardcoding them in scripts.

• Principal Secret: This applies to Azure Active Directory (Azure AD) application access. Similar to secret keys, use them cautiously and store them securely (e.g., Azure Key Vault).

1. Shared Access Signature (SAS):

Benefits:

• Secure: Temporary and revocable, minimizing risks.

• Granular Control: Define specific permissions (read, write, list, etc.) for each SAS token.

Steps to Generate an SAS Token:

1. Navigate to Azure Portal: Open the Azure portal (https://azure.microsoft.com/en-us/get-started/azure-portal) and log in with your credentials.
2. Access Blob Storage Account: Locate "Storage accounts" in the left menu and select your storage account.

3. Configure SAS Settings: Find and click on "Shared access signature" in the settings. Define the permissions, expiry date, and other parameters for your needs.

4. Generate SAS Token: Click on "Generate SAS and connection string" to create the SAS token.

5. Copy and Use SAS Token: Copy the generated SAS token. Use this token to securely access your Blob Storage resources in your code.

2. Secret Key:

Use with Caution:

• High-Risk: Grants full control over your storage account.

• Secure Storage: Store them securely in Azure Key Vault, never hardcode them in scripts.

Steps to Obtain Secret Key:

1. Navigate to Azure Portal: Open the Azure portal and log in.

2. Access Blob Storage Account: Locate and select your storage account.

3. View Secret Keys: Click on "Access keys" to view your storage account keys. Do not store these directly in code. Consider Azure Key Vault for secure storage.

3. Principal Secret (Azure AD Application):

Use for Application Access:

• Grants access to your storage account through an Azure AD application.

• Secure Storage: Store them securely in Azure Key Vault, never hardcode them in scripts.

Steps to Obtain Principal Secret:

1. Navigate to Azure AD Portal: Open the Azure AD portal (https://azure.microsoft.com/en-us/get-started/azure-portal) and log in with your credentials.
2. Access App Registrations: Locate "App registrations" in the left menu.

3. Select Your Application: Find and click on the application for which you want to obtain the principal secret.

4. Access Certificates & Secrets: Inside your application, go to "Certificates & secrets" in the settings menu.

5. Generate New Client Secret (Principal Secret):

   • Under "Client secrets," click on "New client secret."

   • Enter a description, select the expiry duration, and click "Add" to generate the new client secret.

   • Copy the generated client secret immediately as it will be hidden afterward.

Configuring the Meta Information of Azure Blob Reader

Read Using: There are three authentication methods available to connect with Azure in the Azure Blob Reader Component:

• Shared Access Signature

• Secret Key

• Principal Secret

Read using Shared Access Signature

Provide the following details:

• Shared Access Signature: This is a URI that grants restricted access rights to Azure Storage resources.

• Account Name: Provide the Azure account name.

• Container: Provide the container name from where the file is located and which has to be read.

• File Type: There are five (5) types of file extensions available:

  • CSV

  • JSON

  • PARQUET

  • AVRO

  • XML

• Read Directory: This field will be checked by default. If this option is enabled, the component will read data from all the blobs present in the container.

• Blob Name: This field will display only if the Read Directory field is disabled. Enter the specific name of the blob whose data has to be read.

• Limit: Enter a number to limit the number of records that has to be read by the component.

• Column Filter: Enter the column names here. Only the specified columns will be fetched from Azure Blob. In this field, the user needs to fill in the following information:

  • Source Field: Enter the name of the column from the blob. The user can add multiple columns by clicking on the "Add New Column" option.

  • Destination Field: Enter the alias name for the source field.

  • Column Type: Enter the data type of the column.

  • Upload: This option allows the user to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

  • Download Data: This option will download the data filled in the Column Filter field in JSON format.

  • Delete Data: This option will clear all the information filled in the Column Filter field.

• Query: Enter a Spark SQL query in this field. Use inputDf as the table name.

//Sample Spark SQL Query:

Select * from inputDf where department='sales';

Reading Using Shared Access Signature

Read Using Secret Key Option

Provide the following details:

• Account Key: Used to authorize access to data in your storage account via Shared Key authorization.

• Account Name: Provide the Azure account name.

• Container: Provide the container name from where the file is located and which has to be read.

• File Type: There are five (5) types of file extensions available:

  • CSV

  • JSON

  • PARQUET

  • AVRO

  • XML

• Read Directory: This field will be checked by default. If this option is enabled, the component will read data from all the blobs present in the container.

• Blob Name: This field will display only if the Read Directory field is disabled. Enter the specific name of the blob whose data has to be read.

• Limit: Enter a number to limit the number of records that has to be read by the component.

• Column Filter: Enter the column names here. Only the specified columns will be fetched from Azure Blob. In this field, the user needs to fill in the following information:

  • Source Field: Enter the name of the column from the blob. The user can add multiple columns by clicking on the "Add New Column" option.

  • Destination Field: Enter the alias name for the source field.

  • Column Type: Enter the data type of the column.

  • Upload: This option allows the user to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

  • Download Data: This option will download the data filled in the Column Filter field in JSON format.

  • Delete Data: This option will clear all the information filled in the Column Filter field.

• Query: Enter a Spark SQL query in this field. Use inputDf as the table name.

Reading Using Secret Key option

Read Using Principal Secret

Provide the following details:

• Client ID: The unique Application (client) ID assigned to your app by Azure AD when the app was registered.

• Tenant ID: A globally unique identifier (GUID) that is different from your organization name or domain.

• Client Secret: The password of the service principal.

• Account Name: Provide the Azure account name.

• File Type: There are five (5) types of file extensions available:

  • CSV

  • JSON

  • PARQUET

  • AVRO

  • XML

• Read Directory: This field will be checked by default. If this option is enabled, the component will read data from all the blobs present in the container.

• Blob Name: This field will display only if the Read Directory field is disabled. Enter the specific name of the blob whose data has to be read.

• Limit: Enter a number to limit the number of records that has to be read by the component.

• Column Filter: Enter the column names here. Only the specified columns will be fetched from Azure Blob. In this field, the user needs to fill in the following information:

  • Source Field: Enter the name of the column from the blob. The user can add multiple columns by clicking on the "Add New Column" option.

  • Destination Field: Enter the alias name for the source field.

  • Column Type: Enter the data type of the column.

  • Upload: This option allows the user to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

  • Download Data: This option will download the data filled in the Column Filter field in JSON format.

  • Delete Data: This option will clear all the information filled in the Column Filter field.

• Query: Enter a Spark SQL query in this field. Use inputDf as the table name.

Reading Using Principal Secret option

Note: The following fields will be displayed after selecting the following file types:

• CSV: The Header and Infer Schema fields get displayed with CSV as the selected File Type. Enable Header option to get the Header of the reading file and enable Infer Schema option to get true schema of the column in the CSV file.

• JSON: The Multiline and Charset fields get displayed with JSON as the selected File Type.

  • Multiline: This option handles JSON files that contain records spanning multiple lines. Enabling this ensures the JSON parser reads multiline records correctly.

  • Charset: Specify the character set used in the JSON file. This defines the character encoding of the JSON file, such as UTF-8 or ISO-8859-1, ensuring correct interpretation of the file content.

• PARQUET: No extra field gets displayed with PARQUET as the selected File Type.

• AVRO: This File Type provides two drop-down menus.

  • Compression: Select an option out of the Deflate and Snappy options.

    • Deflate: A compression algorithm that balances between compression speed and compression ratio, often resulting in smaller file sizes.

    • Snappy: This compression type is select by default. A fast compression and decompression algorithm developed by Google, optimized for speed rather than maximum compression ratio.

  • Compression Level: This field appears if Deflate compression is selected. It provides a drop-down menu with levels ranging from 0 to 9, indicating the compression intensity.