Azure Blob Reader

Azure Blob Reader is designed to read and access data stored in Azure Blob Storage. Azure Blob Readers typically authenticate with Azure Blob Storage using Azure Active Directory credentials or other authentication mechanisms supported by Azure. This is a spark based component.

All component configurations are classified broadly into the following sections:

Please Note: Please go through the below given demonstration to configure Azure Blob Reader in the pipeline.

Azure Blob Reader

Configuring the Meta Information of Azure Blob Reader

Read Using: There are three authentication methods available to connect with Azure in the Azure Blob Reader Component:

  • Shared Access Signature

  • Secret Key

  • Principal Secret

Read using Shared Access Signature

Provide the following details:

  • Shared Access Signature: This is a URI that grants restricted access rights to Azure Storage resources.

  • Account Name: Provide the Azure account name.

  • Container: Provide the container name from where the file is located and which has to be read.

  • File Type: There are five (5) types of file extensions available:

    • CSV

    • JSON

    • PARQUET

    • AVRO

    • XML

  • Read Directory: This field will be checked by default. If this option is enabled, the component will read data from all the blobs present in the container.

  • Blob Name: This field will display only if the Read Directory field is disabled. Enter the specific name of the blob whose data has to be read.

  • Limit: Enter a number to limit the number of records that has to be read by the component.

  • Column Filter: Enter the column names here. Only the specified columns will be fetched from Azure Blob. In this field, the user needs to fill in the following information:

    • Source Field: Enter the name of the column from the blob. The user can add multiple columns by clicking on the "Add New Column" option.

    • Destination Field: Enter the alias name for the source field.

    • Column Type: Enter the data type of the column.

    • Upload: This option allows the user to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

    • Download Data: This option will download the data filled in the Column Filter field in JSON format.

    • Delete Data: This option will clear all the information filled in the Column Filter field.

  • Query: Enter a Spark SQL query in this field. Use inputDf as the table name.

//Sample Spark SQL Query:

Select * from inputDf where department='sales';
Reading Using Shared Access Signature

Read Using Secret Key Option

Provide the following details:

  • Account Key: Used to authorize access to data in your storage account via Shared Key authorization.

  • Account Name: Provide the Azure account name.

  • Container: Provide the container name from where the file is located and which has to be read.

  • File Type: There are five (5) types of file extensions available:

    • CSV

    • JSON

    • PARQUET

    • AVRO

    • XML

  • Read Directory: This field will be checked by default. If this option is enabled, the component will read data from all the blobs present in the container.

  • Blob Name: This field will display only if the Read Directory field is disabled. Enter the specific name of the blob whose data has to be read.

  • Limit: Enter a number to limit the number of records that has to be read by the component.

  • Column Filter: Enter the column names here. Only the specified columns will be fetched from Azure Blob. In this field, the user needs to fill in the following information:

    • Source Field: Enter the name of the column from the blob. The user can add multiple columns by clicking on the "Add New Column" option.

    • Destination Field: Enter the alias name for the source field.

    • Column Type: Enter the data type of the column.

    • Upload: This option allows the user to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

    • Download Data: This option will download the data filled in the Column Filter field in JSON format.

    • Delete Data: This option will clear all the information filled in the Column Filter field.

  • Query: Enter a Spark SQL query in this field. Use inputDf as the table name.

Reading Using Secret Key option

Read Using Principal Secret

Provide the following details:

  • Client ID: The unique Application (client) ID assigned to your app by Azure AD when the app was registered.

  • Tenant ID: A globally unique identifier (GUID) that is different from your organization name or domain.

  • Client Secret: The password of the service principal.

  • Account Name: Provide the Azure account name.

  • File Type: There are five (5) types of file extensions available:

    • CSV

    • JSON

    • PARQUET

    • AVRO

    • XML

  • Read Directory: This field will be checked by default. If this option is enabled, the component will read data from all the blobs present in the container.

  • Blob Name: This field will display only if the Read Directory field is disabled. Enter the specific name of the blob whose data has to be read.

  • Limit: Enter a number to limit the number of records that has to be read by the component.

  • Column Filter: Enter the column names here. Only the specified columns will be fetched from Azure Blob. In this field, the user needs to fill in the following information:

    • Source Field: Enter the name of the column from the blob. The user can add multiple columns by clicking on the "Add New Column" option.

    • Destination Field: Enter the alias name for the source field.

    • Column Type: Enter the data type of the column.

    • Upload: This option allows the user to upload a data file in CSV, JSON, or EXCEL format. The column names will be automatically fetched from the uploaded data file and filled out in the Name, Alias, and Column Type fields.

    • Download Data: This option will download the data filled in the Column Filter field in JSON format.

    • Delete Data: This option will clear all the information filled in the Column Filter field.

  • Query: Enter a Spark SQL query in this field. Use inputDf as the table name.

Reading Using Principal Secret option

Note: The following fields will be displayed after selecting the following file types:

  • CSV: The Header and Infer Schema fields get displayed with CSV as the selected File Type. Enable Header option to get the Header of the reading file and enable Infer Schema option to get true schema of the column in the CSV file.

  • JSON: The Multiline and Charset fields get displayed with JSON as the selected File Type.

    • Multiline: This option handles JSON files that contain records spanning multiple lines. Enabling this ensures the JSON parser reads multiline records correctly.

    • Charset: Specify the character set used in the JSON file. This defines the character encoding of the JSON file, such as UTF-8 or ISO-8859-1, ensuring correct interpretation of the file content.

  • PARQUET: No extra field gets displayed with PARQUET as the selected File Type.

  • AVRO: This File Type provides two drop-down menus.

    • Compression: Select an option out of the Deflate and Snappy options.

      • Deflate: A compression algorithm that balances between compression speed and compression ratio, often resulting in smaller file sizes.

      • Snappy: This compression type is select by default. A fast compression and decompression algorithm developed by Google, optimized for speed rather than maximum compression ratio.

    • Compression Level: This field appears if Deflate compression is selected. It provides a drop-down menu with levels ranging from 0 to 9, indicating the compression intensity.

Last updated