Sandbox Reader

A Sandbox reader is used to read and access data within a configured sandbox environment.

All component configurations are classified broadly into the following sections:

• ​​Basic Information​​

• Meta Information

• ​Resource Configuration​​

• ​Connection Validation

Before using Sandbox Reader component for reading a file, the user needs to upload a file in Data Sandbox under Data Center module.

Please go through the given walk through for uploading the file in Data Sandbox under Data Center module.

Uploading a file in Data Sandbox

Check out the given video on how to configure a Sandbox Reader component.

Sandbox Reader as a part of Pipeline workflow

Steps to configure Sandbox Reader

• Navigate to the Data Pipeline Editor.

• Expand the Reader section provided under the Component Pallet.

• Drag and drop the Sandbox Reader component to the Workflow Editor.

• Click on the dragged Sandbox Reader to get the component properties tabs.

Basic Information

It is the default tab to open for the component while configuring it.

• Invocation Type: Select an invocation mode out of Real-Time or Batch using the drop-down menu.

• Deployment Type: It displays the deployment type for the reader component. This field comes pre-selected.

• Container Image Version: It displays the image version for the docker container. This field comes pre-selected.

• Failover Event: Select a failover Event from the drop-down menu.

• Batch Size (min 10): Provide the maximum number of records to be processed in one execution cycle (Min limit for this field is 10).

Meta Information

Open the Meta Information tab and fill in all the connection-specific details for the Sandbox Reader.

• Storage Type (*): Network is displayed by default

• Sandbox File: Select the file which is to be read.

• File Type (*): Select a file type from the drop-down menu (CSV, JSON, PARQUET, AVRO and XML are the supported file types)

• Limit: Set a limit for the number of records.

• Query: Insert an SQL query (it supports query containing a join statement as well)

• Column Filter: There is also a section for the selected columns in the Meta Information tab if the user can select some specific columns from the table to read data instead of selecting a complete table so this can be achieved by using the Column Filter section. Select the columns which you want to read and if you want to change the name of the column, then put that name in the alias name section otherwise keep the alias name the same as of column name and then select a Column Type from the drop-down menu.

• Use Download Data and Upload File options to select the desired columns.

  • Upload File: The user can upload the existing system files (CSV, JSON) using the Upload File icon (file size must be less than 2 MB).

  • Download Data (Schema): Users can download the schema structure in JSON format by using the Download Data icon.

or

Use the Download Data and Upload File options to select the desired columns.

• Partition Columns: Provide a unique Key column name to partition data in Spark.

• Click the Save Component in Storage icon after doing all the configurations to save the reader component.

• A notification message appears to inform about the component configuration success.

Please Note:

• (*) the symbol indicates that the field is mandatory.

• Either table or query must be specified for the data readers except for SFTP Reader.

• Column Filter- There should not be a data type mismatch in the Column Type for all the Reader components.

• The Meta Information fields may vary based on the selected File Type. All the possibilities are mentioned below:

• CSV: The Header and Infer Schema fields get displayed with CSV as the selected File Type. Enable Header option to get the Header of the reading file and enable Infer Schema option to get true schema of the column in the CSV file.

• JSON: The Multiline and Charset fields get displayed with JSON as the selected File Type. Check-in the Multiline option if there is any multiline string in the file.

• PARQUET: No extra field gets displayed with PARQUET as the selected File Type.

• AVRO: This File Type provides two drop-down menus.

  • Compression: Select an option out of the Deflate and Snappy options.

  • Compression Level: This field appears for the Deflate compression option. It provides 0 to 9 levels via a drop-down menu.

• XML: Select this option to read XML file. If this option is selected, the following fields will get displayed:

  • Infer schema: Enable this option to get true schema of the column.

  • Path: Provide the path of the file.

  • Root Tag: Provide the root tag from the XML files.

  • Row Tags: Provide the row tags from the XML files.

  • Join Row Tags: Enable this option to join multiple row tags.

Saving the Component

• Click the Save Component in Storage icon after doing all the configurations to save the reader component.​

• A notification message appears to inform about the component configuration success.