# S3 Reader **S3 Reader** component typically authenticate with S3 using AWS credentials, such as an access key ID and secret access key, to gain access to the S3 bucket and its contents. S3 Reader is designed to read and access data stored in an S3 bucket in AWS. All component configurations are classified broadly into the following sections: * [​Basic Information​](https://docs.bdb.ai/data-pipeline-5/components/component-base-configuration) * Meta Information * ​[Resource Configuration​](https://docs.bdb.ai/data-pipeline-5/components/resource-configuration) * ​[Connection Validation](https://docs.bdb.ai/7.6/data-pipeline/components/connection-validation) {% hint style="success" %} *Check out the below-given demonstration to configure the S3 component and use it in a pipeline workflow.* {% endhint %} {% embed url="" %} Configuring S3 Reader to read data from S3 Bucket {% endembed %} * Navigate to the ***Data Pipeline Editor***. * Expand the Reader section provided under the Component Pallet. * Drag and drop the S3 Reader component to the Workflow Editor. * Click on the dragged S3 Reader to get the component properties tabs. ## **Basic Information** It is the default tab to open for the component while configuring it. * **Invocation Type:** Select an invocation mode out of ‘Real-Time’ or ‘Batch’ using the drop-down menu. * **Deployment Type:** It displays the deployment type for the reader component. This field comes pre-selected. * **Container Image Version:** It displays the image version for the docker container. This field comes pre-selected. * **Failover Event:** Select a failover Event from the drop-down menu. * **Batch Size (min 10):** Provide the maximum number of records to be processed in one execution cycle (Min limit for this field is 10). ## Meta Information Open the ‘**Meta Information**’ tab and fill in all the connection-specific details for the S3 Reader.· ### Configuration fields when **SNS Monitor is disabled** * **Bucket Name (\*):** Enter AWS S3 Bucket Name. * **Zone (\*):** Enter S3 Zone. (For eg: us-west-2) * **Access Key (\*):** Provide Access Key ID shared by AWS. * **Secret Key (\*):** Provide Secret Access Key shared by AWS. * **Table (\*):** Mention the Table or file name from S3 location which is to be read. * **File Type (\*):** Select a file type from the drop-down menu (**CSV, JSON, PARQUET, AVRO, XML** and **ORC** are the supported file types) * **Limit:** Set a limit for the number of records to be read. * **Query:** Enter a Spark SQL query. Take ***inputDf*** as table name. ### **Configuration fields when SNS Monitor is enabled** * **Access Key (\*):** Provide Access Key ID shared by AWS. * **Secret Key (\*):** Provide Secret Access Key shared by AWS. * **Table (\*):** Mention the Table or file name from S3 location which is to be read. * **File Type (\*):** Select a file type from the drop-down menu (**CSV, JSON, PARQUET, AVRO** are the supported file types). * **Limit:** Set a limit for the number of records to be read. * **Query:** Enter a Spark SQL query. Take ***inputDf*** as table name. **Sample Spark SQL query for S3 Reader:** ```sql select * from inputDf where Gender = 'Male'; ``` ### Selected Columns There is also a section for the selected columns in the Meta Information tab if the user can select some specific columns from the table to read data instead of selecting a complete table so this can be achieved by using the ‘Selected Columns’ section. Select the columns which you want to read and if you want to change the name of the column, then put that name in the alias name section otherwise keep the alias name the same as of column name and then select a Column Type from the drop-down menu. or Use ‘**Download Data**’ and ‘**Upload File**’ options to select the desired columns. * [x] **Upload File**: The user can upload the existing system files (CSV, JSON) using the ‘**Upload File**’ icon (file size must be less than 2 MB). * [x] **Download Data (Schema)**: Users can download the schema structure in JSON format by using the ‘**Download Data**’ icon. ### **Partition Columns** Provide a unique Key column name on which the partition has been done and has to be read. ## Saving the Component Configuration * Click the ***Save Component in Storage*** icon after doing all the configurations to save the reader component.​ * A notification message appears to inform about the component configuration success. {% hint style="info" %} *Please Note:* * ***(\*)** the symbol indicates that the field is mandatory.* * *Either table or query must be specified for the data readers except for SFTP Reader.* * ***Selected Columns-** There should not be a data type mismatch in the Column Type for all the Reader components.* * *The Meta Information fields may vary based on the selected **File Type**.* * All the possibilities are mentioned below: * **CSV:** ‘**Header**’ and ‘**Infer Schema**’ fields get displayed with CSV as the selected File Type. Enable ***Header*** option to get the Header of the reading file and enable ***Infer Schema*** option to get true schema of the column in the CSV file. * **JSON:** ‘**Multiline**’ and ‘**Charset**’ fields get displayed with JSON as the selected File Type. Check-in the **Multiline** option if there is any multiline string in the file. * **PARQUET:** No extra field gets displayed with PARQUET as the selected File Type. * **AVRO:** This File Type provides two drop-down menus. * Compression: Select an option out of the ‘**Deflate**’ and ‘**Snappy**’ options. * Compression Level: This field appears for the Deflate compression option. It provides 0 to 9 levels via a drop-down menu.​ * **XML:** Select this option to read XML file. If this option is selected, the following fields will get displayed: * **Infer schema:** Enable this option to get true schema of the column. * **Path:** Provide the path of the file. * **Root Tag:** Provide the root tag from the XML files. * **Row Tags:** Provide the row tags from the XML files. * **Join Row Tags:** Enable this option to join multiple row tags. * **ORC:** Select this option to read ORC file. If this option is selected, the following fields will get displayed: * **Push Down:** In ORC (Optimized Row Columnar) file format, *"push down"* typically refers to the ability to push down predicate filters to the storage layer for processing. There will be two options in it: * **True:** When push down is set to True, it indicates that predicate filters can be pushed down to the ORC storage layer for filtering rows at the storage level. This can improve query performance by reducing the amount of data that needs to be read into memory for processing. * **False:** When push down is set to False, predicate filters are not pushed down to the ORC storage layer. Instead, filtering is performed after the data has been read into memory by the processing engine. This may result in more data being read and potentially slower query performance compared to when push down is enabled. {% endhint %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.bdb.ai/data-pipeline-5/components/readers/s3-reader.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.