Big Query Reader

The Big Query Reader Component is designed for efficient data access and retrieval from Google Big Query, a robust data warehousing solution on Google Cloud. It enables applications to execute complex SQL queries and process large datasets seamlessly. This component simplifies data retrieval and processing, making it ideal for data analysis, reporting, and ETL workflows.

All component configurations are classified broadly into the following sections:

• Basic Information

• Meta Information

Steps to Configure the Big Query Reader Component

• Navigate to the Data Pipeline Editor.

• Expand the Reader section provided under the Component Pallet.

• Drag and drop the Big Query Reader component to the Workflow Editor.

• Click on the dragged Big Query Reader to get the component properties tabs.

Basic Information

It is the default tab to open for the component while configuring it.

• Invocation Type: Select an invocation mode from the ‘Real-Time’ or ‘Batch’ using the drop-down menu.

• Deployment Type: It displays the deployment type for the reader component. This field comes pre-selected.

• Batch Size (min 1): Provide the maximum number of records to be processed in one execution cycle (Min limit for this field is 1).

• Failover Event: Select a failover Event from the drop-down menu.

• Container Image Version: It displays the image version for the docker container. This field comes pre-selected.

Meta Information

• Open the Meta Information tab and fill in all the connection-specific details for the Big Query Reader.

• Read using: The 'Service Account' option is available under this field, so select it.

• Upload JSON(*): Upload credential file downloaded from Google Big Query using the Upload icon. You may need to download a JSON from Big Query to upload it here.

Follow these steps to download a JSON from Big Query:

• Open the Big Query Console.

• Click the API & Services inside the Navigation pane.

• Click on Credentials.

• Create Credentials (Service Account).

• After creating the credentials click on the Key.

• Go to Keys in the header, click the Create New option, and download JSON.

• Dataset Id: Mention the Dataset ID from Big Query which is to be read.

• Table Id: Mention the Table ID from Big Query which is to be read.

• Location (*): Mention the location according to your Project.

• Limit: Set a limit for the number of records to be read.

• Query: Enter an SQL Query.

Sample Spark SQL query for Big Query Reader:

Select * from project_id.dataset_id.table_id limit 10

Saving the Component Configuration

• A notification message appears to inform about the component configuration success.