GCS Monitor

The GCS Monitor component continuously monitors a specified folder in a Google Cloud Storage (GCS) bucket. When a new file is detected:

1. The component reads the file name.

2. Triggers an event in the pipeline.

3. Copies the file to a designated location.

4. Removes the file from the monitored folder.

This process repeats automatically for every new file detected in the monitored folder.

Requirements

Before using the GCS Monitor, ensure that:

• You have access to the source GCS bucket and the appropriate Service Account Key file.

• A destination bucket/folder exists for storing copied files.

• The Service Account Key has the necessary permissions (storage.objects.get, storage.objects.delete, storage.objects.create).

Using the GCS Monitor in a Pipeline Workflow

1. Drag and drop the GCS Monitor component into the Workflow Editor.

2. Connect it to input/output events in the pipeline as required.

3. Configure the Basic Information and Meta Information tabs.

4. Save the component to activate monitoring.

Configuration

The GCS Monitor component settings are grouped into:

• Basic Information

• Meta Information

Basic Information Tab

The Basic Information tab defines general component behavior.

Field	Description	Required
Invocation Type	Select execution mode. Only Real-Time is supported.	Yes
Deployment Type	Pre-selected field showing the deployment type of the component.	Yes
Container Image Version	Displays the Docker image version used. Pre-selected.	Yes
Failover Event	Select a failover event to handle errors or retries.	Optional
Batch Size	Maximum records processed in one execution cycle. Minimum: 10.	Yes

Meta Information Tab

The Meta Information tab defines the GCS source and destination details.

Field	Description	Required
Bucket Name	The source GCS bucket name where monitoring is enabled.	Yes
Directory Path	Path of the monitored folder (e.g., monitor/). Use forward slashes (/).	Yes
Copy Directory Path	The destination folder path where files will be copied (e.g., monitor_copy/).	Yes
Choose File	Upload the Service Account Key JSON file for authentication.	Yes
File Name	Auto-populated with the name of the uploaded Service Account Key file.	Yes
Copy Bucket Name	Destination GCS bucket name where copied files are stored.	Yes

Saving the GCS Monitor

• Click the Save Component (Storage icon).

• A success message confirms that the component has been saved.

• Once saved, the GCS Monitor begins monitoring the specified folder continuously.

Example Workflow

1. A new CSV file is uploaded to gs://my-source-bucket/monitor/.

2. The GCS Monitor detects the file and triggers an event.

3. The file is copied to gs://my-destination-bucket/monitor_copy/.

4. The file is removed from the original monitor/ folder.

5. The pipeline processes the file from the destination folder.