Monitoring Pipeline

This Page explains How we can monitor the Pipelines.

The user can monitor a pipeline together with all the components associated with the same by using the Pipeline Monitoring icon. The user gets information about Pipeline components, Status, Types, Last Activated (Date and Time), Last Deactivated (Date and Time), Total Allocated and Consumed CPU%, Total allocated and consumed memory, Number of Records, and Component logs all displayed on the same page.

Go through the below-given video to get a basic idea on the pipeline monitoring functionality.

Displaying Pipeline Monitoring and Data Metrics

• Navigate to the Pipeline List page.

• Click the Monitor icon.

Accessing the Pipeline Monitoring from the Pipeline List page.

Or

• Navigate to the Pipeline Workflow Editor page.

• Click the Pipeline Monitoring icon on the Header panel.

• The Pipeline Monitoring page opens displaying the details of the selected pipeline.

• The Monitor tab opens by default.

• If there are multiple instances for a single component, click on the drop-down.

• Details for each instance will be displayed.

Monitoring page for Docker component in Real-Time

Docker component- Real time

Docker component- Real time

Monitoring page for Docker component in Batch:

Docker component- Batch

Monitoring page for Spark Component:

Monitoring details for Spark component

Monitoring page for Spark Component - Driver:

Spark component- Driver

Monitoring page for Spark Component- Executer:

Spark component- Executer

• If memory allocated to the component is less than required, then it will be displayed in red color.

• Open the Data Metrics tab.

• Specify the time period by providing the from and to dates.

• Choose an interval option or select the custom interval.

• The component specific data metrics get displayed. The green color nodes indicate that the data has been loaded.​ Click on green color icon to get all the details of processed data as shown in the below images.

Data metrics page

• Clear: It will clear all the monitoring and data metrics logs for all the components in the pipeline.

Clear option in monitoring page