Monitoring Pipeline
This Page explains How we can monitor the Pipelines.
The user can monitor a pipeline together with all the components associated with the same by using the Pipeline Monitoring icon. The user gets information about Pipeline components, Status, Types, Last Activated (Date and Time), Last Deactivated (Date and Time), Total Allocated and Consumed CPU%, Total allocated and consumed memory, Number of Records, and Component logs all displayed on the same page.
Go through the below-given video to get a basic idea on the pipeline monitoring functionality.
Navigate to the Pipeline List page.
Click the Monitor icon.
Or
Navigate to the Pipeline Workflow Editor page.
Click the Pipeline Monitoring icon on the Header panel.
The Pipeline Monitoring page opens displaying the details of the selected pipeline.
The Monitor tab opens by default.
If there are multiple instances for a single component, click on the drop-down.
Details for each instance will be displayed.
Monitoring page for Docker component in Real-Time
Monitoring page for Docker component in Batch:
Monitoring page for Spark Component:
Monitoring page for Spark Component - Driver:
Monitoring page for Spark Component- Executer:
If memory allocated to the component is less than required, then it will be displayed in red color.
Open the Data Metrics tab.
Specify the time period by providing the from and to dates.
Choose an interval option or select the custom interval.
The component specific data metrics get displayed. The green color nodes indicate that the data has been loaded. Click on green color icon to get all the details of processed data as shown in the below images.
Clear: It will clear all the monitoring and data metrics logs for all the components in the pipeline.
Last updated
