# Pipeline Monitoring

The user can monitor a pipeline together with all the components associated with the same by using the ***Pipeline Monitoring*** icon. The user gets information about Pipeline components, Status, Types, Last Activated (Date and Time), Last Deactivated (Date and Time), Total Allocated and Consumed CPU%, Total allocated and consumed memory, Number of Records, and Component logs all displayed on the same page.

{% hint style="success" %}
*Go through the below-given video to get a basic idea on the pipeline monitoring functionality.*
{% endhint %}

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2FoSrdvW8384okEF0TXlMD%2Fmonitoring_updated.mp4?alt=media&token=45fda925-5f8b-425f-bc02-a8d735e77751>" %}
Displaying Pipeline Monitoring, Data Metrics and System Logs
{% endembed %}

* Navigate to the ***Pipeline List*** page.
* Click the ***Monitor*** icon.         &#x20;

<figure><img src="https://content.gitbook.com/content/vmyGSlDvDj86v7eh3RHM/blobs/IfRgDfklCEsnvBJXvrc4/MicrosoftTeams-image%20(114).png" alt=""><figcaption><p>Accessing the Pipeline Monitoring from the Pipeline List page.</p></figcaption></figure>

**Or**

* Navigate to the ***Pipeline Workflow Editor*** page.
* Click the ***Pipeline Monitoring*** icon on the Header panel.

<figure><img src="https://content.gitbook.com/content/vmyGSlDvDj86v7eh3RHM/blobs/CjYzf9vGEtgvrUwJjhin/MicrosoftTeams-image%20(115).png" alt=""><figcaption></figcaption></figure>

* The ***Pipeline Monitoring*** page opens displaying the details of the selected pipeline.
* The ***Pipeline Monitoring*** page displays the following information for the selected Job:
  * **Pipeline:** Name of the pipeline.
  * **Status:** Running status of the Job. '***True***' indicates the Job is active, while '***False***' indicates inactivity.
  * **Last Activated:** Date and time when the job was last activated.
  * **Last Deactivated:** Date and time when the pipeline was last deactivated.
  * **Total Allocated CPU:** Total allocated CPU in cores.
  * **Total Allocated Memory:** Total allocated memory in MB.
  * **Total Consumed CPU:** Total consumed CPU by the pipeline in cores.
  * **Total Consumed Memory:** Total consumed memory by the pipeline in MB.
  * **Component Name:** Name of the component which is used in pipeline.
  * **Running:** The running status of the component, displayed as '***UP***' if the component is running, otherwise '***OFF***'.
  * **Type:** Invocation type of component. It may be either ***Real Time*** or ***Batch***.
  * **Instances:** Number of instances used in the component.
  * **Last Processed Size:** Size of the batch (in MB) that was last processed.
  * **Last Processed Count:** Number of Processed records in last batch.
  * **Total Number of Records:** Total number of records processed by the component.
  * **Last Processed Time:** Last processed time of the instance.
  * **Host Name:** Name of the instance of the selected component.
  * **Min CPU Usage:** Minimum CPU usage in cores by the instance.
  * **Max CPU Usage:** Maximum CPU usage in cores by the instance.
  * **Min Memory Usage:** Minimum memory usage in MB by the instance.
  * **Max Memory Usage:** Maximum memory usage in MB by the instance.
  * **CPU Utilization:** Total CPU utilization in cores by the instance.
  * **Memory Utilization:** Total memory utilization in MB by the instance.
* **There will be three tabs in the monitoring page.**
  * **Monitor:** In this tab, it will display information such as the resources allocated, minimum/maximum resource consumption, instances provisioned, the number of records processed by each component, and their running status.
  * **Data Metrics:** Data Metrics will show the number of consumed records, processed records, failed records, and the corresponding failed percentage over a selected time window
  * **System Logs:** In this tab, the user can see the pod logs of every component in the pipeline.

**Once the user clicks on any instance, the page will expand to show the graphical representation of CPU usage, Memory usage and Records Per Process Execution over the given interval of time. For reference, please see the images given below:**

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark>* *The Records Per Process Execution metric showcases the number of records processed from the previous Kafka Event. If the component is not linked to the Kafka Event, the displayed value will be 0.*
{% endhint %}

## Monitor:

* The ***Monitor*** tab opens by default on the monitoring page.     &#x20;

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2F0h8xvIkoEtFfaQzQRRYo%2Fimage.png?alt=media&#x26;token=2af4e0cc-de61-4764-9b1d-ed2da398fd77" alt=""><figcaption><p><em><strong>Pipeline Monitoring Page</strong></em></p></figcaption></figure>

* If there are multiple instances for a single component, click on the drop-down icon.
* Details for each instance will be displayed.

**Monitoring page for Docker component in Real-Time**

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2FBlybJD5zI5eGBWwSsMEZ%2Fimage.png?alt=media&#x26;token=f31638b2-4fd5-4bc5-aa52-6a6427a15c4f" alt=""><figcaption><p><em><strong>Docker component- Real time</strong></em></p></figcaption></figure>

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2FEFK2oog4Dldtnqs9xTNN%2Fimage.png?alt=media&#x26;token=5aa3c2c1-28d9-4401-9957-f6a652ce74ba" alt=""><figcaption><p><em><strong>Docker component- Real time</strong></em></p></figcaption></figure>

**Monitoring page for Docker component in Batch:**

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2Fb2IbzjlMPtWT148n1N93%2Fimage.png?alt=media&#x26;token=2ff8d5d4-4f4a-4691-9626-b809ad4a4205" alt=""><figcaption><p><em><strong>Docker component- Batch</strong></em></p></figcaption></figure>

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2FGal3OuEx8EpCL8ggKd3s%2Fimage.png?alt=media&#x26;token=e84aaf3b-7efb-4d35-9dc6-f456d8cd0a70" alt=""><figcaption><p><em><strong>Docker component- Batch</strong></em></p></figcaption></figure>

**Monitoring page for Spark Component:**

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2FqcRmzzM8mpP9KkFyciz0%2Fimage.png?alt=media&#x26;token=6881c60f-271e-4e7f-93b4-665c570ad65c" alt=""><figcaption><p><em><strong>Monitoring details for Spark component</strong></em></p></figcaption></figure>

**Monitoring page for Spark Component - Driver:**

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2FUqnZqsJYjdFWuxpu4Qzx%2Fimage.png?alt=media&#x26;token=b711b931-5b4b-4d7b-9632-d876f4831c3d" alt=""><figcaption><p><em><strong>Spark component- Driver</strong></em></p></figcaption></figure>

**Monitoring page for Spark Component- Executer:**

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2F9vKH72mYEOYN3E2M6Mm5%2Fimage.png?alt=media&#x26;token=832492ad-c40d-460b-b896-3f77fb4c93c4" alt=""><figcaption><p><strong>Spark Component- Executer</strong></p></figcaption></figure>

* If memory allocated to the component is less than required, then it will be displayed in red color.

## Data Metrics:

* Open the ***Data Metrics*** tab from the pipeline monitoring page.
* Specify the time period by providing the ***from*** and ***to*** dates.
* Choose an interval option or select the custom interval by dragging the pointer.
* The component specific data metrics get displayed. The green color nodes indicate that the data has been loaded.​ Click on the **green** color icon to get all the details of processed data as shown in the below given images.
* Once hovering on the green color icon on the Data Metrics page, it will display the following information:

  * **Start:** Window start time.
  * **End:** Window end time.
  * **Processed:** Number of processed records.
  * **Produced:** Number of records generated after processing.
  * **Failed:** Number of records that failed during processing.
  * **Failed Percentage:** Percentage of failed records, calculated as the ratio between Failed and Processed data.

  By default, the Start and End window times are set within the range of 30 minutes. Users can adjust this window time to 60 minutes and 90 minutes as per their convenience. If there is no failed data within the specified window, the ***Failed*** and ***Failed Percentage*** fields will not be shown when hovering over the green color icon.

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2Frt2FGimXyeDheQCmYITy%2Fimage.png?alt=media&#x26;token=4d32e147-3b02-4169-9c68-3bc3e07ec997" alt=""><figcaption><p><em><strong>Data Metrics Page</strong></em></p></figcaption></figure>

* The user can see the data metrics for all the components by enabling the ***Show all components*** on the Data Metrics page. Please refer the below given image for the reference.

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2Fbi95kEKlibvz1FnVSFLr%2Fimage.png?alt=media&#x26;token=4a9ec623-74a9-44e8-a73f-3f077ffd6689" alt=""><figcaption><p><em><strong>Showing Data Metrics for all the component in the pipeline</strong></em></p></figcaption></figure>

* **Clear:** It will clear all the monitoring and data metrics logs for all the components in the pipeline.&#x20;

{% hint style="info" %}
*<mark style="color:green;">Please Note</mark>: The **Clear** option does not display on the monitoring page if the pipeline is active.*
{% endhint %}

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2F3mseG1UZTK3XNgTfNcn7%2Fimage.png?alt=media&#x26;token=a2dc57df-292b-42a9-b3aa-3f346656fb6c" alt=""><figcaption><p><em><strong>Clear option on Monitoring Page</strong></em></p></figcaption></figure>

* Users can visualize the loaded data in the form of charts by clicking on the green color icon.&#x20;

{% hint style="success" %}
*Please go through the given walk through for the reference.*
{% endhint %}

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fc39ZvXI46qjXzpN3rYAg%2Fuploads%2F78iDzcRJOlVc7oWZcFSK%2F2024-01-16-18-49-44%20(online-video-cutter.com).mp4?alt=media&token=c7d72a2b-56ae-4e65-9826-da072bcf5b64>" %}
***Data Metrics Page***
{% endembed %}

* Once the user clicks on the green color icon, the following page will be opened:

<figure><img src="https://content.gitbook.com/content/vmyGSlDvDj86v7eh3RHM/blobs/ONY39KgvEwMDxdcW7fEj/image.png" alt=""><figcaption></figcaption></figure>

1. **Produced v/s Consumed**

<figure><img src="https://content.gitbook.com/content/vmyGSlDvDj86v7eh3RHM/blobs/9Rdxa0WmaKS96sqYag8W/image.png" alt=""><figcaption><p>Produced v/s Consumed</p></figcaption></figure>

This chart will display the number of records produced to the out event compared to the number of records taken from the previous event over the given time window.

2. **Min v/s Max v/s Avg Elapsed Time**

<figure><img src="https://content.gitbook.com/content/vmyGSlDvDj86v7eh3RHM/blobs/iArGD2qJoqgGdRpJ0Xhy/image.png" alt=""><figcaption><p>Min v/s Max v/s Avg Elapsed Time</p></figcaption></figure>

This chart displays the minimum, maximum, and average time taken (***in milliseconds***) to process a record over the given time window.

* **Min Elapsed Time:** The minimum time taken (***in milliseconds***) to process a record and send it to the out event.
* **Max Elapsed Time:** The maximum time taken (***in milliseconds***) to process a record and send it to the out event.
* **Average Elapsed Time:** The average time taken (***in milliseconds***) to process a record and send it to the out event.

3. **Failed Records**

<figure><img src="https://content.gitbook.com/content/vmyGSlDvDj86v7eh3RHM/blobs/x0kFjxAv68OWw8nApMo5/image.png" alt=""><figcaption><p>Failed Records</p></figcaption></figure>

This chart will display the number of failed records during processing over the given time window for the selected component.

4. **Consumed v/s Failed Records**

<figure><img src="https://content.gitbook.com/content/vmyGSlDvDj86v7eh3RHM/blobs/wYDDOBYkDM1QtH5Ns2og/image.png" alt=""><figcaption><p>Consumed v/s Failed Records</p></figcaption></figure>

This chart will display the ratio of records failed during processing by the component to the total number of records consumed by the component over the given time window.

The user can also analyze the failure for the selected component from the ***Data Metrics*** page by clicking on the ***Analyze Failure*** option. Please see the below given image for reference.

<figure><img src="https://content.gitbook.com/content/vmyGSlDvDj86v7eh3RHM/blobs/bhEX1qe4UndHHpDmAXUx/image.png" alt=""><figcaption><p><em><strong>The Analyze Failure window panel</strong></em></p></figcaption></figure>

Clicking on the ***Analyze Failure*** option will redirect the user to the [***Failure Analysis***](https://docs.bdb.ai/data-pipeline-4/getting-started/pipeline-workflow-editor/pipeline-toolbar/failure-analysis) page.

## System Logs:

In this tab, the user can see the pod logs of every component in the pipeline. The user can access this tab from the monitoring page.

<figure><img src="https://3157314867-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvmyGSlDvDj86v7eh3RHM%2Fuploads%2FkxhOxb5BT4PFssHUKElk%2Fimage.png?alt=media&#x26;token=43d46e54-f280-4f3c-85f6-72789b8f009d" alt=""><figcaption><p><em><strong>System Logs</strong></em></p></figcaption></figure>

On the System Logs tab, the user can find the following options:

* **Selected Pod:** The user can select the Pod for which they want to see the logs.
* **Date Filter:** The user can apply the date filter to see the logs accordingly.
* **Refresh Logs:** The user can refresh the logs for the selected pod.
* **Download Logs:** The user can download the logs for the selected pod.

{% hint style="info" %}
*<mark style="color:green;">**Please Note:**</mark>* ***The system logs on the monitoring page will be displayed only when the pipeline is active.***
{% endhint %}