# List Pipelines

The ***List Pipelines*** option opens the available Pipeline List for the logged-in user. All the saved pipelines by a user get listed on this page. The ***Details*** tab on the right side of the page displays the basic details of the selected pipeline.

{% hint style="info" %}
*<mark style="color:green;">Please Note</mark>*<mark style="color:green;">:</mark> The ***Push to VCS*** and ***Pull Pipeline from GIT*** features are present on the ***List Pipeline*** page and as well as on the ***Pipeline Editor*** page.
{% endhint %}

## Pull & Push Pipeline <a href="#pushing-a-pipeline-into-vcs" id="pushing-a-pipeline-into-vcs"></a>

### Pushing a Pipeline into VCS <a href="#pushing-a-pipeline-into-vcs" id="pushing-a-pipeline-into-vcs"></a>

* Navigate to the ***Pipeline List*** page.
* Select a data pipeline from the displayed list.
* Click the ***Push & Pull Pipeline*** icon for the selected data pipeline.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/IGBVvZmQsagZLEE4OD94/image.png" alt=""><figcaption><p><em><strong>Push/ Pull icon on the Pipeline List page</strong></em></p></figcaption></figure>

* The ***Push/ Pull*** drawer appears. It displays the name of the selected Pipeline. E.g., In the following image the ***Push/ Pull sandboxwriter*** heading, the ***sandboxwriter*** is the name of the selected Pipeline.
* Provide a ***Commit Message*** (required) for the data pipeline version.
* Select a ***Push Type*** out of the below-given choices to push the pipeline:
  1. ***Version Control:*** For versioning of the pipeline in the same environment. In this case, the selected ***Push Type*** is ***Version Control***.
  2. ***GIT Export*** (Migration): This is for pipeline migration. The pushed pipeline can be migrated to the destination environment from the migration window in the  Admin Module.
* Click the **Save** option.​

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/Puxx3YForRAxDEnseCJu/image.png" alt=""><figcaption></figcaption></figure>

* A notification message appears to confirm the completion of the action.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/6WXECHCF4BSY5v9ZkSv2/image.png" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*Check out the below-given illustrations on how to attempt **Version Control** and **Pipeline Migration**.*
{% endhint %}

***Version Control:***

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FARWwq0mHnUobVAJqtFtw%2Fuploads%2FdZUfRuh5iHJQAJy6jX4w%2FPush%20to%20VCS_DP.mp4?alt=media&token=4d879a16-b078-4987-8fc3-a2bcf299aaff>" %}
***Push For Pipeline Versioning***
{% endembed %}

***Pipeline Migration:***

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FARWwq0mHnUobVAJqtFtw%2Fuploads%2FxZnQ0ubbDOc8YB0orVn9%2FPush%20to%20GIT_DP%20Migration.mp4?alt=media&token=c91ce54c-e86b-4feb-857c-b90f860ad634>" %}
***Push For Pipeline Migration***
{% endembed %}

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark>*

* *The* *pipeline pushed to the VCS using the Version Control option, can be pulled directly from the **Pull Pipeline** option.*
* *The user also gets an option to **Push the pipeline to GIT**. **This action will be considered as Pipeline Migration.** The user needs to follow the* [***Pipeline Migration*** ](https://app.gitbook.com/s/MEJKqXNSvcfKLo5znn53/admin-panel-options/git-migration/migrating-a-pipeline)*steps given the* [***Admin***](https://app.gitbook.com/o/BHXEmSpD7W3xDgxnOIpk/s/SuiyPYGtTJrg3jydU0rO/) ***module** of the Platform to **pull a version that has been pushed to GIT.***
  {% endhint %}

### Pulling a Pipeline <a href="#pulling-a-pipeline-from-git" id="pulling-a-pipeline-from-git"></a>

This feature is for pulling the previously moved versions of a pipeline that are committed by the user. This can help a user significantly to recover the lost pipelines or avoid unwanted modifications made to the pipeline.

{% hint style="info" %}
*Check out the walk-through on how to pull a pipeline version from the GIT.*
{% endhint %}

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FARWwq0mHnUobVAJqtFtw%2Fuploads%2FQzb9oVNYhWYagw9midPz%2FPulling%20from%20VCS-DP.mp4?alt=media&token=0f4341f2-e82d-4076-b295-87dc775bbf48>" %}
***Pulling a pipeline version committed earlier***
{% endembed %}

* Navigate to the ***Pipeline List*** page.
* Select a data pipeline from the displayed list.
* Click the ***Pull from GIT*** icon for the selected data pipeline.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/CQhZs4Oy9E4yn6NzZFMN/image.png" alt=""><figcaption><p><strong>Push/ </strong><em><strong>Pull icon on the Pipeline List page</strong></em></p></figcaption></figure>

* The ***Push/ Pull*** drawer opens with the selected Pipeline name.
* Select the data pipeline version by marking the given checkbox.
* Click the ***Ok*** option.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/1aPrMS1u1yIvxLfFTd3d/image.png" alt=""><figcaption><p><em><strong>Pull tab for the selected Pipeline workflow</strong></em></p></figcaption></figure>

* A confirmation message appears to assure the user that the concerned pipeline workflow has been pulled.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/V39jt9LbfnbngYH8S4tC/image.png" alt=""><figcaption></figcaption></figure>

* Another notification appears to inform that the selected pipeline workflow is imported.&#x20;

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/DSkMGj5Bmb6kjjZ3nans/image.png" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark> The pipeline that you pull will be changed to the selected version. Please make sure to manage the versions of the pipeline properly.*
{% endhint %}

## View Pipeline <a href="#view-pipeline" id="view-pipeline"></a>

Clicking on the ***View*** icon will direct the user to the pipeline workflow editor page.

* Navigate to the ***Pipeline List*** page.
* Select a Pipeline from the list.
* Click the ***View*** icon.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/NbOfAJOTScAcgHa2EbkN/image.png" alt=""><figcaption></figcaption></figure>

* The ***Pipeline Editor*** page opens for that pipeline.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/Spb0j9bCmeZFNvhhZmLR/image.png" alt=""><figcaption><p><em><strong>Pipeline Editor page for the selected Pipeline</strong></em></p></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note</mark><mark style="color:green;">:</mark>* *The user can open the Pipeline Editor for the selected pipeline from the list by clicking the **View** icon or the **Pipeline Workflow Editor** icon on the Pipeline List page.*
{% endhint %}

### Searching a Pipeline

*The user can search for a specific pipeline by using the **Search bar** on the Pipeline List. By typing a common name all the existing pipelines having that word will list. E.g., By typing the word **sandbox** all the existing pipelines with that word get listed in the following image:*

​​

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/SijnOapGfhPSFlLYg4RC/image.png" alt=""><figcaption><p><em><strong>Search option for Pipelines</strong></em></p></figcaption></figure>

### **View Pipeline details from the list Pipeline page**

Click on the pipeline to view the information of the pipeline on the list pipeline page. Once clicked on the pipeline name, a menu will open in the right side of the screen showing all the details of the selected pipeline. Please go through the below demonstration:

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FARWwq0mHnUobVAJqtFtw%2Fuploads%2FnDDkl3FqRboLcw7IVkYy%2FPipeline%20Details.mp4?alt=media&token=87eb0f3b-f9b5-4c81-94b2-23a93f16f1c7>" %}
***Pipeline Details***
{% endembed %}

* By clicking on a pipeline name, the following details of that pipeline will be appear below in two tabs:
  * **Pipeline Details**
    * **Components:** Number of components used in the pipeline.
    * **Created By:** Owner name of the pipeline.
    * **Created Date:** Pipeline created date.
    * **Updated By:** Name of the person who has updated the pipeline recently.
    * **Updated Date:** Date when the pipeline is updated.
    * **Description**: If description is inserted while creating the pipeline.
  * **Failed and Ignored components**
    * This section displays the component which failed in the pipeline workflow.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/GZSdNAjN2GtF0IirwKdV/image.png" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark> The Failed and Ignored components will be empty if the pipeline flow is successful. It will only display the failed component for the Pipeline workflows that also failed.*

![](https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/HVDZcVXK1qNBogXEN0hF/image.png)
{% endhint %}

## Pipeline Component Configuration <a href="#monitor-pipeline" id="monitor-pipeline"></a>

{% hint style="success" %}
*Check out the given illustration on the Pipeline Component Configuration page.*
{% endhint %}

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FARWwq0mHnUobVAJqtFtw%2Fuploads%2FO2N8OzpvrR7UUfiYvLzW%2FPipeline%20Component%20Configuration.mp4?alt=media&token=5b6e6203-2a08-4055-b6e5-d3b8f4bd4404>" %}
***Pipeline Component Configuration page***
{% endembed %}

* Navigate to the ***Pipeline List*** page.
* Select a Pipeline from the list.
* Click the ***Pipeline Component Configuration*** icon.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/rFVEQcXrCOWg2vAclRAG/image.png" alt=""><figcaption><p><em><strong>Accessing the Pipeline Component Configuration icon on the Pipeline List</strong></em></p></figcaption></figure>

* All the components used in the selected pipeline will be listed on the configuration page.&#x20;
* There are two tabs on the ***Configuration Page: Basic** (default),* and ***Configuration***.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/2lyAlqKea5rfRs4G2IMf/image.png" alt=""><figcaption><p><em><strong>The Pipeline Component Configuration page</strong></em> </p></figcaption></figure>

* In the ***Basic*** tab, the user can configure **basic information** for the components such as **invocation type, batch size, description**, or **intelligent scaling**.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/fz91FGCY2bBDfjEO2z6L/image.png" alt=""><figcaption></figcaption></figure>

* On the **Configuration** tab, the user can provide resources such as **memory** and **CPU** to the components, as well as set the number of instances for the components.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/gqxkOh9s9n824rzQNloG/image.png" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark>*&#x20;

* *The user can edit the required basic or configuration details for all the pipeline components.*&#x20;
* *The Pipeline Component Configuration page also displays icons to access **Pipeline Testing**, **Failure Analysis**, **Open Pipeline (Editor Workspace)**, and **List Pipeline**.*
  {% endhint %}

## Monitor Pipeline <a href="#monitor-pipeline" id="monitor-pipeline"></a>

By clicking the ***Monitor Pipeline*** icon,  the user will get redirected to the ***Pipeline Monitoring page***.

* Navigate to the ***Pipeline List*** page.
* Select a Pipeline from the list.
* Click the ***Monitor*** icon.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/prK0kF6PNbvLqHm8iMY2/image.png" alt=""><figcaption><p><em><strong>Monitor icon provided on the Pipeline List page</strong></em></p></figcaption></figure>

* The user gets redirected to the Pipeline Monitoring page of the selected Pipeline.
* The ***Monitor*** tab opens by default.
* The user can click the ***Data Metrics*** tab if wants to see depiction of the component specific data metrics.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/FwhLLhYZ9Gvkz96dYcJm/image.png" alt=""><figcaption><p><em><strong>Monitor tab for a Data Pipeline</strong></em></p></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note</mark>: Check out the* [***Monitoring Pipeline***](https://docs.bdb.ai/data-pipeline-2/getting-started/monitoring-pipeline) *page for more details on this topic.*
{% endhint %}

## Failure Analysis <a href="#test-pipeline" id="test-pipeline"></a>

Failure analysis is a central failure mechanism. Here, the user can identify the failure reason. Failures of any pipeline stored at a particular location(collection). From there you can query your failed data in the Failure Analysis UI. It displays the failed records along with cause, event time, and pipeline Id.

* Navigate to the ***Pipeline List*** page.
* Select a Pipeline with Failed status from the list.&#x20;
* Click on the ***Failure Analysis*** icon.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/iLthdWpJVEMBudfrqi4D/image.png" alt=""><figcaption><p><strong>Failure Analysis icon in the Pipeline list</strong></p></figcaption></figure>

* The user gets redirected to the ***Failure Analysis*** page.
* The user can further get failure analysis of the specific component that failed in the pipeline workflow.
* Use the ellipsis icon to get redirected to the ***Monitoring*** and ***Data Metrics*** options for a component.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/2P7eZjBqbYfmgCifoZcU/image.png" alt=""><figcaption><p><em><strong>The Failure Analysis page for the selected Pipeline flow</strong></em></p></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark> Refer to the* [***Failure Analysis***](https://docs.bdb.ai/data-pipeline-2/getting-started/pipeline-workflow-editor/pipeline-toolbar/failure-analysis) *for more details on this.*
{% endhint %}

## Sharing a Pipeline <a href="#sharing-a-pipeline" id="sharing-a-pipeline"></a>

The user can share a pipeline with one or multiple users and user Groups using the ***Share Pipeline*** option.

{% hint style="success" %}
*Check out the following walk-through on how to share a pipeline with user/ user group and exclude the user from a shared pipeline.*
{% endhint %}

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FARWwq0mHnUobVAJqtFtw%2Fuploads%2FSC6ZWeSje5OYqzlefOPg%2FSharing%20or%20Excluding%20the%20Pipeline%20rights.mp4?alt=media&token=90ee44c0-c0bd-4d08-9f6a-76fd5e2983bf>" %}
***Sharing a Pipeline and Excluding a shared Pipeline***
{% endembed %}

* Click the ***List Pipelines*** icon to open the Pipeline List.
* Select a Pipeline from the pipeline list.
* Click on the ***Share Pipeline*** icon.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/w1FG1IbVt8nIOr2GJWZe/image.png" alt=""><figcaption><p><em><strong>Accessing the Share icon from the List Pipeline page</strong></em></p></figcaption></figure>

* The ***Share Pipeline*** window opens.
* Select an option out of the given choices: ***User*** (default tab) and ***User Group*** or ***Exclude User*** (the **Exclude User** option can be chosen, if the pipeline is already shared with a user/user group and you wish to exclude them from the privilege).
* Select a user or user(s) from the displayed list of users (In case of the User Group(s) tab, it displays the names of the User Groups.
* Click the arrow to move the selected User(s)/ User Group(s).

​

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/5i1k8FYObUDu3iMpf3wQ/image.png" alt=""><figcaption></figcaption></figure>

* The selected user(s)/user group(s) get moved to the box given on the right (In case of the ***Exclude User*** option the selected user/ the user moved to the right-side box looses the access of the shared pipeline).
* Click the ***Save*** option.​

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/kINqtJVKWd9y9KcvE0Zu/image.png" alt=""><figcaption></figcaption></figure>

* Privilege for the selected pipeline gets updated and the same gets communicated through a message.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/kMqrp10fN0ukiiD9z2sK/image.png" alt=""><figcaption><p><strong>Confirmation message after a Pipeline gets shared</strong></p></figcaption></figure>

* By completing the steps mentioned above, the concerned pipeline gets successfully shared with the selected user/user group or the selected users can also be excluded from their privileges for the concerned pipeline.

{% hint style="info" %}
*<mark style="color:green;">Please Note</mark>:*

* *An admin user can **View**, **Edit/Modify, Share, and*** ***Delete** a shared pipeline.*
* *A non-admin user can **View** and **Edit/ Modify** a shared Pipeline, but it can't **Share** it further or **Delete** it.*
  {% endhint %}

## Activate/Deactivate Pipeline <a href="#activate-deactivate-pipeline" id="activate-deactivate-pipeline"></a>

The user can activate/deactivate the pipeline by clicking on the Activate/ Deactivate icon.

* Click the ***List Pipelines*** icon to open the Pipeline list.
* Select a Pipeline from the pipeline list.
* Click on the ***Activate Pipeline*** icon.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/FOOVT38WzQiLvFMziEIw/image.png" alt=""><figcaption><p><em><strong>Accessing the Activate option on the Pipeline List</strong></em></p></figcaption></figure>

* The ***Confirmation*** dialog box opens to get the confirmation for the action.
* Click the ***YES*** option.

&#x20;     ![](https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/WPfIsLNirrFtAKTG904o/image.png)​

* A confirmation message appears.
* The pipeline gets activated. The ***Activate Pipeline*** icon turns into ***Deactivate Pipeline***.
* The same gets communicated through the ***Status*** icon color.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/NE8qg6UgNuEJaXDDldbV/image.png" alt=""><figcaption><p><em><strong>Activating Pipeline</strong></em></p></figcaption></figure>

* Click the ***Deactivate Pipeline*** icon.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/oF9F7ITCq81GrXQpT3Hw/image.png" alt=""><figcaption><p><em><strong>Accessing the Deactivate icon</strong></em></p></figcaption></figure>

* The ***Confirmation*** dialog box opens to get the confirmation for the action.
* Click the ***YES*** option.

&#x20;     ![](https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/nvQXiBTvNq0VEX5mQQf0/image.png)​

* A confirmation message appears.
* The pipeline gets deactivated. The same gets communicated through the color of the ***Status*** icon.

<figure><img src="https://content.gitbook.com/content/q2i9CKCFbySxr6jRoJfA/blobs/y1i5QXsoM12U8Pcsj2VE/image.png" alt=""><figcaption><p><em><strong>Deactivating a Pipeline from the Pipeline List</strong></em></p></figcaption></figure>