List Pipelines
The List Pipelines option opens the available Pipeline List for the logged-in user. All the saved pipelines by a user are listed on this page. Please refer to the below image for reference:
Pin & Unpin Pipelines
The Pin and Unpin feature allows users to prioritize and easily access specific pipelines within a list. This functionality is beneficial for managing tasks, projects, or workflows efficiently. This feature is available on each pipeline card in the list view.
Navigate to the List Pipelines page.
Select a Pipeline from the list that you wish to pin.
Click the Pin icon provided for the selected pipeline.
The Pipeline gets pinned to the list and it will be moved to the top of the list if it is the first pinned pipeline.
The user can pin multiple pipelines to the list.
Click the Unpin icon for a pinned pipeline.
The pipeline will be unpinned from the list.
Pull & Push Pipeline
Pushing a Pipeline into VCS
Navigate to the Pipeline List page.
Select a data pipeline from the displayed list.
Click the Push & Pull Pipeline icon for the selected data pipeline.
The Push/ Pull drawer appears. It displays the name of the selected Pipeline. E.g., In the following image the Push/ Pull Restaurant Analysis WK5 heading, the Restaurant Analysis WK5 is the name of the selected Pipeline.
Provide a Commit Message (required) for the data pipeline version.
Select a Push Type out of the below-given choices to push the pipeline:
Version Control: For versioning of the pipeline in the same environment. In this case, the selected Push Type is Version Control.
GIT Export (Migration): This is for pipeline migration. The pushed pipeline can be migrated to the destination environment from the migration window in the Admin Module.
Click the Save option.​
The Push/ Pull window for a Pipeline Based on the selected Push Type the pipeline gets moved to Git or VCS, and a notification message appears to confirm the completion of the action.
Check out the illustrations below on the Version Control and Pipeline Migration functionalities.
Version Control:
Pipeline Migration:
Pulling a Pipeline
The Pull feature helps users pull the previously moved versions of a pipeline from the VCS. Thus, it can help the users significantly to recover the lost pipelines or avoid unwanted modifications made to the pipeline.
Check out the walk-through on how to pull a pipeline version from the VCS.
Navigate to the Pipeline List page.
Select a data pipeline from the displayed list.
Click the Pull from GIT icon for the selected data pipeline.
Pull/Push icon provided on the Pipeline Lis The Push/ Pull drawer opens with the selected Pipeline name.
Select the data pipeline version by marking the given checkbox.
Click the Save option.
A notification appears to inform that the selected pipeline workflow is imported.
View Pipeline
Clicking on the View icon will direct the user to the pipeline workflow editor page.
Navigate to the Pipeline List page.
Select a Pipeline from the list.
Click the View icon.
Accessing the View icon on the Pipeline List The Pipeline Editor page opens for that pipeline.
Searching a Pipeline
The user can search for a specific pipeline by using the Search bar on the Pipeline List. By typing a common name all the existing pipelines having that word will be listed. E.g., By typing the letters 'resta' all the existing pipelines with those letters get listed.
View Pipeline Details from the List Pipelines page
Click on the pipeline to view the information of the pipeline on the list pipeline page. Once clicked on the pipeline name, a menu will open on the right side of the screen showing all the details of the selected pipeline.
Please look at the demonstration to understand how the Pipeline details are displayed on the List Pipelines page.
By clicking on a pipeline name, the following details of that pipeline will appear below in two tabs:
Pipeline Details
The Pipeline Details panel displays key metadata: the creator, last updater, last activation timestamp, and last deactivation timestamp. This information helps users track changes and manage their workflows effectively.
Components: Number of components used in the pipeline.
Created: Indicates the Pipeline owner's name (with Date and Time stamp).
Updated: Name of the person who has updated the pipeline (with Date and Time stamp).
Last Activated: Displays the person's name who last activated the pipeline (with Date and Time stamp).
Last Deactivated: Displays the person's name who last deactivated the pipeline (with Date and Time stamp).
Description: This field will show the description of the pipeline if given by the pipeline owner.
Total Component Config
Total Allocated Max CPU: The maximum allocated CPU in cores.
Total Allocated Min CPU: The minimum allocated CPU in cores.
Total Allocated Max Memory: The maximum allocated memory in MB.
Total Allocated Min Memory: The minimum allocated memory in MB.
Failed and Ignored components
This section displays the component that has been failed or ignored in the selected pipeline workflow.
Users can click the Failed Components & Ignored Components options to display the failed or ignored component names. The information icon provided next to the names of the Failed or Ignored components will redirect the user to get more information about those components.
Failed Component
Ignoring a Failed Component
Navigate to the Failed and Ignored Components section for a pipeline.
Click the Ignore Failure Components icon provided for Failed Components.
The Confirm dialog box opens.
Use the Comment space to comment on the Failed component.
Click the Yes option to ignore the Failed Components.
Ignored Component
Please Note: The Failed and Ignored Components tab will be empty if the pipeline flow is successful.
Pipeline Component Configuration
Check out the illustration on the Pipeline Component Configuration page.
Navigate to the Pipeline List page.
Select a Pipeline from the list.
Click the Pipeline Component Configuration icon.
The configuration page will list all the components used in the selected pipeline. The Basic Info tab opens by default.
Click the Configuration tab to access the pipeline components' configuration details. The user may modify the details and update it using the Save option.
Sharing a Pipeline
Users can share a pipeline with one or multiple users and user Groups using the Share Pipeline option.
Check out the following walk-through on how to share a pipeline with user/ user group and exclude the user from a shared pipeline.
Click the List Pipelines icon to open the Pipeline List.
Select a Pipeline from the pipeline list.
Click on the Share Pipeline icon.
The Share Pipeline window opens.
Select an option from the given choices: User (the default tab) and User Group or Exclude User (the Exclude User option can be chosen if the pipeline is already shared with a user/user group and you wish to exclude them from the privilege).
Select a user or user(s) from the displayed list of users (In case of the User Group(s) tab, it displays the names of the User Groups.
Click the arrow to move the selected User(s)/ User Group(s).
Sharing a Pipeline The selected user(s)/user group(s) get moved to the box given on the right (In case of the Exclude User option the selected user/ the user moved to the right-side box looses the access of the shared pipeline).
Click the Save option.​
Privilege for the selected pipeline gets updated and the same gets communicated through a message.
Confirmation message after a Pipeline gets shared By completing the steps mentioned above, the concerned pipeline gets successfully shared with the selected user/user group or the selected users can also be excluded from their privileges for the concerned pipeline.
Monitor Pipeline
By clicking the Pipeline Monitor icon, the user will be redirected to the Pipeline Monitoring page.
Navigate to the List Pipelines page.
Select a Pipeline from the list.
Click the Pipeline Monitor icon.
The user gets redirected to the Pipeline Monitoring page of the selected Pipeline. The Monitor tab opens by default.
Click the expand icon to display the monitor page in detail.
The user may click on the Data Metrics and System Logs tabs to get more details on the pipeline.
Failure Analysis
Failure analysis is a central failure mechanism. Here, the user can identify the failure reason. Failure of any pipeline stored at a particular location (collection). From there you can query your failed data in the Failure Analysis UI. It displays the failed records, cause, event time, and pipeline ID.
Check out the illustration to access the Failure Analysis for a selected Pipeline.
Navigate to the Pipeline List page.
Select a Pipeline with Failed status from the list.
Click on the Failure Analysis icon.
Accessing the Failure Analysis option from the List Pipelines page The user gets redirected to the Failure Analysis page.
The user can further get the failure analysis of the specific component that failed in the pipeline workflow.
Use the ellipsis icon to get redirected to the Monitoring and Data Metrics options for a component.
Deleting a Pipeline
Navigate to the List Pipelines page.
Select a Pipeline from the list.
Click the Delete Pipeline icon for the pipeline.
Deleting a pipeline from List Pipelines page The Confirm dialog box appears to confirm the action.
Click the YES option to confirm the deletion.
The selected Pipeline gets removed from the list.
the Noigicaion message after deleting a Pipeline
Activate/Deactivate Pipeline
The user can activate/deactivate the pipeline by clicking on the Activate/ Deactivate icon.
Click the List Pipelines icon to open the Pipeline list.
Select a Pipeline from the pipeline list.
Click on the Activate Pipeline icon.
Accessing the Activate option on the List Pipelines The Confirmation dialog box opens to get the confirmation for the action.
Click the YES option.
A confirmation message appears.
The pipeline gets activated. The Activate Pipeline icon turns into Deactivate Pipeline.
The Status option diaplays UP and turns green in color.
Click the Deactivate Pipeline icon to deactivate the pipeline.
Accessing the Deactivate Pipeline icon The Confirmation dialog box opens to get the confirmation for the action.
Click the YES option.
A confirmation message appears.
The pipeline gets deactivated. The Deactivate Actions icon turns Activate.
The same gets communicated through the color of the Status icon which displays OFF and turns Gray.
Deactivating a Pipeline from the List Pipelines Page
Last updated