List Pipelines
The List Pipelines option opens the available Pipeline List for the logged-in user. All the saved pipelines by a user get listed on this page. Please refer the below image for the reference:
Please Note:
If the logged-in user has Admin access, the user can see all the pipelines created by all users.
The non-admin users get to access the list of pipelines either created by them or shared to them by the other users.
Pull & Push Pipeline
Pushing a Pipeline into VCS
Navigate to the Pipeline List page.
Select a data pipeline from the displayed list.
Click the Push & Pull Pipeline icon for the selected data pipeline.
Please Note:
The Push to VCS and Pull Pipeline from GIT features are present on the List Pipeline page and as well as on the Pipeline Editor page.
The Push/ Pull drawer appears. It displays the name of the selected Pipeline. E.g., In the following image the Push/ Pull Restaurant Analysis WK5 heading, the Restaurant Analysis WK5 is the name of the selected Pipeline.
Provide a Commit Message (required) for the data pipeline version.
Select a Push Type out of the below-given choices to push the pipeline:
Version Control: For versioning of the pipeline in the same environment. In this case, the selected Push Type is Version Control.
GIT Export (Migration): This is for pipeline migration. The pushed pipeline can be migrated to the destination environment from the migration window in the Admin Module.
Click the Save option.
A notification message appears to confirm the completion of the action.
If the user chooses Git Export (Migration), the notification message comes like below:
Check out the below-given illustrations on how to attempt Version Control and Pipeline Migration.
Version Control:
Pipeline Migration:
Please Note:
The pipeline pushed to the VCS using the Version Control option, can be pulled directly from the Pull Pipeline option.
The user also gets an option to Push the pipeline to GIT. This action will be considered as Pipeline Migration. The user needs to follow the Pipeline Migration steps given the Admin module of the Platform to pull a version that has been pushed to GIT.
Pulling a Pipeline
This feature is for pulling the previously moved versions of a pipeline that are committed by the user. This can help a user significantly to recover the lost pipelines or avoid unwanted modifications made to the pipeline.
Check out the walk-through on how to pull a pipeline version from the GIT.
Navigate to the Pipeline List page.
Select a data pipeline from the displayed list.
Click the Pull from GIT icon for the selected data pipeline.
The Push/ Pull drawer opens with the selected Pipeline name.
Select the data pipeline version by marking the given checkbox.
Click the Ok option.
A confirmation message appears to assure the user that the concerned pipeline workflow has been pulled.
Another notification appears to inform that the selected pipeline workflow is imported.
Please Note: The pipeline that you pull will be changed to the selected version. Please make sure to manage the versions of the pipeline properly.
View Pipeline
Clicking on the View icon will direct the user to the pipeline workflow editor page.
Navigate to the Pipeline List page.
Select a Pipeline from the list.
Click the View icon.
The Pipeline Editor page opens for that pipeline.
Please Note: The user can open the Pipeline Editor for the selected pipeline from the list by clicking the View icon or the Pipeline Workflow Editor icon on the Pipeline List page.
Searching a Pipeline
The user can search for a specific pipeline by using the Search bar on the Pipeline List. By typing a common name all the existing pipelines having that word will list. E.g., By typing the word sandbox all the existing pipelines with that word get listed in the following image:
View Pipeline details from the list Pipeline page
Click on the pipeline to view the information of the pipeline on the list pipeline page. Once clicked on the pipeline name, a menu will open in the right side of the screen showing all the details of the selected pipeline. Please go through the below demonstration:
By clicking on a pipeline name, the following details of that pipeline will be appear below in two tabs:
Pipeline Details
Components: Number of components used in the pipeline.
Created By: Owner name of the pipeline.
Created Date: Pipeline created date.
Updated By: Name of the person who has updated the pipeline recently.
Updated Date: Date when the pipeline is updated.
Description: This field will show the description of the pipeline if given.
Total Component Config
Total Allocated Max Cpu: The maximum allocated CPU in cores.
Total Allocated Min Cpu: The minimum allocated CPU in cores.
Total Allocated Max Memory: The maximum allocated memory in MB.
Total Allocated Min Memory: The minimum allocated memory in MB.
Failed and Ignored components
This section displays the component which has failed or ignored in the pipeline workflow.
Please Note: The Failed and Ignored components tab will be empty if the pipeline flow is successful.
Pipeline Component Configuration
Check out the given illustration on the Pipeline Component Configuration page.
Navigate to the Pipeline List page.
Select a Pipeline from the list.
Click the Pipeline Component Configuration icon.
All the components used in the selected pipeline will be listed on the configuration page.
Please refer this link for more details on Pipeline Component Configuration page.
Please Note:
The user can edit the required basic or configuration details for all the pipeline components.
The Pipeline Component Configuration page also displays icons to access Pipeline Testing, Failure Analysis, Open Pipeline (Editor Workspace), and List Pipeline.
Monitor Pipeline
By clicking the Pipeline Monitor icon, the user will get redirected to the Pipeline Monitoring page.
Navigate to the Pipeline List page.
Select a Pipeline from the list.
Click the Pipeline Monitor icon.
The user gets redirected to the Pipeline Monitoring page of the selected Pipeline.
Please Note: Check out the Pipeline Monitoring page for more details on this topic.
Failure Analysis
Failure analysis is a central failure mechanism. Here, the user can identify the failure reason. Failures of any pipeline stored at a particular location(collection). From there you can query your failed data in the Failure Analysis UI. It displays the failed records along with cause, event time, and pipeline Id.
Navigate to the Pipeline List page.
Select a Pipeline with Failed status from the list.
Click on the Failure Analysis icon.
The user gets redirected to the Failure Analysis page.
The user can further get failure analysis of the specific component that failed in the pipeline workflow.
Use the ellipsis icon to get redirected to the Monitoring and Data Metrics options for a component.
Please Note: Refer to the Failure Analysis for more details on this.
Deleting a Pipeline
Navigate to the List Pipeline page.
Select a Pipeline from the list.
Click the Delete icon for the pipeline.
The Confirm dialog box appears to confirm the action.
Click YES option to confirm the deletion.
The selected Pipeline gets removed from the list.
Sharing a Pipeline
The user can share a pipeline with one or multiple users and user Groups using the Share Pipeline option.
Check out the following walk-through on how to share a pipeline with user/ user group and exclude the user from a shared pipeline.
Click the List Pipelines icon to open the Pipeline List.
Select a Pipeline from the pipeline list.
Click on the Share Pipeline icon.
The Share Pipeline window opens.
Select an option out of the given choices: User (default tab) and User Group or Exclude User (the Exclude User option can be chosen, if the pipeline is already shared with a user/user group and you wish to exclude them from the privilege).
Select a user or user(s) from the displayed list of users (In case of the User Group(s) tab, it displays the names of the User Groups.
Click the arrow to move the selected User(s)/ User Group(s).
The selected user(s)/user group(s) get moved to the box given on the right (In case of the Exclude User option the selected user/ the user moved to the right-side box looses the access of the shared pipeline).
Click the Save option.
Privilege for the selected pipeline gets updated and the same gets communicated through a message.
By completing the steps mentioned above, the concerned pipeline gets successfully shared with the selected user/user group or the selected users can also be excluded from their privileges for the concerned pipeline.
Please Note:
An admin user can View, Edit/Modify, Share, and Delete a shared pipeline.
A non-admin user can View and Edit/ Modify a shared Pipeline, but it can't Share it further or Delete it.
Activate/Deactivate Pipeline
The user can activate/deactivate the pipeline by clicking on the Activate/ Deactivate icon.
Click the List Pipelines icon to open the Pipeline list.
Select a Pipeline from the pipeline list.
Click on the Activate Pipeline icon.
The Confirmation dialog box opens to get the confirmation for the action.
Click the YES option.
A confirmation message appears.
The pipeline gets activated. The Activate Pipeline icon turns into Deactivate Pipeline.
The same gets communicated through the Status icon color.
Click the Deactivate Pipeline icon to deactivate the pipeline.
The Confirmation dialog box opens to get the confirmation for the action.
Click the YES option.
A confirmation message appears.
The pipeline gets deactivated. The same gets communicated through the color of the Status icon.
Last updated
