AI Agents

Get information on the intelligent component of the pipeline to automate data ingestion and related processes.

What is an AI Agent?

An AI Agent is a software program or system designed to operate autonomously or semi-autonomously by perceiving its environment, processing information, and taking actions to achieve defined goals. It can interact with humans, other systems, or the physical world.

AI Agents in Data Pipeline

Within a data pipeline, an AI agent operates as a sophisticated component that automates key processes such as data ingestion, processing, and transformation, as well as complex decision-making. By leveraging AI and machine learning methodologies, it augments the efficiency of the pipeline, enabling the extraction of critical insights, optimization of workflows, and the initiation of autonomous actions.

AI Agents act as intelligent orchestration layers—automating tasks such as ingestion, preprocessing, anomaly detection, and triggering downstream actions.

Role of an AI Agent in a Data Pipeline

Data Ingestion & Preprocessing

• Collects data from multiple sources such as APIs, databases, and sensors.

• Cleans, normalizes, and transforms raw data before storage or further processing.

Intelligent Data Processing

• Detects anomalies, missing values, or inconsistencies.

• Applies feature engineering or automated data labeling.

Automated Decision-Making

• Executes ML model predictions for classification, clustering, or forecasting.

• Triggers alerts or initiates downstream processes based on real-time insights.

Optimization & Orchestration

• Dynamically adjusts pipeline parameters based on real-time conditions.

• Improves data flow efficiency through intelligent resource allocation.

Self-Learning & Adaptation

• Continuously improves accuracy and efficiency by learning from new data patterns.

• Refines decisions through iterative feedback loops.

Agentic Execution Loop

AI Agents operate in a loop-based execution cycle:

1. Perception – Capture input data or environment signals.

2. Reasoning – Process information using AI/ML or LLM outputs.

3. Action – Call a function, trigger a tool, or run a workflow step.

4. Observation – Evaluate the results of the action.

5. Iteration – Repeat the loop until a satisfactory outcome is achieved.

This iterative cycle ensures adaptability and continuous improvement of pipeline workflows.

Please note: Agents are useful when you need an LLM to determine the workflow of an app. But they often overkill.

AI Agent Use Case

Check the illustration on how to use an AI Agent component in the pipeline workflow.

Please note: This use case demonstrates how to consume an exported Agent as a tool inside a pipeline workflow. Users can also configure an AI Agent component using the Data Pipeline interface.

Exporting an AI Agent as a Tool

The Export as Tool feature allows users to register an AI Agent script as a reusable tool within the Data Science Lab (DSL). This enables integration of agentic logic into other workflows and pipelines.

Prerequisites

Before exporting an AI Agent as a tool, ensure the following:

• You have permission to access the DSL module.

• At least one AI Agent is created under your user account.

Steps to Export an AI Agent as a Tool

1

Access & Open the Data Science Lab

• Log in to the platform and navigate to the Data Science Lab module using the Navigation panel.

2

Access Agentic Tools

• Navigate to the Agentic Tools list page.

• Select an active agentic tool from the displayed list.

  

3

Open Workspace

• You will be redirected to the Workspace interface of the selected agentic tool.

4

Select Notebook

• From the Repo folder, select the notebook containing the agentic script.

Please note: For newly created agentic projects, you must first create an agentic script in a notebook before exporting.

5

Open Context Menu

• Click the ellipsis (⋮) icon next to the notebook.

• From the context menu, select Register.

  

6

Register Notebook

• The Register dialog box appears.

• Select the Export as a Tool option.

• Review the script preview displayed in the Preview section.

• Click Finish to complete the export process.

  

7

Confirmation

• A success message confirms that the tool has been created and registered successfully.

  

• Once registered, the exported tool becomes available in the Agentic Tools list for reuse across other projects and pipelines.

Consuming an Exported AI Agent

Once an AI Agent script is successfully registered as a tool, it can be integrated into a Data Pipeline Workflow using the AI Agent component. This allows the exported tool to participate in automated data flows alongside other pipeline components.

Steps to Consume an Exported AI Agent

1

Access the Data Engineering Module

• Navigate to the Data Engineering module.

• Open the Pipelines list page.

• Select a pipeline designed for agentic workflows.

• Example: The mail_agent pipeline containing an email listener and out-event components.

Please note: If no pipeline exists, create a new pipeline before proceeding.

2

Add the AI Agent Component

• Click the Add Component/Event icon.

• In the Components/Events panel, open the Components tab.

• Expand the AI Agents component category.

• Drag and drop the AI Agent component onto the canvas.

  • The component will auto-connect to the existing event component in the workflow.

3

Configure the AI Agent Component

• Basic Information

  • The Basic Information tab opens by default when the component is selected.

• Meta Information

  • Open the Meta Information tab.

  • Set the Number of Outputs and define the Output Node for the agentic component.

  • Create or add an event to the agent component.

4

More Configuration for Agent Component

• Agent Configuration

• Configure the following Meta Information for the dragged Agent Component:

  • Role: Define the role of the agent.

  • Description: Provide details clarifying the agent's task.

  • Select Agentic Project: Choose a registered agentic project from the dropdown list.

• Task Details

  • Navigate to the Task Details section and configure:

    • Out Event: Select the out event from the dropdown.

    • Description: Enter clear task instructions for the agent to execute.

    • JSON Schema: Upload a JSON schema that defines the expected output format.

  • Click Save Component in Storage.

    • A success message confirms the component properties have been saved.

5

Update and Run the Pipeline

• Click the Update Pipeline icon to apply changes.

  • A success message confirms the pipeline workflow has been updated.

• Run or activate the pipeline.

6

Monitor Execution

• Component Status

  • Open the Component Status tab to check the execution status of the agentic tool.

• Log Panel

  • Click the Logs icon in the toolbar to open the Log Panel.

  • Use the Refresh icon to update the logs.

    • A confirmation message indicates that the latest logs have been added.

    • Review the detailed runtime logs for the AI Agent in the Logs tab.

• By following these steps, users can successfully integrate an exported AI Agent tool within a Data Pipeline, configure its behavior, and monitor its execution in real-time.