Python Script

The Python Script component works as a normal python compile.

Please Note: The Python Script component can be used as a Reader, an API Data Ingestion component, a Transformation, or a writer component. The component can only return either in pandas df or a list of dictionaries.

All component configurations are classified broadly into 3 section

• Basic

• Metadata

• Resource Configuration

Steps to configure Python Script (Custom Python Script)

• Drag and drop the Python Script to the Workflow Editor.

• The Python script will read data from an input event and will pass the processed data into an output event, so create and connect two events to the Python Script component.

• One reader is needed to pass the data to the input event. (in this case, the ES Reader component is used).

• Click the dragged Python Script component to get the component properties tabs.

Basic Information Tab

It is the default tab to open for the Python Script component while configuring the component.

• Select an Invocation type from the drop-down menu to confirm the running mode of the reader component. The supported invocation type for this component is Real-time.

• Deployment Type: It displays the deployment type for the component. This field comes pre-selected.

• Container Image Version: It displays the image version for the docker container. This field comes pre-selected.

• Failover Event: Select a failover Event from the drop-down menu.

• Batch Size: Provide the maximum number of records to be processed in one execution cycle (Min limit for this field is 10).

Meta Information Tab

Open the Meta Information Tab to open the fields and configure them.

• Component Name: Provide a name for the Python Script component.

Please Note: The component name should be without space and special characters. Use the underscore symbol to show space in between words.

• Python Script: Insert the Python script containing at least one function. The function should not have an argument, data frame argument, or custom argument.

• Start Function Name: It displays all the function names used in the python script in a drop-down menu. Select one function name with which you want to start.

• In Event Data Type: Provide input data type as a data frame or list.

• External Library: Provide the external library name in this field. Insert multiple library names separated by commas.

• Input Data: Use custom argument names as keys and provide the required value.

Saving the Component Configuration

• Click the Save Component in Storage icon.

• Click the Update Pipeline icon to save the Pipeline workflow. (After getting the success message)

• Activate the Pipeline workflow.

• Open the Log section to see the logs.

• The Python Script component is ready to read the data coming from the input event, it transforms the data and returns output data.

Please Note: The below-given instructions should be followed while writing a Python script in the Data Pipeline:

• The Python script needs to be written inside a valid Python function. E.g., The entire code body should be inside the proper indentation of the function (Use 4 spaces per indentation level.)

• The Python script should have at least one main function. Multiple functions are acceptable, and one function can call another function.

  • It should be written above the calling function body (if the called function is an outer function).

  • It should be written above the calling statement (if called function is an inner function).

• Spaces are the preferred indentation method.

• Do not use "type" as the function argument as it is a predefined keyword.

• The code in the core Python distribution should always use UTF-8.

• Single-quoted strings and double-quoted strings are considered the same in Python.

• All the packages used in the function need to import explicitly before writing the function.

• The Python script should return data in the form of a data frame or list only. The form of data should be defined while writing the function.

• If the user uses some Kafka event data for transformation, then the first argument of the function should be a data frame or list.

• If the user needs to use some external library, the user needs to mention the library name in the external libraries field. If the user wants to use multiple external libraries, the library names should be separated by a comma.

• If you need to pass some external input in your main function, then you can use the input data field. The key name should be the same according to the variable's name and value that is put as per the requirement.

• We can use that component as a reader, transformation, and writer.

Python Script Examples:

The Custom Python Script transform component supports 3 types of scripts in the Data Pipeline.

1. As Reader Component: If you don’t have any in Event then you can use no argument function. For Example:

import json
import requests
import pandas as pd
def getmovies_result():
    data = requests.get("http://www.omdbapi.com/?s=water&apikey=ba5d53d4")
    loaded_json = json.loads(data.content)
    data = loaded_json['Search']
    df = pd.DataFrame.from_dict(data, orient='columns')
    return df

2. As Transformation component: If you have data to execute some operation, then use the first argument as data or a list of dictionaries. For Example,

def getdata(df):
    cond1 = df['Unit Price'] > 450
    filter_df = df[cond1]
    return filter_df

Here the df holds the data coming from the previous event as argument to the pram of the method.

3. Custom Argument with Data: If there is a custom argument with the data-frame i.e. the data is coming from the previous event and we have passed the custom argument to the parameter of the function. here df will hold the data from the previous event and the second param: arg range can be given in the input data section of the component.

#
def getdata(df, range):
    cond1 = df['Unit Price'] >  range
    filter_df = df[cond1]
    return filter_df